[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 439095939 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.003242] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005016] kvm-guest: setup PV IPIs [ 0.007849] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008026] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009013] pid_max: default: 32768 minimum: 301 [ 0.011047] LSM: Security Framework initializing [ 0.012057] Yama: becoming mindful. [ 0.013059] SELinux: Initializing. [ 0.014091] *** VALIDATE selinux *** [ 0.023419] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027795] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029116] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030094] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031115] *** VALIDATE tmpfs *** [ 0.032471] *** VALIDATE proc *** [ 0.034131] *** VALIDATE cgroup *** [ 0.035014] *** VALIDATE cgroup2 *** [ 0.036284] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038167] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040030] Spectre V2 : User space: Vulnerable [ 0.041012] Speculative Store Bypass: Vulnerable [ 0.043886] debug: unmapping init [mem 0xffffffffa1e59000-0xffffffffa1e60fff] [ 0.045937] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046768] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047023] ... version: 2 [ 0.048013] ... bit width: 48 [ 0.049012] ... generic registers: 4 [ 0.050012] ... value mask: 0000ffffffffffff [ 0.051013] ... max period: 00007fffffffffff [ 0.052013] ... fixed-purpose events: 3 [ 0.052964] ... event mask: 000000070000000f [ 0.053324] rcu: Hierarchical SRCU implementation. [ 0.055484] smp: Bringing up secondary CPUs ... [ 0.056639] x86: Booting SMP configuration: [ 0.057028] .... node #0, CPUs: #1 #2 #3 [ 0.060338] smp: Brought up 1 node, 4 CPUs [ 0.062016] smpboot: Max logical packages: 1 [ 0.063029] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.126686] node 0 deferred pages initialised in 60ms [ 0.129372] devtmpfs: initialized [ 0.130167] x86/mm: Memory block size: 128MB [ 0.132524] gcov: version magic: 0x41383552 [ 0.134075] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.135061] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.136292] pinctrl core: initialized pinctrl subsystem [ 0.137197] [ 0.137614] ************************************************************* [ 0.138012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.139007] ** ** [ 0.140009] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.141009] ** ** [ 0.142008] ** This means that this kernel is built to expose internal ** [ 0.143014] ** IOMMU data structures, which may compromise security on ** [ 0.144010] ** your system. ** [ 0.145012] ** ** [ 0.146011] ** If you see this message and you are not debugging the ** [ 0.147018] ** kernel, report this immediately to your vendor! ** [ 0.148021] ** ** [ 0.149015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.150021] ************************************************************* [ 0.151722] NET: Registered protocol family 16 [ 0.152530] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.153060] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.154117] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.155465] cpuidle: using governor menu [ 0.158481] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.160578] PCI: Using configuration type 1 for base access [ 0.163141] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.173132] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.174024] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.176080] cryptd: max_cpu_qlen set to 1000 [ 0.179076] ACPI: Added _OSI(Module Device) [ 0.180017] ACPI: Added _OSI(Processor Device) [ 0.181015] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.182012] ACPI: Added _OSI(Processor Aggregator Device) [ 0.186244] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.188727] ACPI: Interpreter enabled [ 0.189064] ACPI: PM: (supports S0 S3 S4 S5) [ 0.190000] ACPI: Using IOAPIC for interrupt routing [ 0.192132] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.196406] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.207578] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.210050] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.212022] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.216091] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.221424] acpiphp: Slot [2] registered [ 0.222104] acpiphp: Slot [5] registered [ 0.224138] acpiphp: Slot [6] registered [ 0.225110] acpiphp: Slot [3] registered [ 0.227173] acpiphp: Slot [4] registered [ 0.228114] acpiphp: Slot [7] registered [ 0.230091] acpiphp: Slot [8] registered [ 0.231162] acpiphp: Slot [9] registered [ 0.232137] acpiphp: Slot [10] registered [ 0.234095] acpiphp: Slot [11] registered [ 0.235120] acpiphp: Slot [12] registered [ 0.236095] acpiphp: Slot [13] registered [ 0.238117] acpiphp: Slot [14] registered [ 0.239092] acpiphp: Slot [15] registered [ 0.241165] acpiphp: Slot [16] registered [ 0.242130] acpiphp: Slot [17] registered [ 0.243163] acpiphp: Slot [18] registered [ 0.245100] acpiphp: Slot [19] registered [ 0.246085] acpiphp: Slot [20] registered [ 0.247088] acpiphp: Slot [21] registered [ 0.249096] acpiphp: Slot [22] registered [ 0.250156] acpiphp: Slot [23] registered [ 0.252104] acpiphp: Slot [24] registered [ 0.253093] acpiphp: Slot [25] registered [ 0.254153] acpiphp: Slot [26] registered [ 0.256114] acpiphp: Slot [27] registered [ 0.258105] acpiphp: Slot [28] registered [ 0.259114] acpiphp: Slot [29] registered [ 0.260122] acpiphp: Slot [30] registered [ 0.262119] acpiphp: Slot [31] registered [ 0.264099] PCI host bridge to bus 0000:00 [ 0.265017] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.268049] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.270022] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.273027] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.275022] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.278026] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.279263] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.282097] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.286414] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.293020] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.297055] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.300020] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.303021] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.305016] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.307622] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.310680] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.313048] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.315865] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.319840] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.329015] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.334016] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.338755] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.347017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.359020] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.385019] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.400614] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.407017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.415020] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.435018] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.448279] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.451398] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.453469] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.456421] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.459235] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.463036] iommu: Default domain type: Passthrough [ 0.464470] SCSI subsystem initialized [ 0.465141] ACPI: bus type USB registered [ 0.467107] usbcore: registered new interface driver usbfs [ 0.469080] usbcore: registered new interface driver hub [ 0.471077] usbcore: registered new device driver usb [ 0.472166] pps_core: LinuxPPS API ver. 1 registered [ 0.474014] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.477061] PTP clock support registered [ 0.480213] EDAC MC: Ver: 3.0.0 [ 0.482182] PCI: Using ACPI for IRQ routing [ 0.484767] NetLabel: Initializing [ 0.485010] NetLabel: domain hash size = 128 [ 0.487012] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.489082] NetLabel: unlabeled traffic allowed by default [ 0.491109] vgaarb: loaded [ 0.493367] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.495014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.501452] clocksource: Switched to clocksource kvm-clock [ 0.616242] VFS: Disk quotas dquot_6.6.0 [ 0.617787] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.620150] *** VALIDATE ramfs *** [ 0.621225] *** VALIDATE hugetlbfs *** [ 0.622855] pnp: PnP ACPI init [ 0.625415] pnp: PnP ACPI: found 6 devices [ 0.652128] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.655378] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.657655] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.659627] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.661686] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.663245] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.665132] NET: Registered protocol family 2 [ 0.667466] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.672028] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.675859] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.680194] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.684055] TCP: Hash tables configured (established 65536 bind 65536) [ 0.686601] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.689608] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.692468] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.695469] NET: Registered protocol family 1 [ 0.697852] RPC: Registered named UNIX socket transport module. [ 0.699754] RPC: Registered udp transport module. [ 0.701259] RPC: Registered tcp transport module. [ 0.702799] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.704531] NET: Registered protocol family 44 [ 0.705704] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.707541] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.709508] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.711682] PCI: CLS 0 bytes, default 64 [ 0.713175] Unpacking initramfs... [ 2.114658] debug: unmapping init [mem 0xffff96843cc64000-0xffff96843ffcffff] [ 2.118383] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.120690] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.123840] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.579132] Initialise system trusted keyrings [ 2.580751] Key type blacklist registered [ 2.582021] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.588967] zbud: loaded [ 2.591512] *** VALIDATE nfs *** [ 2.592239] *** VALIDATE nfs4 *** [ 2.593497] pstore: using deflate compression [ 2.596863] Platform Keyring initialized [ 2.672449] NET: Registered protocol family 38 [ 2.673687] Key type asymmetric registered [ 2.674614] Asymmetric key parser 'x509' registered [ 2.675796] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.677644] io scheduler mq-deadline registered [ 2.678714] io scheduler kyber registered [ 2.679818] io scheduler bfq registered [ 2.680933] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.683204] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.684941] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.686762] ACPI: Power Button [PWRF] [ 2.691580] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.696572] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.703035] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.731456] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.758210] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.762233] Non-volatile memory driver v1.3 [ 2.763203] Linux agpgart interface v0.103 [ 2.787810] virtio_blk virtio1: [vda] 145904 512-byte logical blocks (74.7 MB/71.2 MiB) [ 2.789323] vda: detected capacity change from 0 to 74702848 [ 2.807442] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.809056] vdb: detected capacity change from 0 to 1073741824 [ 2.817106] libphy: Fixed MDIO Bus: probed [ 2.833497] usbcore: registered new interface driver usbserial_generic [ 2.838353] usbserial: USB Serial support registered for generic [ 2.840829] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.847496] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.850281] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.854687] mousedev: PS/2 mouse device common for all mice [ 2.857765] rtc_cmos 00:05: RTC can wake from S4 [ 2.860682] rtc_cmos 00:05: registered as rtc0 [ 2.862214] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.865212] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.865341] intel_pstate: CPU model not supported [ 2.870990] hid: raw HID events driver (C) Jiri Kosina [ 2.871807] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.873311] usbcore: registered new interface driver usbhid [ 2.877659] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 2.877888] usbhid: USB HID core driver [ 2.881359] drop_monitor: Initializing network drop monitor service [ 2.882945] Initializing XFRM netlink socket [ 2.884605] NET: Registered protocol family 10 [ 2.887358] Segment Routing with IPv6 [ 2.888374] NET: Registered protocol family 17 [ 2.889961] mpls_gso: MPLS GSO support [ 2.896335] RAS: Correctable Errors collector initialized. [ 2.898227] AVX version of gcm_enc/dec engaged. [ 2.899276] AES CTR mode by8 optimization enabled [ 2.970694] sched_clock: Marking stable (2970671776, 0)->(3879309515, -908637739) [ 2.974820] registered taskstats version 1 [ 2.977229] Loading compiled-in X.509 certificates [ 2.979364] zswap: loaded using pool lzo/zbud [ 3.004652] Key type big_key registered [ 3.018339] Key type encrypted registered [ 3.019807] ima: No TPM chip found, activating TPM-bypass! [ 3.021661] ima: Allocated hash algorithm: sha1 [ 3.022917] ima: No architecture policies found [ 3.024454] evm: Initialising EVM extended attributes: [ 3.026106] evm: security.selinux [ 3.027332] evm: security.ima [ 3.028188] evm: security.capability [ 3.029483] evm: HMAC attrs: 0x1 [ 3.031566] rtc_cmos 00:05: setting system clock to 2026-08-15 07:17:00 UTC (1786778220) [ 3.037188] debug: unmapping init [mem 0xffffffffa2e03000-0xffffffffa2ffffff] [ 3.040014] debug: unmapping init [mem 0xffffffffa1b82000-0xffffffffa1e58fff] [ 3.049199] Write protecting the kernel read-only data: 28672k [ 3.052922] debug: unmapping init [mem 0xffffffffa0203000-0xffffffffa03fffff] [ 3.055853] debug: unmapping init [mem 0xffffffffa0b14000-0xffffffffa0bfffff] [ 3.088587] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.095717] systemd[1]: Detected virtualization kvm. [ 3.097484] systemd[1]: Detected architecture x86-64. [ 3.099255] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.123629] systemd[1]: No hostname configured. [ 3.125509] systemd[1]: Set hostname to . [ 3.127520] random: systemd: uninitialized urandom read (16 bytes read) [ 3.129555] systemd[1]: Initializing machine ID from random generator. [ 3.171721] random: ln: uninitialized urandom read (6 bytes read) [ 3.234039] random: systemd: uninitialized urandom read (16 bytes read) [ 3.235946] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.238511] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.241886] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket. Starting Apply Kernel Variables... [ OK ] Started Memstrack Anylazing Service. Starting Journal Service... [ OK ] Reached target Timers. Starting Setup Virtual Console... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 3.769943] device-mapper: uevent: version 1.0.3 [ 3.772120] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.443424] virtio_net virtio0 ens2: renamed from eth0 [ 4.610080] scsi host0: ata_piix [ 4.674915] scsi host1: ata_piix [ 4.676540] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 4.678161] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 8.272591] dracut-initqueue[584]: RTNETLINK answers: File exists [ 9.534952] random: crng init done [ 9.536712] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 9.886976] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.085694] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.309383] SELinux: Disabled at runtime. [ 11.363356] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.371974] systemd[1]: Detected virtualization kvm. [ 11.373786] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 11.864180] systemd[1]: initrd-switch-root.service: Succeeded. [ 11.867640] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 11.873682] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 11.879142] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 11.885846] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 11.896158] systemd[1]: Starting Journal Service... Starting Journal Service... [ 11.900229] systemd[1]: proc-sys-fs-binfmt_misc.automount: Refusing to start, unit to trigger not loaded. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Apply Kernel Variables... [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on udev Control Socket. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target rpc_pipefs.target. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice User and Session Slice. Activating swap /dev/disk/by-label/SWAP... Starting Remount Root and Kernel File Systems... [ OK ] Stopped target Initrd File Systems. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Slices. Mounting POSIX Message Queue File System... Mounting Huge Pages File System... [ 11.988836] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting Kernel Debug File System... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Starting udev Coldplug all Devices... [ OK ] Created slice system-getty.slice. [ OK ] Started Journal Service. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 12.318763] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 12.663539] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.697564] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.794729] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.807724] EDAC sbridge: Ver: 1.1.2 [ 14.027598] Key type dns_resolver registered [ 14.326569] NFS: Registering the id_resolver key type [ 14.328415] Key type id_resolver registered [ 14.330035] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started D-Bus System Message Bus. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. Starting Hostname Service... [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg201-client login: [ 49.495254] libcfs: loading out-of-tree module taints kernel. [ 49.615442] Key type ._llcrypt registered [ 49.624360] Key type .llcrypt registered [ 50.513900] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 50.559154] alg: No test for adler32 (adler32-zlib) [ 52.023258] Lustre: Lustre: Build Version: 2.17.57_1_g05abe65 [ 53.057094] LNet: Added LNI 192.168.202.1@tcp [8/256/0/180] [ 54.855380] Key type lgssc registered [ 55.412239] hrtimer: interrupt took 4395271 ns [ 56.925448] Lustre: Echo OBD driver; http://www.lustre.org/ [ 208.691646] Lustre: Mounted lustre-client - version 2.17.57_1_g05abe65 [ 214.855534] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 225.773760] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing check_logdir /tmp/testlogs/ [ 231.257382] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing yml_node [ 234.464707] Lustre: lustre-OST0000-osc-ffff96849059c800: disconnect after 23s idle [ 235.428824] Lustre: DEBUG MARKER: Client: 2.17.57.1 [ 237.732946] Lustre: DEBUG MARKER: MDS: 2.17.57.1 [ 240.325572] Lustre: DEBUG MARKER: OSS: 2.17.57.1 [ 242.084797] Lustre: DEBUG MARKER: -----============= acceptance-small: recovery-small ============----- Sat Aug 15 03:20:57 EDT 2026 [ 263.073949] Lustre: DEBUG MARKER: excepting tests: 136 [ 265.343606] Lustre: DEBUG MARKER: === recovery-small: start setup 03:21:21 (1786778481) === [ 270.038455] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing check_config_client /mnt/lustre [ 290.050814] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 301.535267] Lustre: DEBUG MARKER: === recovery-small: finish setup 03:21:57 (1786778517) === [ 303.093983] Lustre: DEBUG MARKER: == recovery-small test 1: create, chmod, stat: drop req, drop rep ========================================================== 03:21:59 (1786778519) [ 320.479166] Lustre: 10014:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786778521/real 1786778521] req@ffff968498f67480 x1873572811255680/t0(0) o700->lustre-MDT0000-mdc-ffff96849059c800@192.168.202.101@tcp:30/10 lens 264/248 e 0 to 1 dl 1786778537 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 320.521775] Lustre: lustre-MDT0000-mdc-ffff96849059c800: Connection to lustre-MDT0000 (at 192.168.202.101@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 320.584708] Lustre: lustre-MDT0000-mdc-ffff96849059c800: Connection restored to 192.168.202.101@tcp (at 192.168.202.101@tcp) [ 338.911300] Lustre: 10035:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786778540/real 1786778540] req@ffff968498f66a00 x1873572811257472/t0(0) o36->lustre-MDT0000-mdc-ffff96849059c800@192.168.202.101@tcp:12/10 lens 520/576 e 0 to 1 dl 1786778556 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 338.941385] Lustre: lustre-MDT0000-mdc-ffff96849059c800: Connection to lustre-MDT0000 (at 192.168.202.101@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 338.988381] Lustre: lustre-MDT0000-mdc-ffff96849059c800: Connection restored to 192.168.202.101@tcp (at 192.168.202.101@tcp) [ 356.831206] Lustre: 10060:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786778558/real 1786778558] req@ffff968498f67800 x1873572811258752/t0(0) o101->lustre-MDT0000-mdc-ffff96849059c800@192.168.202.101@tcp:12/10 lens 576/1152 e 0 to 1 dl 1786778574 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'tchmod.0' uid:0 gid:0 projid:0 [ 356.875415] Lustre: lustre-MDT0000-mdc-ffff96849059c800: Connection to lustre-MDT0000 (at 192.168.202.101@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 356.931098] Lustre: lustre-MDT0000-mdc-ffff96849059c800: Connection restored to 192.168.202.101@tcp (at 192.168.202.101@tcp) [ 376.287207] Lustre: 10082:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786778577/real 1786778577] req@ffff96849855bb80 x1873572811260928/t0(0) o36->lustre-MDT0000-mdc-ffff96849059c800@192.168.202.101@tcp:12/10 lens 488/512 e 0 to 1 dl 1786778593 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'tchmod.0' uid:0 gid:0 projid:4294967295 [ 376.312271] Lustre: lustre-MDT0000-mdc-ffff96849059c800: Connection to lustre-MDT0000 (at 192.168.202.101@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 376.370076] Lustre: lustre-MDT0000-mdc-ffff96849059c800: Connection restored to 192.168.202.101@tcp (at 192.168.202.101@tcp) [ 395.231193] Lustre: 10107:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786778596/real 1786778596] req@ffff968491eec380 x1873572811262208/t0(0) o34->lustre-MDT0000-mdc-ffff96849059c800@192.168.202.101@tcp:12/10 lens 472/728 e 0 to 1 dl 1786778612 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'statone.0' uid:0 gid:0 projid:0 [ 395.262914] Lustre: lustre-MDT0000-mdc-ffff96849059c800: Connection to lustre-MDT0000 (at 192.168.202.101@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 395.333975] Lustre: lustre-MDT0000-mdc-ffff96849059c800: Connection restored to 192.168.202.101@tcp (at 192.168.202.101@tcp) [ 414.175221] Lustre: 10128:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786778615/real 1786778615] req@ffff9684880caa00 x1873572811263488/t0(0) o34->lustre-MDT0000-mdc-ffff96849059c800@192.168.202.101@tcp:12/10 lens 472/728 e 0 to 1 dl 1786778631 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'statone.0' uid:0 gid:0 projid:0 [ 414.221643] Lustre: lustre-MDT0000-mdc-ffff96849059c800: Connection to lustre-MDT0000 (at 192.168.202.101@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 414.270585] Lustre: lustre-MDT0000-mdc-ffff96849059c800: Connection restored to 192.168.202.101@tcp (at 192.168.202.101@tcp) [ 423.139762] Lustre: DEBUG MARKER: == recovery-small test 4: open: drop req, drop rep ======= 03:23:59 (1786778639) [ 439.776077] Lustre: 10734:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786778641/real 1786778641] req@ffff96849855b480 x1873572811265792/t0(0) o101->lustre-MDT0000-mdc-ffff96849059c800@192.168.202.101@tcp:12/10 lens 576/1152 e 0 to 1 dl 1786778657 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'cat.0' uid:0 gid:0 projid:0 [ 439.844459] Lustre: lustre-MDT0000-mdc-ffff96849059c800: Connection to lustre-MDT0000 (at 192.168.202.101@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 439.882336] Lustre: lustre-MDT0000-mdc-ffff96849059c800: Connection restored to 192.168.202.101@tcp (at 192.168.202.101@tcp) [ 466.291251] Lustre: DEBUG MARKER: == recovery-small test 5: rename: drop req, drop rep ===== 03:24:42 (1786778682) [ 484.319976] Lustre: 11356:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786778685/real 1786778685] req@ffff968498559c00 x1873572811271936/t0(0) o101->lustre-MDT0000-mdc-ffff96849059c800@192.168.202.101@tcp:12/10 lens 576/1152 e 0 to 1 dl 1786778701 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'mv.0' uid:0 gid:0 projid:0 [ 484.368803] Lustre: 11356:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 484.383345] Lustre: lustre-MDT0000-mdc-ffff96849059c800: Connection to lustre-MDT0000 (at 192.168.202.101@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 484.398023] Lustre: Skipped 1 previous similar message [ 484.428034] Lustre: lustre-MDT0000-mdc-ffff96849059c800: Connection restored to 192.168.202.101@tcp (at 192.168.202.101@tcp) [ 484.443736] Lustre: Skipped 1 previous similar message [ 511.279685] Lustre: DEBUG MARKER: == recovery-small test 6: link, unlink: drop req, drop rep ========================================================== 03:25:27 (1786778727) [ 534.496894] Lustre: lustre-OST0001-osc-ffff96849059c800: disconnect after 24s idle [ 534.516399] Lustre: Skipped 1 previous similar message [ 548.322321] Lustre: 12010:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786778749/real 1786778749] req@ffff968491eee300 x1873572811283584/t0(0) o36->lustre-MDT0000-mdc-ffff96849059c800@192.168.202.101@tcp:12/10 lens 512/440 e 0 to 1 dl 1786778765 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'link.0' uid:0 gid:0 projid:4294967295 [ 548.364112] Lustre: 12010:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 566.754369] Lustre: lustre-MDT0000-mdc-ffff96849059c800: Connection to lustre-MDT0000 (at 192.168.202.101@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 566.770359] Lustre: Skipped 3 previous similar messages [ 566.817300] Lustre: lustre-MDT0000-mdc-ffff96849059c800: Connection restored to 192.168.202.101@tcp (at 192.168.202.101@tcp) [ 566.831308] Lustre: Skipped 3 previous similar messages [ 594.678563] Lustre: DEBUG MARKER: == recovery-small test 8: touch: drop rep (bug 1423) ===== 03:26:50 (1786778810) [ 621.948953] Lustre: DEBUG MARKER: == recovery-small test 9: pause bulk on OST (bug 1420) === 03:27:17 (1786778837) [ 637.586988] Lustre: DEBUG MARKER: == recovery-small test 10a: finish request on server after client eviction (bug 1521) ========================================================== 03:27:33 (1786778853) [ 638.616436] Lustre: *** cfs_fail_loc=305, val=0*** [ 653.281103] Lustre: lustre-OST0001-osc-ffff96849059c800: disconnect after 20s idle [ 654.799390] Lustre: *** cfs_fail_loc=305, val=0*** [ 670.191806] Lustre: *** cfs_fail_loc=305, val=0*** [ 686.532567] Lustre: *** cfs_fail_loc=305, val=0*** [ 702.969481] Lustre: *** cfs_fail_loc=305, val=0*** [ 718.311158] Lustre: *** cfs_fail_loc=305, val=0*** [ 734.695983] Lustre: *** cfs_fail_loc=305, val=0*** [ 750.427755] LustreError: lustre-MDT0000-mdc-ffff96849059c800: operation ldlm_enqueue to node 192.168.202.101@tcp failed: rc = -107 [ 750.436573] Lustre: lustre-MDT0000-mdc-ffff96849059c800: Connection to lustre-MDT0000 (at 192.168.202.101@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 750.442494] Lustre: Skipped 2 previous similar messages [ 750.448841] LustreError: lustre-MDT0000-mdc-ffff96849059c800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 750.457388] LustreError: 13970:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff96849059c800: namespace resource [0x200000007:0x1:0x0].0x0 (ffff968484d1dc00) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 750.493795] Lustre: lustre-MDT0000-mdc-ffff96849059c800: Connection restored to 192.168.202.101@tcp (at 192.168.202.101@tcp) [ 750.508141] Lustre: Skipped 2 previous similar messages [ 757.984228] Lustre: DEBUG MARKER: == recovery-small test 10b: re-send BL AST =============== 03:29:34 (1786778974) [ 781.963407] Lustre: DEBUG MARKER: == recovery-small test 10c: re-send BL AST vs reconnect race (LU-5569) ========================================================== 03:29:57 (1786778997) [ 782.415566] Lustre: *** cfs_fail_loc=305, val=0*** [ 782.417509] Lustre: Skipped 1 previous similar message [ 789.877594] Lustre: DEBUG MARKER: == recovery-small test 10d: test failed blocking ast ===== 03:30:06 (1786779006) [ 793.215822] Lustre: Unmounted lustre-client [ 793.706575] Lustre: Mounted lustre-client - version 2.17.57_1_g05abe65 [ 795.835526] LustreError: lustre-OST0000-osc-ffff968488056000: operation ldlm_enqueue to node 192.168.202.101@tcp failed: rc = -107 [ 795.860144] LustreError: lustre-OST0000-osc-ffff968488056000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 795.882281] Lustre: 2355:0:(llite_lib.c:4340:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.202.101@tcp:/lustre/fid: [0x200000403:0x1:0x0]/ may get corrupted (rc -108) [ 798.172373] Lustre: Unmounted lustre-client [ 804.488707] Lustre: DEBUG MARKER: == recovery-small test 10e: re-send BL AST vs reconnect race 2 ========================================================== 03:30:20 (1786779020) [ 806.136568] Lustre: DEBUG MARKER: SKIP: recovery-small test_10e need two clients [ 807.533464] Lustre: DEBUG MARKER: == recovery-small test 11: wake up a thread waiting for completion after eviction (b=2460) ========================================================== 03:30:23 (1786779023) [ 835.184644] Lustre: DEBUG MARKER: == recovery-small test 12: recover from timed out resend in ptlrpcd (b=2494) ========================================================== 03:30:51 (1786779051) [ 835.292464] Lustre: DEBUG MARKER: multiop /mnt/lustre/f12.recovery-small OS_c [ 852.963493] Lustre: 17644:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786779054/real 1786779054] req@ffff96849855b480 x1873572811355264/t0(0) o35->lustre-MDT0000-mdc-ffff968488056000@192.168.202.101@tcp:23/10 lens 392/624 e 0 to 1 dl 1786779070 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'multiop.0' uid:0 gid:0 projid:0 [ 852.977233] Lustre: 17644:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 879.223815] Lustre: DEBUG MARKER: == recovery-small test 13: mdc_readpage restart test (bug 1138) ========================================================== 03:31:35 (1786779095) [ 908.216723] Lustre: DEBUG MARKER: == recovery-small test 14: mdc_readpage resend test (bug 1138) ========================================================== 03:32:04 (1786779124) [ 918.328313] Lustre: DEBUG MARKER: == recovery-small test 15: failed open (-ENOMEM) ========= 03:32:14 (1786779134) [ 927.816879] Lustre: DEBUG MARKER: == recovery-small test 16: timeout bulk put, don't evict client (2732) ========================================================== 03:32:23 (1786779143) [ 972.852752] Lustre: DEBUG MARKER: == recovery-small test 17a: timeout bulk get, don't evict client (2732) ========================================================== 03:33:09 (1786779189) [ 1026.905493] Lustre: DEBUG MARKER: == recovery-small test 17b: timeout bulk get, dont evict client (3582) ========================================================== 03:34:02 (1786779242) [ 1028.710881] Lustre: DEBUG MARKER: SKIP: recovery-small test_17b Needs multiple clients [ 1030.795416] Lustre: DEBUG MARKER: == recovery-small test 18a: manual ost invalidate clears page cache immediately ========================================================== 03:34:06 (1786779246) [ 1031.776585] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 1031.857060] Lustre: lustre-OST0001-osc-ffff968488056000: Connection to lustre-OST0001 (at 192.168.202.101@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 1031.869383] Lustre: Skipped 5 previous similar messages [ 1031.885128] LustreError: lustre-OST0001-osc-ffff968488056000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 1031.898980] Lustre: lustre-OST0001-osc-ffff968488056000: Connection restored to 192.168.202.101@tcp (at 192.168.202.101@tcp) [ 1031.917651] Lustre: Skipped 4 previous similar messages [ 1041.054597] Lustre: DEBUG MARKER: == recovery-small test 18b: eviction and reconnect clears page cache (2766) ========================================================== 03:34:16 (1786779256) [ 1047.537489] LustreError: lustre-OST0000-osc-ffff968488056000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 1071.848433] Lustre: DEBUG MARKER: == recovery-small test 18c: Dropped connect reply after eviction handing (14755) ========================================================== 03:34:47 (1786779287) [ 1075.112966] LustreError: lustre-OST0000-osc-ffff968488056000: operation ost_statfs to node 192.168.202.101@tcp failed: rc = -107 [ 1085.436733] Lustre: Evicted from lustre-OST0000_UUID (at 192.168.202.101@tcp) after server handle changed from 0x36ec956a3685bb8 to 0x36ec956a3685d08 [ 1085.479175] LustreError: lustre-OST0000-osc-ffff968488056000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 1094.831626] Lustre: DEBUG MARKER: == recovery-small test 19a: test expired_lock_main on mds (2867) ========================================================== 03:35:10 (1786779310) [ 1095.222470] Lustre: Mounted lustre-client - version 2.17.57_1_g05abe65 [ 1095.226136] Lustre: Skipped 1 previous similar message [ 1112.543380] Lustre: 2353:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786779314/real 1786779314] req@ffff968498f66a00 x1873572811413120/t0(0) o103->lustre-MDT0000-mdc-ffff968488056000@192.168.202.101@tcp:17/18 lens 328/224 e 0 to 1 dl 1786779330 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 [ 1112.565915] Lustre: 2353:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 1200.786206] LustreError: lustre-MDT0000-mdc-ffff968488056000: operation ldlm_enqueue to node 192.168.202.101@tcp failed: rc = -107 [ 1200.796506] LustreError: lustre-MDT0000-mdc-ffff968488056000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 1200.802445] LustreError: 23890:0:(file.c:6156:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5 [ 1202.362473] Lustre: Unmounted lustre-client [ 1207.612811] Lustre: DEBUG MARKER: == recovery-small test 19b: test expired_lock_main on ost (2867) ========================================================== 03:37:04 (1786779424) [ 1208.129947] Lustre: Mounted lustre-client - version 2.17.57_1_g05abe65 [ 1316.346093] LustreError: lustre-OST0000-osc-ffff968488056000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 1317.027893] Lustre: Unmounted lustre-client [ 1323.826795] Lustre: DEBUG MARKER: == recovery-small test 19c: check reconnect and lock resend do not trigger expired_lock_main ========================================================== 03:39:00 (1786779540) [ 1324.341174] Lustre: Mounted lustre-client - version 2.17.57_1_g05abe65 [ 1328.716030] Lustre: Unmounted lustre-client [ 1340.580369] Lustre: DEBUG MARKER: == recovery-small test 20a: ldlm_handle_enqueue error (should return error) ========================================================== 03:39:16 (1786779556) [ 1342.115097] LustreError: lustre-OST0000-osc-ffff968488056000: operation ldlm_enqueue to node 192.168.202.101@tcp failed: rc = -12 [ 1348.000633] Lustre: DEBUG MARKER: == recovery-small test 20b: ldlm_handle_enqueue error (should return error) ========================================================== 03:39:24 (1786779564) [ 1355.559576] Lustre: DEBUG MARKER: == recovery-small test 21a: drop close request while close and open are both in flight ========================================================== 03:39:31 (1786779571) [ 1383.369573] Lustre: DEBUG MARKER: == recovery-small test 21b: drop open request while close and open are both in flight ========================================================== 03:39:59 (1786779599) [ 1529.796904] Lustre: DEBUG MARKER: == recovery-small test 21c: drop both request while close and open are both in flight ========================================================== 03:42:26 (1786779746) [ 1552.361807] Lustre: lustre-MDT0000-mdc-ffff968488056000: Connection to lustre-MDT0000 (at 192.168.202.101@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 1552.377167] Lustre: Skipped 18 previous similar messages [ 1552.398917] Lustre: lustre-MDT0000-mdc-ffff968488056000: Connection restored to 192.168.202.101@tcp (at 192.168.202.101@tcp) [ 1552.418710] Lustre: Skipped 16 previous similar messages [ 1561.667816] Lustre: DEBUG MARKER: == recovery-small test 21d: drop close reply while close and open are both in flight ========================================================== 03:42:57 (1786779777) [ 1593.397238] Lustre: DEBUG MARKER: == recovery-small test 21e: drop open reply while close and open are both in flight ========================================================== 03:43:29 (1786779809) [ 1732.575203] Lustre: 29644:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786779812/real 1786779812] req@ffff968498559f80 x1873572811522944/t0(0) o36->lustre-MDT0000-mdc-ffff968488056000@192.168.202.101@tcp:12/10 lens 488/512 e 0 to 1 dl 1786779950 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:4294967295 [ 1732.614204] Lustre: 29644:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 15 previous similar messages [ 1734.708695] Lustre: DEBUG MARKER: == recovery-small test 21f: drop both reply while close and open are both in flight ========================================================== 03:45:50 (1786779950) [ 1761.104668] Lustre: DEBUG MARKER: == recovery-small test 21g: drop open reply and close request while close and open are both in flight ========================================================== 03:46:17 (1786779977) [ 1787.738444] Lustre: DEBUG MARKER: == recovery-small test 21h: drop open request and close reply while close and open are both in flight ========================================================== 03:46:43 (1786780003) [ 1814.832335] Lustre: DEBUG MARKER: == recovery-small test 22: drop close request and do mknod ========================================================== 03:47:11 (1786780031) [ 1840.659333] Lustre: DEBUG MARKER: == recovery-small test 23: client hang when close a file after mds crash ========================================================== 03:47:36 (1786780056) [ 1867.743653] LustreError: MGC192.168.202.101@tcp: Connection to MGS (at 192.168.202.101@tcp) was lost; in progress operations using this service will fail [ 1877.990792] Lustre: Evicted from MGS (at 192.168.202.101@tcp) after server handle changed from 0x36ec956a36852b2 to 0x36ec956a3687795 [ 1893.723396] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1894.885437] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1903.251432] Lustre: DEBUG MARKER: == recovery-small test 24a: fsync error (should return error) ========================================================== 03:48:39 (1786780119) [ 1905.421171] LustreError: lustre-OST0000-osc-ffff968488056000: operation ost_write to node 192.168.202.101@tcp failed: rc = -107 [ 1905.433575] LustreError: Skipped 1 previous similar message [ 1905.473540] LustreError: lustre-OST0000-osc-ffff968488056000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 1905.511251] Lustre: 2355:0:(llite_lib.c:4340:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.202.101@tcp:/lustre/fid: [0x200000404:0x30:0x0]// may get corrupted (rc -5) [ 1913.315923] Lustre: DEBUG MARKER: == recovery-small test 24b: test dirty page discard due to client eviction ========================================================== 03:48:49 (1786780129) [ 1915.083896] LustreError: lustre-OST0000-osc-ffff968488056000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 1915.127536] Lustre: 2354:0:(llite_lib.c:4340:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.202.101@tcp:/lustre/fid: [0x200000404:0x35:0x0]// may get corrupted (rc -108) [ 1922.534656] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 03:48:58 (1786780138) [ 1925.217947] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node [ 1927.203415] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 03:49:03 (1786780143) [ 1928.934494] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node [ 1930.965753] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 03:49:06 (1786780146) [ 1934.726567] LustreError: lustre-MDT0000-mdc-ffff968488056000: operation ldlm_enqueue to node 192.168.202.101@tcp failed: rc = -19 [ 1934.743905] LustreError: Skipped 2 previous similar messages [ 1953.772120] LustreError: MGC192.168.202.101@tcp: Connection to MGS (at 192.168.202.101@tcp) was lost; in progress operations using this service will fail [ 1953.782076] BUG: spinlock recursion on CPU#0, ptlrpcd_rcv/2352 [ 1953.792677] lock: 0xffff96848811cae0, .magic: dead4ead, .owner: ptlrpcd_rcv/2352, .owner_cpu: 0 [ 1953.798221] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O -------- - - 4.18.0rh8.10-debug #2 [ 1953.805273] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 1953.811465] Call Trace: [ 1953.813679] ? dump_stack+0xbb/0x10e [ 1953.817152] ? spin_dump.cold.3+0x32/0x65 [ 1953.818460] ? do_raw_spin_lock+0x10f/0x190 [ 1953.824136] ? _raw_spin_lock+0x21/0x30 [ 1953.825388] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 1953.828295] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 1953.831069] ? mdc_commit_open+0x91/0xb0 [mdc] [ 1953.833036] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 1953.838118] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 1953.843755] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 1953.853966] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 1953.860429] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 1953.865323] ? do_wait_intr+0xf0/0xf0 [ 1953.867317] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 1953.879793] ? kthread+0x1d1/0x200 [ 1953.880953] ? set_kthread_struct+0x70/0x70 [ 1953.882350] ? ret_from_fork+0x1f/0x30 [ 1980.055009] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [ 1980.057909] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 1980.058991] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [ 1980.059049] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 1980.059055] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 1980.059117] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O -------- - - 4.18.0rh8.10-debug #2 [ 1980.059123] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 1980.059130] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 1980.059434] RIP: 0010:native_safe_halt+0x2e/0x50 [ 1980.059446] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 1980.059450] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 1980.059454] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 1980.059457] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 1980.059459] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 1980.059460] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 1980.059463] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 1980.059466] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 1980.059469] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1980.059471] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 1980.059480] Call Trace: [ 1980.059486] [ 1980.059491] ? show_regs.cold.9+0x22/0x2f [ 1980.059503] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 1980.059842] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 1980.059847] ? watchdog+0x40/0x40 [ 1980.059852] ? hrtimer_interrupt+0x128/0x2c0 [ 1980.059856] ? smp_apic_timer_interrupt+0xa6/0x270 [ 1980.059865] ? apic_timer_interrupt+0xf/0x20 [ 1980.059869] [ 1980.059872] ? native_safe_halt+0x2e/0x50 [ 1980.059876] kvm_wait+0xcd/0xe0 [ 1980.059886] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 1980.059895] do_raw_spin_lock+0x177/0x190 [ 1980.059900] _raw_spin_lock+0x21/0x30 [ 1980.059905] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 1980.060133] process_one_work+0x2c8/0x700 [ 1980.060140] worker_thread+0x296/0x6e0 [ 1980.060144] ? rescuer_thread+0x570/0x570 [ 1980.060148] kthread+0x1d1/0x200 [ 1980.060153] ? set_kthread_struct+0x70/0x70 [ 1980.060157] ret_from_fork+0x1f/0x30 [ 1980.060445] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [ 1980.060451] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 1980.060518] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 1980.060522] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 1980.060524] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 1980.060536] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 1980.060541] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 1980.060545] RAX: 0000000000007e8e RBX: ffff96848811cae0 RCX: 0000000000000001 [ 1980.060547] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 1980.060550] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 1980.060552] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 1980.060554] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 1980.060558] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 1980.060561] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1980.060563] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 1980.060571] Call Trace: [ 1980.060574] [ 1980.060577] ? show_regs.cold.9+0x22/0x2f [ 1980.060586] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 1980.060591] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 1980.060596] ? watchdog+0x40/0x40 [ 1980.060600] ? hrtimer_interrupt+0x128/0x2c0 [ 1980.060605] ? smp_apic_timer_interrupt+0xa6/0x270 [ 1980.060611] ? apic_timer_interrupt+0xf/0x20 [ 1980.060617] [ 1980.060619] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 1980.060625] do_raw_spin_lock+0x177/0x190 [ 1980.060631] _raw_spin_lock+0x21/0x30 [ 1980.060635] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 1980.061215] ? do_raw_spin_unlock+0x75/0x190 [ 1980.061221] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 1980.061382] ? do_wait_intr+0xf0/0xf0 [ 1980.061390] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 1980.061514] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 1980.061676] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 1980.061830] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 1980.061937] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 1980.061968] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 1980.061987] mdc_intent_lock+0x353/0x7e0 [mdc] [ 1980.062004] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 1980.063311] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 1980.063612] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 1980.063655] lmv_intent_open+0x338/0x1150 [lmv] [ 1980.063691] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 1980.066083] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 1980.066167] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 1980.066198] ll_intent_lock+0xc5/0xaa0 [lustre] [ 1980.066295] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 1980.066350] ll_lookup_it+0xe0e/0x2270 [lustre] [ 1980.066404] ? do_raw_spin_unlock+0x75/0x190 [ 1980.066411] ? _raw_spin_unlock+0x12/0x30 [ 1980.066417] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 1980.066438] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 1980.066451] ll_atomic_open+0x36a/0xd70 [lustre] [ 1980.066506] lookup_open+0x356/0xb40 [ 1980.066513] do_last+0x313/0xfc0 [ 1980.066517] ? nd_jump_root+0xe5/0x160 [ 1980.066520] ? path_init+0x437/0x520 [ 1980.066523] path_openat+0xf7/0x500 [ 1980.066527] ? do_raw_spin_unlock+0x75/0x190 [ 1980.066533] do_filp_open+0x99/0x140 [ 1980.066539] ? getname_flags+0x6e/0x330 [ 1980.066544] ? __check_object_size+0xff/0x256 [ 1980.066552] ? do_raw_spin_unlock+0x75/0x190 [ 1980.066556] ? _raw_spin_unlock+0x12/0x30 [ 1980.066561] do_sys_openat2+0x2b4/0x410 [ 1980.066565] do_sys_open+0x73/0xa0 [ 1980.066570] __x64_sys_openat+0x24/0x30 [ 1980.066573] do_syscall_64+0xc1/0x440 [ 1980.066580] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 1980.066587] RIP: 0033:0x7ff54af45332 [ 1980.066595] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 1980.066599] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 1980.066604] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 1980.066606] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 1980.066608] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 1980.066610] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 1980.066612] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 1980.078469] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 1980.078482] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 1980.078485] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 1980.079259] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 1980.079268] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 1980.079274] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 1980.079277] RDX: 0000000000003e4a RSI: 0000000000000003 RDI: ffff96848811cae0 [ 1980.079279] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 1980.079282] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 1980.079284] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 1980.079287] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 1980.079289] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1980.079291] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 1980.079299] Call Trace: [ 1980.079304] [ 1980.079309] ? show_regs.cold.9+0x22/0x2f [ 1980.079320] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 1980.079325] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 1980.079328] ? watchdog+0x40/0x40 [ 1980.079331] ? hrtimer_interrupt+0x128/0x2c0 [ 1980.079333] ? smp_apic_timer_interrupt+0xa6/0x270 [ 1980.079341] ? apic_timer_interrupt+0xf/0x20 [ 1980.079345] [ 1980.079347] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 1980.079350] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 1980.079353] do_raw_spin_lock+0x177/0x190 [ 1980.079356] _raw_spin_lock+0x21/0x30 [ 1980.079362] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 1980.079774] obd_mod_free+0x46/0x140 [ptlrpc] [ 1980.079907] mdc_commit_open+0x91/0xb0 [mdc] [ 1980.079935] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 1980.080112] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 1980.080325] after_reply+0xe2c/0x1780 [ptlrpc] [ 1980.080473] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 1980.080633] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 1980.080804] ? do_wait_intr+0xf0/0xf0 [ 1980.080812] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 1980.080947] kthread+0x1d1/0x200 [ 1980.080952] ? set_kthread_struct+0x70/0x70 [ 1980.080955] ret_from_fork+0x1f/0x30 [ 1980.542741] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 1980.573114] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 1980.579355] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 1980.587560] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 1980.595834] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 1980.609896] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 1980.612838] RAX: 000000000000549c RBX: ffff96848811cae0 RCX: 0000000000000001 [ 1980.616531] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 1980.620184] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 1980.630732] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 1980.638808] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 1980.645511] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 1980.650900] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1980.653742] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 1980.657252] Call Trace: [ 1980.658352] [ 1980.659298] ? show_regs.cold.9+0x22/0x2f [ 1980.661073] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 1980.666304] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 1980.669345] ? watchdog+0x40/0x40 [ 1980.671537] ? hrtimer_interrupt+0x128/0x2c0 [ 1980.673692] ? smp_apic_timer_interrupt+0xa6/0x270 [ 1980.675726] ? apic_timer_interrupt+0xf/0x20 [ 1980.679356] [ 1980.680708] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 1980.683851] do_raw_spin_lock+0x177/0x190 [ 1980.686510] _raw_spin_lock+0x21/0x30 [ 1980.688614] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 1980.691793] ? do_raw_spin_unlock+0x75/0x190 [ 1980.694598] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 1980.698463] ? do_wait_intr+0xf0/0xf0 [ 1980.700568] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 1980.702448] mdc_close+0x53b/0xd70 [mdc] [ 1980.704469] lmv_close+0x18d/0x490 [lmv] [ 1980.706391] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 1980.709626] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 1980.712609] ll_md_real_close+0xa5/0x2a0 [lustre] [ 1980.715085] ll_md_close+0x274/0x800 [lustre] [ 1980.717604] ll_file_release+0x128/0xc50 [lustre] [ 1980.721785] __fput+0x10d/0x440 [ 1980.724683] ____fput+0x12/0x20 [ 1980.727130] task_work_run+0xc6/0x110 [ 1980.731898] exit_to_usermode_loop+0x1dd/0x1f0 [ 1980.735026] do_syscall_64+0x3d6/0x440 [ 1980.738720] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 1980.742827] RIP: 0033:0x7ff54af45cd8 [ 1980.747323] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 1980.763926] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 1980.771496] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 1980.778581] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 1980.784855] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 1980.791893] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 1980.794601] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2001.384423] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 42s! [ 2001.393350] Showing busy workqueues and worker pools: [ 2001.398967] workqueue events: flags=0x0 [ 2001.403171] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2001.409893] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 2001.415552] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2001.421445] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4] [ 2001.431475] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2001.442982] pending: vmstat_shepherd [ 2001.448961] workqueue events_long: flags=0x0 [ 2001.450606] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2001.465076] pending: lu_objects_flush [obdclass] [ 2001.471196] workqueue events_unbound: flags=0x2 [ 2001.478638] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/512 refcnt=3 [ 2001.482660] pending: flush_memcg_stats_dwork [ 2001.484842] workqueue events_power_efficient: flags=0x80 [ 2001.487388] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2001.492508] pending: neigh_periodic_work [ 2001.495949] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2001.498917] pending: do_cache_clean, check_lifetime [ 2001.502148] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2001.507410] pending: neigh_periodic_work [ 2001.509429] workqueue writeback: flags=0x4a [ 2001.515298] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2001.518209] pending: wb_workfn, wb_workfn [ 2001.523950] workqueue ipv6_addrconf: flags=0x40008 [ 2001.529866] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2001.537630] pending: addrconf_verify_work [ 2001.551869] workqueue knbd0-recv: flags=0x1a [ 2001.560598] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2001.575119] in-flight: 657:recv_work [ 2001.579792] workqueue ptlrpc_pinger: flags=0x2 [ 2001.585872] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2001.588378] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 2001.627054] pool 8: cpus=0-3 flags=0x4 nice=0 hung=42s workers=4 idle: 20831 12642 28859 [ 2001.650109] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=1991s workers=2 idle: 659 [ 2008.058017] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [ 2008.059031] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 2008.060911] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [ 2008.060922] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2008.061003] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2008.061009] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2008.061011] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2008.061027] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2008.061032] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2008.061036] RAX: 0000000000006216 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2008.061039] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 2008.061040] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2008.061043] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 2008.061045] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 2008.061049] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 2008.061052] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2008.061054] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 2008.061062] Call Trace: [ 2008.061067] [ 2008.061072] ? show_regs.cold.9+0x22/0x2f [ 2008.061086] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2008.061094] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2008.061098] ? watchdog+0x40/0x40 [ 2008.061102] ? hrtimer_interrupt+0x128/0x2c0 [ 2008.061106] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2008.061117] ? apic_timer_interrupt+0xf/0x20 [ 2008.061122] [ 2008.061125] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2008.061131] do_raw_spin_lock+0x177/0x190 [ 2008.061136] _raw_spin_lock+0x21/0x30 [ 2008.061141] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2008.062532] Modules linked in: lustre(O) [ 2008.062533] ? do_raw_spin_unlock+0x75/0x190 [ 2008.062542] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2008.062613] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2008.062618] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2008.062621] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2008.062630] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2008.062634] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2008.062639] RAX: 0000000000000588 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2008.062641] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 2008.062643] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2008.062646] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 2008.062648] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 2008.062650] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 2008.062653] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2008.062655] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 2008.062662] Call Trace: [ 2008.062665] [ 2008.062668] ? show_regs.cold.9+0x22/0x2f [ 2008.062685] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2008.062691] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2008.062696] ? watchdog+0x40/0x40 [ 2008.062701] ? hrtimer_interrupt+0x128/0x2c0 [ 2008.062705] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2008.062714] ? apic_timer_interrupt+0xf/0x20 [ 2008.062718] [ 2008.062720] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2008.062727] do_raw_spin_lock+0x177/0x190 [ 2008.062731] _raw_spin_lock+0x21/0x30 [ 2008.062546] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2008.062736] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2008.063142] ? do_raw_spin_unlock+0x75/0x190 [ 2008.063148] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2008.063496] ? do_wait_intr+0xf0/0xf0 [ 2008.063505] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2008.063952] ? do_wait_intr+0xf0/0xf0 [ 2008.064109] mdc_close+0x53b/0xd70 [mdc] [ 2008.064151] lmv_close+0x18d/0x490 [lmv] [ 2008.064178] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 2008.064354] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2008.064471] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2008.064555] ll_md_close+0x274/0x800 [lustre] [ 2008.064732] ll_file_release+0x128/0xc50 [lustre] [ 2008.064829] __fput+0x10d/0x440 [ 2008.064840] ____fput+0x12/0x20 [ 2008.064844] task_work_run+0xc6/0x110 [ 2008.064853] exit_to_usermode_loop+0x1dd/0x1f0 [ 2008.064860] do_syscall_64+0x3d6/0x440 [ 2008.064865] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2008.064943] RIP: 0033:0x7ff54af45cd8 [ 2008.064951] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2008.064955] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2008.064970] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 2008.064972] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 2008.064974] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 2008.064976] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 2008.064978] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2008.063974] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2008.077256] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2008.077336] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2008.077342] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2008.077349] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2008.088385] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2008.110029] [ 2008.110040] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2008.110073] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2008.110078] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2008.110083] RAX: 0000000000006018 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2008.110085] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 2008.110088] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 2008.110090] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 2008.110092] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 2008.110096] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 2008.110099] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2008.110101] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 2008.110109] Call Trace: [ 2008.110115] [ 2008.110121] ? show_regs.cold.9+0x22/0x2f [ 2008.110133] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2008.110140] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2008.110145] ? watchdog+0x40/0x40 [ 2008.110149] ? hrtimer_interrupt+0x128/0x2c0 [ 2008.110154] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2008.110165] ? apic_timer_interrupt+0xf/0x20 [ 2008.110170] [ 2008.110174] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2008.110179] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2008.110185] do_raw_spin_lock+0x177/0x190 [ 2008.110190] _raw_spin_lock+0x21/0x30 [ 2008.110194] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2008.110586] process_one_work+0x2c8/0x700 [ 2008.110594] worker_thread+0x296/0x6e0 [ 2008.110599] ? rescuer_thread+0x570/0x570 [ 2008.110602] kthread+0x1d1/0x200 [ 2008.110608] ? set_kthread_struct+0x70/0x70 [ 2008.110611] ret_from_fork+0x1f/0x30 [ 2008.990414] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2008.995457] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2008.998018] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2009.000203] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2009.002103] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2009.004532] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2009.007468] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2009.010895] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2009.014569] lmv_intent_open+0x338/0x1150 [lmv] [ 2009.018657] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2009.023618] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2009.028970] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2009.032304] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2009.034921] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2009.041469] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2009.048785] ? do_raw_spin_unlock+0x75/0x190 [ 2009.052816] ? _raw_spin_unlock+0x12/0x30 [ 2009.057497] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2009.060154] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2009.067632] ll_atomic_open+0x36a/0xd70 [lustre] [ 2009.072440] lookup_open+0x356/0xb40 [ 2009.074649] do_last+0x313/0xfc0 [ 2009.077554] ? nd_jump_root+0xe5/0x160 [ 2009.079527] ? path_init+0x437/0x520 [ 2009.081031] path_openat+0xf7/0x500 [ 2009.083019] ? do_raw_spin_unlock+0x75/0x190 [ 2009.085042] do_filp_open+0x99/0x140 [ 2009.086138] ? getname_flags+0x6e/0x330 [ 2009.087974] ? __check_object_size+0xff/0x256 [ 2009.089227] ? do_raw_spin_unlock+0x75/0x190 [ 2009.090908] ? _raw_spin_unlock+0x12/0x30 [ 2009.092032] do_sys_openat2+0x2b4/0x410 [ 2009.093089] do_sys_open+0x73/0xa0 [ 2009.093960] __x64_sys_openat+0x24/0x30 [ 2009.095536] do_syscall_64+0xc1/0x440 [ 2009.097289] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2009.099537] RIP: 0033:0x7ff54af45332 [ 2009.101650] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2009.111259] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2009.115203] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 2009.125778] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 2009.133193] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 2009.138255] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 2009.144326] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 2013.840116] rcu: INFO: rcu_sched self-detected stall on CPU [ 2013.846079] rcu: 0-...!: (30756 ticks this GP) idle=98a/1/0x4000000000000002 softirq=125154/125154 fqs=959 [ 2013.854976] (t=60000 jiffies g=253585 q=2690) [ 2013.858108] rcu: rcu_sched kthread starved for 54960 jiffies! g253585 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [ 2013.864609] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 2013.870285] rcu: RCU grace-period kthread stack dump: [ 2013.874548] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 2013.880465] Call Trace: [ 2013.881361] __schedule+0x351/0xcb0 [ 2013.884545] schedule+0xc0/0x180 [ 2013.888515] schedule_timeout+0xb4/0x190 [ 2013.891092] ? do_raw_spin_unlock+0x75/0x190 [ 2013.894113] ? __next_timer_interrupt+0x160/0x160 [ 2013.895999] ? prepare_to_swait_event+0xf3/0x140 [ 2013.898665] rcu_gp_kthread+0x8dd/0x15b0 [ 2013.901134] ? force_qs_rnp+0x2c0/0x2c0 [ 2013.904592] kthread+0x1d1/0x200 [ 2013.907581] ? set_kthread_struct+0x70/0x70 [ 2013.910171] ret_from_fork+0x1f/0x30 [ 2013.914308] rcu: Stack dump where RCU GP kthread last ran: [ 2013.917189] Sending NMI from CPU 0 to CPUs 2: [ 2013.920703] NMI backtrace for cpu 2 skipped: idling at native_safe_halt+0x2e/0x50 [ 2013.924514] NMI backtrace for cpu 0 [ 2013.927862] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2013.936893] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2013.943147] Call Trace: [ 2013.944915] [ 2013.946128] ? dump_stack+0xbb/0x10e [ 2013.948278] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 2013.951082] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 2013.954400] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 2013.956959] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 2013.959922] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 2013.964185] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 2013.966469] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 2013.969953] ? update_process_times+0x6d/0xc0 [ 2013.972623] ? tick_sched_handle.isra.18+0x27/0x90 [ 2013.975500] ? tick_sched_timer+0x53/0xb0 [ 2013.977499] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2013.979980] ? tick_sched_handle.isra.18+0x90/0x90 [ 2013.982271] ? hrtimer_interrupt+0x128/0x2c0 [ 2013.984176] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2013.987118] ? apic_timer_interrupt+0xf/0x20 [ 2013.989427] [ 2013.991133] ? native_safe_halt+0x2e/0x50 [ 2013.993376] ? kvm_wait+0xcd/0xe0 [ 2013.996116] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2014.000169] ? do_raw_spin_lock+0x177/0x190 [ 2014.002383] ? _raw_spin_lock+0x21/0x30 [ 2014.005036] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2014.007515] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 2014.010590] ? mdc_commit_open+0x91/0xb0 [mdc] [ 2014.015534] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2014.018404] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2014.022113] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 2014.024948] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2014.028092] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2014.030787] ? do_wait_intr+0xf0/0xf0 [ 2014.032698] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2014.036159] ? kthread+0x1d1/0x200 [ 2014.038404] ? set_kthread_struct+0x70/0x70 [ 2014.041145] ? ret_from_fork+0x1f/0x30 [ 2032.097369] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 73s! [ 2032.104520] Showing busy workqueues and worker pools: [ 2032.111837] workqueue events: flags=0x0 [ 2032.113077] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2032.118949] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 2032.127564] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2032.136496] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc] [ 2032.145568] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2032.149474] pending: vmstat_shepherd [ 2032.156459] workqueue events_long: flags=0x0 [ 2032.161417] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2032.166594] pending: lu_objects_flush [obdclass] [ 2032.172064] workqueue events_unbound: flags=0x2 [ 2032.175050] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/512 refcnt=3 [ 2032.178793] pending: flush_memcg_stats_dwork [ 2032.180848] workqueue events_power_efficient: flags=0x80 [ 2032.183638] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2032.186874] pending: neigh_periodic_work [ 2032.188769] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2032.191927] pending: do_cache_clean, check_lifetime [ 2032.194317] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2032.197338] pending: neigh_periodic_work [ 2032.199507] workqueue writeback: flags=0x4a [ 2032.201334] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2032.204864] pending: wb_workfn, wb_workfn [ 2032.206893] workqueue ipv6_addrconf: flags=0x40008 [ 2032.209227] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2032.212310] pending: addrconf_verify_work [ 2032.214514] workqueue knbd0-recv: flags=0x1a [ 2032.216745] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2032.219345] in-flight: 657:recv_work [ 2032.221266] workqueue ptlrpc_pinger: flags=0x2 [ 2032.223014] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2032.225941] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 2032.228533] pool 8: cpus=0-3 flags=0x4 nice=0 hung=73s workers=4 idle: 20831 12642 28859 [ 2032.232451] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2022s workers=2 idle: 659 [ 2036.060656] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [ 2036.064117] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [ 2036.071535] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2036.071730] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2036.071738] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2036.071741] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2036.071756] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2036.071760] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2036.071764] RAX: 0000000000000411 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2036.071767] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 2036.071769] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2036.071770] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 2036.071773] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 2036.071776] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 2036.071778] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2036.071781] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 2036.071789] Call Trace: [ 2036.071794] [ 2036.071800] ? show_regs.cold.9+0x22/0x2f [ 2036.071811] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2036.071818] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2036.071822] ? watchdog+0x40/0x40 [ 2036.071826] ? hrtimer_interrupt+0x128/0x2c0 [ 2036.071829] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2036.071841] ? apic_timer_interrupt+0xf/0x20 [ 2036.071846] [ 2036.071849] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2036.071856] do_raw_spin_lock+0x177/0x190 [ 2036.071861] _raw_spin_lock+0x21/0x30 [ 2036.071866] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2036.072191] ? do_raw_spin_unlock+0x75/0x190 [ 2036.072196] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2036.072411] ? do_wait_intr+0xf0/0xf0 [ 2036.072421] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2036.081437] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2036.082864] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2036.082873] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2036.082876] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2036.082892] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2036.082896] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2036.082901] RAX: 00000000000045c3 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2036.082904] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 2036.082906] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2036.082908] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 2036.082910] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 2036.082914] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 2036.082917] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2036.082919] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 2036.082929] Call Trace: [ 2036.082934] [ 2036.082941] ? show_regs.cold.9+0x22/0x2f [ 2036.082953] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2036.082960] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2036.082965] ? watchdog+0x40/0x40 [ 2036.082970] ? hrtimer_interrupt+0x128/0x2c0 [ 2036.082974] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2036.082985] ? apic_timer_interrupt+0xf/0x20 [ 2036.082990] [ 2036.082993] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2036.083000] do_raw_spin_lock+0x177/0x190 [ 2036.083004] _raw_spin_lock+0x21/0x30 [ 2036.083009] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2036.117172] mdc_close+0x53b/0xd70 [mdc] [ 2036.117227] lmv_close+0x18d/0x490 [lmv] [ 2036.117258] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 2036.117779] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2036.117888] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2036.117964] ll_md_close+0x274/0x800 [lustre] [ 2036.118056] ll_file_release+0x128/0xc50 [lustre] [ 2036.118136] __fput+0x10d/0x440 [ 2036.118147] ____fput+0x12/0x20 [ 2036.118152] task_work_run+0xc6/0x110 [ 2036.118160] exit_to_usermode_loop+0x1dd/0x1f0 [ 2036.118167] do_syscall_64+0x3d6/0x440 [ 2036.118172] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2036.118181] RIP: 0033:0x7ff54af45cd8 [ 2036.118188] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2036.118192] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2036.118197] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 2036.118199] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 2036.118201] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 2036.118202] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 2036.118205] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2036.891182] ? do_raw_spin_unlock+0x75/0x190 [ 2036.895264] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2036.901230] ? do_wait_intr+0xf0/0xf0 [ 2036.906135] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2036.912947] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2036.927268] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2036.934903] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2036.941878] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2036.949850] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2036.953409] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2036.959447] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2036.967520] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2036.981340] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2036.995870] lmv_intent_open+0x338/0x1150 [lmv] [ 2037.003817] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2037.008412] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2037.012867] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2037.014368] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2037.017426] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2037.029352] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2037.031172] ? do_raw_spin_unlock+0x75/0x190 [ 2037.036043] ? _raw_spin_unlock+0x12/0x30 [ 2037.039321] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2037.042414] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2037.046473] ll_atomic_open+0x36a/0xd70 [lustre] [ 2037.050407] lookup_open+0x356/0xb40 [ 2037.052896] do_last+0x313/0xfc0 [ 2037.055944] ? nd_jump_root+0xe5/0x160 [ 2037.058824] ? path_init+0x437/0x520 [ 2037.065855] path_openat+0xf7/0x500 [ 2037.068868] ? do_raw_spin_unlock+0x75/0x190 [ 2037.071903] do_filp_open+0x99/0x140 [ 2037.074818] ? getname_flags+0x6e/0x330 [ 2037.082756] ? __check_object_size+0xff/0x256 [ 2037.091670] ? do_raw_spin_unlock+0x75/0x190 [ 2037.095197] ? _raw_spin_unlock+0x12/0x30 [ 2037.100344] do_sys_openat2+0x2b4/0x410 [ 2037.102954] do_sys_open+0x73/0xa0 [ 2037.106865] __x64_sys_openat+0x24/0x30 [ 2037.110149] do_syscall_64+0xc1/0x440 [ 2037.112165] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2037.124390] RIP: 0033:0x7ff54af45332 [ 2037.136019] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2037.181645] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2037.193872] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 2037.201423] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 2037.205722] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 2037.220990] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 2037.231904] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 2040.059020] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 2040.066507] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2040.120854] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2040.143112] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2040.154853] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2040.163903] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2040.170237] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2040.194798] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2040.206700] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2040.216078] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 2040.223809] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 2040.227874] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 2040.233815] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 2040.244860] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 2040.254063] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2040.265253] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 2040.277757] Call Trace: [ 2040.279069] [ 2040.280340] ? show_regs.cold.9+0x22/0x2f [ 2040.285012] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2040.288642] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2040.293010] ? watchdog+0x40/0x40 [ 2040.301865] ? hrtimer_interrupt+0x128/0x2c0 [ 2040.306086] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2040.313914] ? apic_timer_interrupt+0xf/0x20 [ 2040.315250] [ 2040.320953] ? native_safe_halt+0x2e/0x50 [ 2040.323607] kvm_wait+0xcd/0xe0 [ 2040.326604] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2040.331841] do_raw_spin_lock+0x177/0x190 [ 2040.338272] _raw_spin_lock+0x21/0x30 [ 2040.339917] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2040.346834] process_one_work+0x2c8/0x700 [ 2040.350820] worker_thread+0x296/0x6e0 [ 2040.352826] ? rescuer_thread+0x570/0x570 [ 2040.358731] kthread+0x1d1/0x200 [ 2040.361183] ? set_kthread_struct+0x70/0x70 [ 2040.366261] ret_from_fork+0x1f/0x30 [ 2060.055049] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 2060.058705] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2060.084902] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2060.088914] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2060.095050] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2060.096004] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2060.100088] RSP: 0000:ffffaf9fc1b97bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2060.101808] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2060.104178] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 2060.106514] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 2060.109840] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 2060.112335] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2060.114051] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 2060.116629] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2060.118158] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 2060.120197] Call Trace: [ 2060.120837] [ 2060.121215] ? show_regs.cold.9+0x22/0x2f [ 2060.122205] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2060.123741] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2060.125040] ? watchdog+0x40/0x40 [ 2060.125916] ? hrtimer_interrupt+0x128/0x2c0 [ 2060.127038] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2060.128421] ? apic_timer_interrupt+0xf/0x20 [ 2060.129516] [ 2060.130025] ? native_safe_halt+0x2e/0x50 [ 2060.131041] kvm_wait+0xcd/0xe0 [ 2060.131960] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2060.133634] do_raw_spin_lock+0x177/0x190 [ 2060.134740] _raw_spin_lock+0x21/0x30 [ 2060.135723] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2060.137532] obd_mod_free+0x46/0x140 [ptlrpc] [ 2060.139018] mdc_commit_open+0x91/0xb0 [mdc] [ 2060.141151] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2060.143431] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2060.145154] after_reply+0xe2c/0x1780 [ptlrpc] [ 2060.147580] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2060.149583] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2060.151941] ? do_wait_intr+0xf0/0xf0 [ 2060.153967] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2060.157573] kthread+0x1d1/0x200 [ 2060.158967] ? set_kthread_struct+0x70/0x70 [ 2060.160820] ret_from_fork+0x1f/0x30 [ 2062.303044] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 103s! [ 2062.305576] Showing busy workqueues and worker pools: [ 2062.307698] workqueue events: flags=0x0 [ 2062.309626] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2062.312339] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 2062.319628] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 2062.325909] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 2062.335174] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2062.340044] pending: vmstat_shepherd [ 2062.347009] workqueue events_long: flags=0x0 [ 2062.351219] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2062.358562] pending: lu_objects_flush [obdclass] [ 2062.363759] workqueue events_unbound: flags=0x2 [ 2062.365196] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/512 refcnt=3 [ 2062.373124] pending: flush_memcg_stats_dwork [ 2062.375324] workqueue events_power_efficient: flags=0x80 [ 2062.377816] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2062.381670] pending: neigh_periodic_work [ 2062.384008] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2062.386999] pending: do_cache_clean, check_lifetime [ 2062.394342] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2062.403037] pending: neigh_periodic_work [ 2062.406074] workqueue writeback: flags=0x4a [ 2062.409492] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2062.412516] pending: wb_workfn, wb_workfn [ 2062.414596] workqueue ipv6_addrconf: flags=0x40008 [ 2062.416978] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2062.419690] pending: addrconf_verify_work [ 2062.421738] workqueue knbd0-recv: flags=0x1a [ 2062.423348] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2062.425757] in-flight: 657:recv_work [ 2062.427204] workqueue ptlrpc_pinger: flags=0x2 [ 2062.428727] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2062.431551] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 2062.434147] pool 8: cpus=0-3 flags=0x4 nice=0 hung=103s workers=4 idle: 20831 12642 28859 [ 2062.437447] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2052s workers=2 idle: 659 [ 2064.058023] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [ 2064.062341] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2064.063666] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [ 2064.099659] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2064.099672] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2064.099675] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2064.099690] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2064.099694] RSP: 0018:ffffaf9fc1543ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2064.099700] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2064.099703] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 2064.099704] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2064.099706] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 2064.099709] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 2064.099712] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 2064.099715] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2064.099717] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 2064.099726] Call Trace: [ 2064.099732] [ 2064.099738] ? show_regs.cold.9+0x22/0x2f [ 2064.099747] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2064.099754] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2064.099758] ? watchdog+0x40/0x40 [ 2064.099762] ? hrtimer_interrupt+0x128/0x2c0 [ 2064.099765] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2064.099772] ? apic_timer_interrupt+0xf/0x20 [ 2064.099777] [ 2064.099780] ? native_safe_halt+0x2e/0x50 [ 2064.099783] kvm_wait+0xcd/0xe0 [ 2064.099791] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2064.099813] do_raw_spin_lock+0x177/0x190 [ 2064.099818] _raw_spin_lock+0x21/0x30 [ 2064.099822] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2064.100394] ? do_raw_spin_unlock+0x75/0x190 [ 2064.100400] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2064.100618] ? do_wait_intr+0xf0/0xf0 [ 2064.100627] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2064.100843] mdc_close+0x53b/0xd70 [mdc] [ 2064.101027] lmv_close+0x18d/0x490 [lmv] [ 2064.101125] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 2064.101471] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2064.101588] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2064.101677] ll_md_close+0x274/0x800 [lustre] [ 2064.101992] ll_file_release+0x128/0xc50 [lustre] [ 2064.102150] __fput+0x10d/0x440 [ 2064.102161] ____fput+0x12/0x20 [ 2064.102166] task_work_run+0xc6/0x110 [ 2064.102235] exit_to_usermode_loop+0x1dd/0x1f0 [ 2064.102309] do_syscall_64+0x3d6/0x440 [ 2064.102314] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2064.102324] RIP: 0033:0x7ff54af45cd8 [ 2064.102332] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2064.102336] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2064.102342] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 2064.102344] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 2064.102346] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 2064.102349] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 2064.102351] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2064.511844] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2064.585602] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2064.594438] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2064.602756] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2064.604936] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2064.616183] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 2064.619174] RAX: 0000000000007d06 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2064.622587] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 2064.627149] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2064.631283] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 2064.635541] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 2064.639823] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 2064.644477] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2064.647980] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 2064.658802] Call Trace: [ 2064.659639] [ 2064.660547] ? show_regs.cold.9+0x22/0x2f [ 2064.669557] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2064.671271] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2064.677952] ? watchdog+0x40/0x40 [ 2064.679849] ? hrtimer_interrupt+0x128/0x2c0 [ 2064.682707] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2064.689694] ? apic_timer_interrupt+0xf/0x20 [ 2064.694478] [ 2064.695937] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2064.700514] do_raw_spin_lock+0x177/0x190 [ 2064.701978] _raw_spin_lock+0x21/0x30 [ 2064.704559] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2064.708587] ? do_raw_spin_unlock+0x75/0x190 [ 2064.711550] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2064.723468] ? do_wait_intr+0xf0/0xf0 [ 2064.725628] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2064.728899] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2064.732463] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2064.735562] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2064.736971] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2064.740486] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2064.743224] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2064.745908] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2064.750239] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2064.756481] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2064.758765] lmv_intent_open+0x338/0x1150 [lmv] [ 2064.760925] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2064.763180] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2064.766227] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2064.768447] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2064.769784] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2064.771621] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2064.773110] ? do_raw_spin_unlock+0x75/0x190 [ 2064.774484] ? _raw_spin_unlock+0x12/0x30 [ 2064.775940] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2064.777180] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2064.779199] ll_atomic_open+0x36a/0xd70 [lustre] [ 2064.780932] lookup_open+0x356/0xb40 [ 2064.781997] do_last+0x313/0xfc0 [ 2064.782975] ? nd_jump_root+0xe5/0x160 [ 2064.783952] ? path_init+0x437/0x520 [ 2064.784933] path_openat+0xf7/0x500 [ 2064.785699] ? do_raw_spin_unlock+0x75/0x190 [ 2064.786817] do_filp_open+0x99/0x140 [ 2064.787819] ? getname_flags+0x6e/0x330 [ 2064.788909] ? __check_object_size+0xff/0x256 [ 2064.790068] ? do_raw_spin_unlock+0x75/0x190 [ 2064.791950] ? _raw_spin_unlock+0x12/0x30 [ 2064.793119] do_sys_openat2+0x2b4/0x410 [ 2064.794147] do_sys_open+0x73/0xa0 [ 2064.795053] __x64_sys_openat+0x24/0x30 [ 2064.796317] do_syscall_64+0xc1/0x440 [ 2064.797224] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2064.798602] RIP: 0033:0x7ff54af45332 [ 2064.799489] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2064.805016] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2064.807373] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 2064.809476] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 2064.811192] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 2064.812903] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 2064.815076] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 2068.066503] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 2068.070003] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2068.130772] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2068.153812] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2068.170667] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2068.173008] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2068.182719] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2068.200108] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2068.210480] RAX: 0000000000007a2e RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2068.215690] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 2068.225899] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 2068.239636] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 2068.244993] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 2068.253012] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 2068.257320] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2068.266586] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 2068.276493] Call Trace: [ 2068.277504] [ 2068.278186] ? show_regs.cold.9+0x22/0x2f [ 2068.286709] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2068.297046] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2068.308545] ? watchdog+0x40/0x40 [ 2068.314308] ? hrtimer_interrupt+0x128/0x2c0 [ 2068.315824] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2068.322383] ? apic_timer_interrupt+0xf/0x20 [ 2068.324009] [ 2068.324787] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2068.338775] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2068.348673] do_raw_spin_lock+0x177/0x190 [ 2068.352387] _raw_spin_lock+0x21/0x30 [ 2068.358553] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2068.368738] process_one_work+0x2c8/0x700 [ 2068.374812] worker_thread+0x296/0x6e0 [ 2068.377561] ? rescuer_thread+0x570/0x570 [ 2068.379052] kthread+0x1d1/0x200 [ 2068.382013] ? set_kthread_struct+0x70/0x70 [ 2068.385529] ret_from_fork+0x1f/0x30 [ 2088.055018] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 2088.058823] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2088.081986] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2088.087572] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2088.105001] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2088.112668] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2088.134489] RSP: 0000:ffffaf9fc1b97bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2088.141306] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2088.150897] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 2088.155601] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 2088.163291] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 2088.172563] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2088.178821] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 2088.183944] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2088.189172] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 2088.196829] Call Trace: [ 2088.199926] [ 2088.202398] ? show_regs.cold.9+0x22/0x2f [ 2088.205881] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2088.214591] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2088.218195] ? watchdog+0x40/0x40 [ 2088.220771] ? hrtimer_interrupt+0x128/0x2c0 [ 2088.223391] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2088.228927] ? apic_timer_interrupt+0xf/0x20 [ 2088.232919] [ 2088.233800] ? native_safe_halt+0x2e/0x50 [ 2088.235289] kvm_wait+0xcd/0xe0 [ 2088.238093] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2088.241810] do_raw_spin_lock+0x177/0x190 [ 2088.245248] _raw_spin_lock+0x21/0x30 [ 2088.248434] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2088.252287] obd_mod_free+0x46/0x140 [ptlrpc] [ 2088.255475] mdc_commit_open+0x91/0xb0 [mdc] [ 2088.260570] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2088.268115] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2088.276770] after_reply+0xe2c/0x1780 [ptlrpc] [ 2088.286226] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2088.290847] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2088.299924] ? do_wait_intr+0xf0/0xf0 [ 2088.302473] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2088.312270] kthread+0x1d1/0x200 [ 2088.321189] ? set_kthread_struct+0x70/0x70 [ 2088.327519] ret_from_fork+0x1f/0x30 [ 2092.060500] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 2092.063210] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 2092.078845] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2092.078925] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2092.078932] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2092.078935] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2092.078951] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2092.078955] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2092.078960] RAX: 0000000000007e98 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2092.078963] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 2092.078965] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2092.079765] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 2092.079768] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 2092.079772] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 2092.079775] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2092.079777] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 2092.079786] Call Trace: [ 2092.079791] [ 2092.079798] ? show_regs.cold.9+0x22/0x2f [ 2092.079808] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2092.079814] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2092.079818] ? watchdog+0x40/0x40 [ 2092.079823] ? hrtimer_interrupt+0x128/0x2c0 [ 2092.079827] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2092.079835] ? apic_timer_interrupt+0xf/0x20 [ 2092.079839] [ 2092.079842] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2092.079850] do_raw_spin_lock+0x177/0x190 [ 2092.079855] _raw_spin_lock+0x21/0x30 [ 2092.079861] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2092.085458] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2092.085544] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2092.085552] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2092.085556] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2092.085569] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2092.085574] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2092.085579] RAX: 0000000000007d90 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2092.085581] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 2092.085583] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2092.085817] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 2092.085821] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 2092.085825] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 2092.085828] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2092.085834] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 2092.085841] Call Trace: [ 2092.085847] [ 2092.085852] ? show_regs.cold.9+0x22/0x2f [ 2092.085861] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2092.085868] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2092.085873] ? watchdog+0x40/0x40 [ 2092.085878] ? hrtimer_interrupt+0x128/0x2c0 [ 2092.085882] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2092.085891] ? apic_timer_interrupt+0xf/0x20 [ 2092.085896] [ 2092.085899] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2092.085906] do_raw_spin_lock+0x177/0x190 [ 2092.085911] _raw_spin_lock+0x21/0x30 [ 2092.085916] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2092.086425] ? do_raw_spin_unlock+0x75/0x190 [ 2092.086432] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2092.086640] ? do_wait_intr+0xf0/0xf0 [ 2092.086649] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2092.086879] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2092.087382] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2092.089218] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2092.089344] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2092.089383] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2092.089410] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2092.089439] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2092.089514] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2092.091132] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2092.091184] lmv_intent_open+0x338/0x1150 [lmv] [ 2092.091218] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2092.091347] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2092.091440] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2092.091462] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2092.091547] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2092.091646] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2092.091741] ? do_raw_spin_unlock+0x75/0x190 [ 2092.092118] ? _raw_spin_unlock+0x12/0x30 [ 2092.092125] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2092.092162] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2092.092182] ll_atomic_open+0x36a/0xd70 [lustre] [ 2092.092269] lookup_open+0x356/0xb40 [ 2092.092278] do_last+0x313/0xfc0 [ 2092.092282] ? nd_jump_root+0xe5/0x160 [ 2092.092287] ? path_init+0x437/0x520 [ 2092.092291] path_openat+0xf7/0x500 [ 2092.092297] ? do_raw_spin_unlock+0x75/0x190 [ 2092.092301] do_filp_open+0x99/0x140 [ 2092.092307] ? getname_flags+0x6e/0x330 [ 2092.092311] ? __check_object_size+0xff/0x256 [ 2092.092318] ? do_raw_spin_unlock+0x75/0x190 [ 2092.092321] ? _raw_spin_unlock+0x12/0x30 [ 2092.092325] do_sys_openat2+0x2b4/0x410 [ 2092.092329] do_sys_open+0x73/0xa0 [ 2092.092333] __x64_sys_openat+0x24/0x30 [ 2092.092336] do_syscall_64+0xc1/0x440 [ 2092.092342] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2092.092350] RIP: 0033:0x7ff54af45332 [ 2092.092406] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2092.092411] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2092.092416] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 2092.092418] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 2092.092420] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 2092.092422] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 2092.092425] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 2092.511121] BUG: workqueue lockup - pool [ 2092.514344] ? do_raw_spin_unlock+0x75/0x190 [ 2092.514367] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2092.520609] cpus=0-3 flags=0x4 nice=0 stuck for 133s! [ 2092.520629] Showing busy workqueues and worker pools: [ 2092.520643] workqueue events: flags=0x0 [ 2092.520646] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2092.520656] pending: ldlm_pools_recalc_task [ptlrpc] [ 2092.526390] ? do_wait_intr+0xf0/0xf0 [ 2092.526412] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2092.526694] mdc_close+0x53b/0xd70 [mdc] [ 2092.526732] lmv_close+0x18d/0x490 [lmv] [ 2092.533278] , cpu_latency_work [ptlrpc] [ 2092.536444] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 2092.536529] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 2092.536907] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2092.536917] pending: vmstat_shepherd [ 2092.536925] workqueue events_long: flags=0x0 [ 2092.536928] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2092.536952] pending: lu_objects_flush [obdclass] [ 2092.537806] workqueue events_unbound: flags=0x2 [ 2092.552289] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 2092.552945] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2092.555581] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/512 refcnt=5 [ 2092.555599] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn [ 2092.555622] workqueue events_power_efficient: flags=0x80 [ 2092.555624] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2092.555634] pending: neigh_periodic_work [ 2092.555643] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2092.555651] pending: do_cache_clean, check_lifetime [ 2092.555664] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2092.555671] pending: neigh_periodic_work [ 2092.555805] workqueue writeback: flags=0x4a [ 2092.555809] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2092.555816] pending: wb_workfn, wb_workfn [ 2092.555836] workqueue ipv6_addrconf: flags=0x40008 [ 2092.555839] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2092.555847] pending: addrconf_verify_work [ 2092.555901] workqueue knbd0-recv: flags=0x1a [ 2092.555905] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2092.555914] in-flight: 657:recv_work [ 2092.555925] workqueue ptlrpc_pinger: flags=0x2 [ 2092.555927] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2092.555932] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 2092.556903] pool 8: cpus=0-3 flags=0x4 nice=0 hung=133s workers=4 idle: 20831 12642 28859 [ 2092.556920] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2082s workers=2 idle: 659 [ 2092.759810] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2092.761851] ll_md_close+0x274/0x800 [lustre] [ 2092.763270] ll_file_release+0x128/0xc50 [lustre] [ 2092.764960] __fput+0x10d/0x440 [ 2092.766196] ____fput+0x12/0x20 [ 2092.767591] task_work_run+0xc6/0x110 [ 2092.769062] exit_to_usermode_loop+0x1dd/0x1f0 [ 2092.770791] do_syscall_64+0x3d6/0x440 [ 2092.772591] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2092.774610] RIP: 0033:0x7ff54af45cd8 [ 2092.776205] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2092.783692] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2092.786817] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 2092.789828] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 2092.792571] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 2092.795705] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 2092.798603] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2096.066611] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 2096.076055] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2096.120162] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2096.127076] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2096.131534] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2096.135337] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2096.138544] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2096.147668] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2096.151586] RAX: 0000000000007f35 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2096.155886] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 2096.159449] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 2096.162701] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 2096.166766] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 2096.170503] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 2096.174717] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2096.177704] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 2096.180356] Call Trace: [ 2096.181738] [ 2096.182940] ? show_regs.cold.9+0x22/0x2f [ 2096.184651] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2096.186500] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2096.188761] ? watchdog+0x40/0x40 [ 2096.190628] ? hrtimer_interrupt+0x128/0x2c0 [ 2096.192449] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2096.194810] ? apic_timer_interrupt+0xf/0x20 [ 2096.197092] [ 2096.198169] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2096.201212] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2096.203583] do_raw_spin_lock+0x177/0x190 [ 2096.206268] _raw_spin_lock+0x21/0x30 [ 2096.207830] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2096.210810] process_one_work+0x2c8/0x700 [ 2096.212323] worker_thread+0x296/0x6e0 [ 2096.214119] ? rescuer_thread+0x570/0x570 [ 2096.216089] kthread+0x1d1/0x200 [ 2096.217594] ? set_kthread_struct+0x70/0x70 [ 2096.219679] ret_from_fork+0x1f/0x30 [ 2120.055627] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 2120.058602] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 2120.062149] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 2120.062166] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2120.062298] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2120.062304] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2120.062307] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2120.062321] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2120.062325] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2120.062329] RAX: 00000000000009d3 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2120.062332] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 2120.062334] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2120.062336] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 2120.062338] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 2120.062341] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 2120.062345] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2120.062347] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 2120.062355] Call Trace: [ 2120.062361] [ 2120.062367] ? show_regs.cold.9+0x22/0x2f [ 2120.062377] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2120.062385] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2120.062389] ? watchdog+0x40/0x40 [ 2120.062394] ? hrtimer_interrupt+0x128/0x2c0 [ 2120.062398] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2120.062407] ? apic_timer_interrupt+0xf/0x20 [ 2120.062412] [ 2120.062415] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2120.062422] do_raw_spin_lock+0x177/0x190 [ 2120.062427] _raw_spin_lock+0x21/0x30 [ 2120.062431] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2120.062846] ? do_raw_spin_unlock+0x75/0x190 [ 2120.062852] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2120.063330] ? do_wait_intr+0xf0/0xf0 [ 2120.063339] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2120.063604] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2120.063922] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2120.064334] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2120.064439] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) [ 2120.064459] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2120.064501] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2120.064599] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2120.064649] obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul [ 2120.064630] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2120.064675] crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2120.064706] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2120.064712] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2120.064715] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2120.064728] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2120.064733] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2120.064840] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2120.064844] RDX: 0000000000007c17 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 2120.064846] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 2120.064848] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 2120.064850] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2120.064853] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 2120.064855] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2120.064857] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 2120.064865] Call Trace: [ 2120.064869] [ 2120.064881] ? show_regs.cold.9+0x22/0x2f [ 2120.064890] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2120.064896] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2120.064900] ? watchdog+0x40/0x40 [ 2120.064904] ? hrtimer_interrupt+0x128/0x2c0 [ 2120.064909] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2120.064918] ? apic_timer_interrupt+0xf/0x20 [ 2120.064922] [ 2120.064925] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2120.064930] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2120.064935] do_raw_spin_lock+0x177/0x190 [ 2120.064940] _raw_spin_lock+0x21/0x30 [ 2120.064946] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2120.064722] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2120.065704] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2120.065737] lmv_intent_open+0x338/0x1150 [lmv] [ 2120.065626] obd_mod_free+0x46/0x140 [ptlrpc] [ 2120.066239] mdc_commit_open+0x91/0xb0 [mdc] [ 2120.065992] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2120.066365] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2120.066446] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2120.066469] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2120.066629] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2120.066721] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2120.067127] ? do_raw_spin_unlock+0x75/0x190 [ 2120.067133] ? _raw_spin_unlock+0x12/0x30 [ 2120.067140] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2120.067170] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2120.066637] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2120.067188] ll_atomic_open+0x36a/0xd70 [lustre] [ 2120.070423] lookup_open+0x356/0xb40 [ 2120.070436] do_last+0x313/0xfc0 [ 2120.070441] ? nd_jump_root+0xe5/0x160 [ 2120.070446] ? path_init+0x437/0x520 [ 2120.070451] path_openat+0xf7/0x500 [ 2120.070456] ? do_raw_spin_unlock+0x75/0x190 [ 2120.070463] do_filp_open+0x99/0x140 [ 2120.070469] ? getname_flags+0x6e/0x330 [ 2120.070474] ? __check_object_size+0xff/0x256 [ 2120.070483] ? do_raw_spin_unlock+0x75/0x190 [ 2120.070487] ? _raw_spin_unlock+0x12/0x30 [ 2120.070494] do_sys_openat2+0x2b4/0x410 [ 2120.070498] do_sys_open+0x73/0xa0 [ 2120.070502] __x64_sys_openat+0x24/0x30 [ 2120.070506] do_syscall_64+0xc1/0x440 [ 2120.070925] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2120.070932] RIP: 0033:0x7ff54af45332 [ 2120.070941] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2120.070945] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2120.070950] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 2120.070952] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 2120.070954] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 2120.070956] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 2120.070958] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 2120.078930] Modules linked in: lustre(O) [ 2120.083278] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2120.109393] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2120.109473] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2120.109544] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2120.109548] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2120.109562] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2120.109566] RSP: 0018:ffffaf9fc1543ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2120.109570] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2120.109572] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 2120.109574] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2120.109576] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 2120.109578] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 2120.109582] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 2120.109585] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2120.109587] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 2120.109596] Call Trace: [ 2120.109601] [ 2120.109607] ? show_regs.cold.9+0x22/0x2f [ 2120.109616] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2120.109622] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2120.109626] ? watchdog+0x40/0x40 [ 2120.109630] ? hrtimer_interrupt+0x128/0x2c0 [ 2120.109634] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2120.109640] ? apic_timer_interrupt+0xf/0x20 [ 2120.109645] [ 2120.109648] ? native_safe_halt+0x2e/0x50 [ 2120.109652] kvm_wait+0xcd/0xe0 [ 2120.109659] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2120.109668] do_raw_spin_lock+0x177/0x190 [ 2120.109672] _raw_spin_lock+0x21/0x30 [ 2120.109677] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2120.110114] ? do_raw_spin_unlock+0x75/0x190 [ 2120.110119] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2120.111192] ? do_wait_intr+0xf0/0xf0 [ 2120.111203] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2120.111461] mdc_close+0x53b/0xd70 [mdc] [ 2120.121004] after_reply+0xe2c/0x1780 [ptlrpc] [ 2120.126415] lmv_close+0x18d/0x490 [lmv] [ 2120.126465] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 2120.126658] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2120.126972] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2120.127057] ll_md_close+0x274/0x800 [lustre] [ 2120.127132] ll_file_release+0x128/0xc50 [lustre] [ 2120.127213] __fput+0x10d/0x440 [ 2120.127223] ____fput+0x12/0x20 [ 2120.127227] task_work_run+0xc6/0x110 [ 2120.127235] exit_to_usermode_loop+0x1dd/0x1f0 [ 2120.127243] do_syscall_64+0x3d6/0x440 [ 2120.127247] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2120.127255] RIP: 0033:0x7ff54af45cd8 [ 2120.127264] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2120.127268] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2120.127274] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 2120.127276] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 2120.127278] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 2120.127291] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 2120.127293] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2120.873978] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2120.876642] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2120.879072] ? do_wait_intr+0xf0/0xf0 [ 2120.880766] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2120.883136] kthread+0x1d1/0x200 [ 2120.884826] ? set_kthread_struct+0x70/0x70 [ 2120.886327] ret_from_fork+0x1f/0x30 [ 2122.726239] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 163s! [ 2122.735155] Showing busy workqueues and worker pools: [ 2122.743470] workqueue events: flags=0x0 [ 2122.748400] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2122.757261] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 2122.766077] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 2122.775422] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 2122.785463] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2122.794564] pending: vmstat_shepherd [ 2122.799873] workqueue events_long: flags=0x0 [ 2122.801292] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2122.804838] pending: lu_objects_flush [obdclass] [ 2122.807389] workqueue events_unbound: flags=0x2 [ 2122.809333] pwq 8: cpus=0-3 flags=0x4 nice=0 active=4/512 refcnt=6 [ 2122.818766] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2122.831152] workqueue events_power_efficient: flags=0x80 [ 2122.839014] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2122.850252] pending: neigh_periodic_work [ 2122.856413] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2122.865613] pending: do_cache_clean, check_lifetime [ 2122.868794] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2122.879349] pending: neigh_periodic_work [ 2122.882277] workqueue writeback: flags=0x4a [ 2122.884013] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2122.886542] pending: wb_workfn, wb_workfn [ 2122.888427] workqueue ipv6_addrconf: flags=0x40008 [ 2122.893724] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2122.900219] pending: addrconf_verify_work [ 2122.904195] workqueue knbd0-recv: flags=0x1a [ 2122.906450] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2122.911781] in-flight: 657:recv_work [ 2122.913410] workqueue ptlrpc_pinger: flags=0x2 [ 2122.915369] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2122.918152] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 2122.920524] pool 8: cpus=0-3 flags=0x4 nice=0 hung=164s workers=4 idle: 20831 12642 28859 [ 2122.924049] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2113s workers=2 idle: 659 [ 2124.059015] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 2124.067841] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2124.111936] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2124.118611] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2124.126982] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2124.131476] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2124.138089] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2124.150633] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2124.155617] RAX: 0000000000003b4f RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2124.159251] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 2124.162340] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 2124.164538] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 2124.170303] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 2124.177142] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 2124.182322] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2124.185927] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 2124.192309] Call Trace: [ 2124.193621] [ 2124.194736] ? show_regs.cold.9+0x22/0x2f [ 2124.197506] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2124.201454] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2124.203520] ? watchdog+0x40/0x40 [ 2124.205623] ? hrtimer_interrupt+0x128/0x2c0 [ 2124.207596] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2124.210432] ? apic_timer_interrupt+0xf/0x20 [ 2124.213591] [ 2124.215558] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2124.219975] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2124.222076] do_raw_spin_lock+0x177/0x190 [ 2124.223581] _raw_spin_lock+0x21/0x30 [ 2124.225920] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2124.230054] process_one_work+0x2c8/0x700 [ 2124.233601] worker_thread+0x296/0x6e0 [ 2124.237017] ? rescuer_thread+0x570/0x570 [ 2124.240641] kthread+0x1d1/0x200 [ 2124.243540] ? set_kthread_struct+0x70/0x70 [ 2124.246400] ret_from_fork+0x1f/0x30 [ 2148.055140] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 2148.058023] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 2148.061816] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 2148.061831] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2148.061900] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2148.061906] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2148.061909] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2148.061924] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2148.061928] RSP: 0000:ffffaf9fc15234f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2148.061932] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2148.061934] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 2148.061936] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2148.061937] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 2148.061939] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 2148.061942] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 2148.061945] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2148.061947] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 2148.061955] Call Trace: [ 2148.061960] [ 2148.061965] ? show_regs.cold.9+0x22/0x2f [ 2148.061974] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2148.061981] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2148.061985] ? watchdog+0x40/0x40 [ 2148.061989] ? hrtimer_interrupt+0x128/0x2c0 [ 2148.061993] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2148.061999] ? apic_timer_interrupt+0xf/0x20 [ 2148.062003] [ 2148.062006] ? native_safe_halt+0x2e/0x50 [ 2148.062011] kvm_wait+0xcd/0xe0 [ 2148.062017] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2148.062025] do_raw_spin_lock+0x177/0x190 [ 2148.062029] _raw_spin_lock+0x21/0x30 [ 2148.062033] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2148.062342] ? do_raw_spin_unlock+0x75/0x190 [ 2148.062348] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2148.062730] ? do_wait_intr+0xf0/0xf0 [ 2148.062739] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2148.062930] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2148.063148] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2148.063371] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2148.063494] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2148.063531] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2148.063560] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2148.063587] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2148.063681] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2148.065075] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2148.065112] lmv_intent_open+0x338/0x1150 [lmv] [ 2148.065150] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2148.065273] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2148.065362] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2148.065528] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2148.065621] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2148.065705] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2148.065881] ? do_raw_spin_unlock+0x75/0x190 [ 2148.065890] ? _raw_spin_unlock+0x12/0x30 [ 2148.065897] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2148.065926] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2148.065946] ll_atomic_open+0x36a/0xd70 [lustre] [ 2148.066030] lookup_open+0x356/0xb40 [ 2148.066040] do_last+0x313/0xfc0 [ 2148.066045] ? nd_jump_root+0xe5/0x160 [ 2148.066050] ? path_init+0x437/0x520 [ 2148.066055] path_openat+0xf7/0x500 [ 2148.066059] ? do_raw_spin_unlock+0x75/0x190 [ 2148.066063] do_filp_open+0x99/0x140 [ 2148.066069] ? getname_flags+0x6e/0x330 [ 2148.066074] ? __check_object_size+0xff/0x256 [ 2148.066082] ? do_raw_spin_unlock+0x75/0x190 [ 2148.066085] ? _raw_spin_unlock+0x12/0x30 [ 2148.066090] do_sys_openat2+0x2b4/0x410 [ 2148.066094] do_sys_open+0x73/0xa0 [ 2148.066098] __x64_sys_openat+0x24/0x30 [ 2148.066102] do_syscall_64+0xc1/0x440 [ 2148.066109] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2148.066115] RIP: 0033:0x7ff54af45332 [ 2148.066124] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2148.066127] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2148.066132] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 2148.066135] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 2148.066137] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 2148.066139] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 2148.066141] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 2148.069941] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2148.070020] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2148.070029] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2148.070032] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2148.070049] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2148.070053] RSP: 0000:ffffaf9fc1b97bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2148.070058] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2148.070060] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 2148.070063] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 2148.070065] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 2148.070068] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2148.070070] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 2148.070074] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2148.070076] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 2148.070085] Call Trace: [ 2148.070090] [ 2148.070097] ? show_regs.cold.9+0x22/0x2f [ 2148.070109] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2148.070115] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2148.070119] ? watchdog+0x40/0x40 [ 2148.070124] ? hrtimer_interrupt+0x128/0x2c0 [ 2148.070128] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2148.070135] ? apic_timer_interrupt+0xf/0x20 [ 2148.070139] [ 2148.070143] ? native_safe_halt+0x2e/0x50 [ 2148.070148] kvm_wait+0xcd/0xe0 [ 2148.070155] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2148.070163] do_raw_spin_lock+0x177/0x190 [ 2148.070168] _raw_spin_lock+0x21/0x30 [ 2148.070173] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2148.070683] obd_mod_free+0x46/0x140 [ptlrpc] [ 2148.070963] mdc_commit_open+0x91/0xb0 [mdc] [ 2148.071007] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2148.071201] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2148.077152] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2148.077315] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2148.077324] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2148.077327] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2148.077342] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2148.077413] RSP: 0018:ffffaf9fc1543ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2148.077418] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2148.077420] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 2148.077423] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2148.077484] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 2148.077486] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 2148.077489] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 2148.077492] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2148.077494] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 2148.077501] Call Trace: [ 2148.077507] [ 2148.077513] ? show_regs.cold.9+0x22/0x2f [ 2148.077523] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2148.077529] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2148.077534] ? watchdog+0x40/0x40 [ 2148.077538] ? hrtimer_interrupt+0x128/0x2c0 [ 2148.077541] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2148.077548] ? apic_timer_interrupt+0xf/0x20 [ 2148.077553] [ 2148.077557] ? native_safe_halt+0x2e/0x50 [ 2148.077561] kvm_wait+0xcd/0xe0 [ 2148.077569] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2148.077578] do_raw_spin_lock+0x177/0x190 [ 2148.077583] _raw_spin_lock+0x21/0x30 [ 2148.077587] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2148.078078] ? do_raw_spin_unlock+0x75/0x190 [ 2148.078084] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2148.078298] ? do_wait_intr+0xf0/0xf0 [ 2148.078306] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2148.078761] mdc_close+0x53b/0xd70 [mdc] [ 2148.078807] lmv_close+0x18d/0x490 [lmv] [ 2148.078840] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 2148.079023] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2148.079131] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2148.079220] ll_md_close+0x274/0x800 [lustre] [ 2148.079302] ll_file_release+0x128/0xc50 [lustre] [ 2148.079399] __fput+0x10d/0x440 [ 2148.079408] ____fput+0x12/0x20 [ 2148.079413] task_work_run+0xc6/0x110 [ 2148.079420] exit_to_usermode_loop+0x1dd/0x1f0 [ 2148.079723] do_syscall_64+0x3d6/0x440 [ 2148.079728] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2148.079737] RIP: 0033:0x7ff54af45cd8 [ 2148.079745] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2148.079750] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2148.079755] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 2148.079758] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 2148.079760] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 2148.079762] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 2148.079764] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2148.856638] after_reply+0xe2c/0x1780 [ptlrpc] [ 2148.860219] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2148.862342] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2148.864307] ? do_wait_intr+0xf0/0xf0 [ 2148.865821] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2148.868320] kthread+0x1d1/0x200 [ 2148.869652] ? set_kthread_struct+0x70/0x70 [ 2148.871581] ret_from_fork+0x1f/0x30 [ 2152.059063] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 2152.067302] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2152.104052] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2152.115855] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2152.120798] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2152.123678] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2152.125848] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2152.140995] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2152.145270] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2152.150441] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 2152.153331] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 2152.156935] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 2152.159964] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 2152.162356] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 2152.165773] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2152.170761] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 2152.176899] Call Trace: [ 2152.178747] [ 2152.179359] ? show_regs.cold.9+0x22/0x2f [ 2152.184027] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2152.186015] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2152.187867] ? watchdog+0x40/0x40 [ 2152.189642] ? hrtimer_interrupt+0x128/0x2c0 [ 2152.195162] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2152.201765] ? apic_timer_interrupt+0xf/0x20 [ 2152.204257] [ 2152.205631] ? native_safe_halt+0x2e/0x50 [ 2152.213863] kvm_wait+0xcd/0xe0 [ 2152.217646] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2152.223787] do_raw_spin_lock+0x177/0x190 [ 2152.225341] _raw_spin_lock+0x21/0x30 [ 2152.227673] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2152.231352] process_one_work+0x2c8/0x700 [ 2152.233352] worker_thread+0x296/0x6e0 [ 2152.237973] ? rescuer_thread+0x570/0x570 [ 2152.240296] kthread+0x1d1/0x200 [ 2152.241888] ? set_kthread_struct+0x70/0x70 [ 2152.243495] ret_from_fork+0x1f/0x30 [ 2153.439101] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 194s! [ 2153.447520] Showing busy workqueues and worker pools: [ 2153.450791] workqueue events: flags=0x0 [ 2153.454519] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2153.458338] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 2153.464104] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 2153.471912] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 2153.487908] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2153.499418] pending: vmstat_shepherd [ 2153.501711] workqueue events_long: flags=0x0 [ 2153.507121] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2153.513525] pending: lu_objects_flush [obdclass] [ 2153.521572] workqueue events_unbound: flags=0x2 [ 2153.527735] pwq 8: cpus=0-3 flags=0x4 nice=0 active=4/512 refcnt=6 [ 2153.532861] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2153.540826] workqueue events_power_efficient: flags=0x80 [ 2153.548161] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2153.562769] pending: neigh_periodic_work [ 2153.571105] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2153.575076] pending: do_cache_clean, check_lifetime [ 2153.578780] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2153.587734] pending: neigh_periodic_work [ 2153.592752] workqueue writeback: flags=0x4a [ 2153.599463] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2153.605827] pending: wb_workfn, wb_workfn [ 2153.609249] workqueue ipv6_addrconf: flags=0x40008 [ 2153.613082] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2153.618455] pending: addrconf_verify_work [ 2153.622771] workqueue knbd0-recv: flags=0x1a [ 2153.625748] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2153.635729] in-flight: 657:recv_work [ 2153.645278] workqueue ptlrpc_pinger: flags=0x2 [ 2153.652135] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2153.656609] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 2153.663041] pool 8: cpus=0-3 flags=0x4 nice=0 hung=194s workers=4 idle: 20831 12642 28859 [ 2153.673800] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2143s workers=2 idle: 659 [ 2176.062382] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 2176.063908] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 2176.072284] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2176.072366] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2176.072373] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2176.072375] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2176.072400] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2176.072409] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 2176.072414] RAX: 0000000000007e18 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2176.072416] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 2176.072418] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2176.072420] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 2176.072422] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 2176.072425] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 2176.072428] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2176.072430] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 2176.072437] Call Trace: [ 2176.072450] [ 2176.072456] ? show_regs.cold.9+0x22/0x2f [ 2176.072465] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2176.072471] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2176.072475] ? watchdog+0x40/0x40 [ 2176.072479] ? hrtimer_interrupt+0x128/0x2c0 [ 2176.072483] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2176.072491] ? apic_timer_interrupt+0xf/0x20 [ 2176.072495] [ 2176.072498] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2176.072504] do_raw_spin_lock+0x177/0x190 [ 2176.072509] _raw_spin_lock+0x21/0x30 [ 2176.072514] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2176.072808] ? do_raw_spin_unlock+0x75/0x190 [ 2176.072813] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2176.073018] ? do_wait_intr+0xf0/0xf0 [ 2176.073027] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2176.073708] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2176.073917] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2176.074813] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2176.074920] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2176.075620] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2176.075652] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2176.075681] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2176.075777] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2176.076739] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2176.076774] lmv_intent_open+0x338/0x1150 [lmv] [ 2176.076804] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2176.076899] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2176.076993] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2176.077014] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2176.077234] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2176.077312] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2176.077681] ? do_raw_spin_unlock+0x75/0x190 [ 2176.077691] ? _raw_spin_unlock+0x12/0x30 [ 2176.077698] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2176.077730] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2176.077750] ll_atomic_open+0x36a/0xd70 [lustre] [ 2176.077835] lookup_open+0x356/0xb40 [ 2176.077843] do_last+0x313/0xfc0 [ 2176.077849] ? nd_jump_root+0xe5/0x160 [ 2176.077853] ? path_init+0x437/0x520 [ 2176.077858] path_openat+0xf7/0x500 [ 2176.077863] ? do_raw_spin_unlock+0x75/0x190 [ 2176.077867] do_filp_open+0x99/0x140 [ 2176.077873] ? getname_flags+0x6e/0x330 [ 2176.077877] ? __check_object_size+0xff/0x256 [ 2176.077884] ? do_raw_spin_unlock+0x75/0x190 [ 2176.077887] ? _raw_spin_unlock+0x12/0x30 [ 2176.077892] do_sys_openat2+0x2b4/0x410 [ 2176.077896] do_sys_open+0x73/0xa0 [ 2176.077900] __x64_sys_openat+0x24/0x30 [ 2176.077903] do_syscall_64+0xc1/0x440 [ 2176.077909] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2176.077915] RIP: 0033:0x7ff54af45332 [ 2176.077924] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2176.077928] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2176.077933] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 2176.077936] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 2176.077937] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 2176.077954] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 2176.077957] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 2176.533800] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2176.574900] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2176.581830] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2176.587483] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2176.590971] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2176.611338] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2176.616660] RAX: 0000000000006687 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2176.622746] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 2176.630128] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2176.636076] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 2176.640543] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 2176.643306] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 2176.650104] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2176.653637] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 2176.657707] Call Trace: [ 2176.660371] [ 2176.661492] ? show_regs.cold.9+0x22/0x2f [ 2176.665134] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2176.670121] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2176.672728] ? watchdog+0x40/0x40 [ 2176.675943] ? hrtimer_interrupt+0x128/0x2c0 [ 2176.677690] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2176.681681] ? apic_timer_interrupt+0xf/0x20 [ 2176.684680] [ 2176.686899] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2176.690710] do_raw_spin_lock+0x177/0x190 [ 2176.695585] _raw_spin_lock+0x21/0x30 [ 2176.703831] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2176.708546] ? do_raw_spin_unlock+0x75/0x190 [ 2176.713164] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2176.716461] ? do_wait_intr+0xf0/0xf0 [ 2176.718916] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2176.725302] mdc_close+0x53b/0xd70 [mdc] [ 2176.728632] lmv_close+0x18d/0x490 [lmv] [ 2176.731389] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 2176.737475] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2176.744975] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2176.747367] ll_md_close+0x274/0x800 [lustre] [ 2176.751059] ll_file_release+0x128/0xc50 [lustre] [ 2176.754011] __fput+0x10d/0x440 [ 2176.755039] ____fput+0x12/0x20 [ 2176.757404] task_work_run+0xc6/0x110 [ 2176.760210] exit_to_usermode_loop+0x1dd/0x1f0 [ 2176.763140] do_syscall_64+0x3d6/0x440 [ 2176.765364] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2176.773308] RIP: 0033:0x7ff54af45cd8 [ 2176.777256] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2176.793157] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2176.801821] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 2176.810066] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 2176.815199] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 2176.817678] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 2176.822378] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2180.055306] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [ 2180.060022] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2180.063955] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 2180.131748] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2180.131766] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2180.131770] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2180.131788] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2180.131793] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2180.131799] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2180.131802] RDX: 0000000000006638 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 2180.131804] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 2180.131806] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 2180.131808] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2180.131811] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 2180.131813] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2180.131816] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 2180.131824] Call Trace: [ 2180.131831] [ 2180.132714] ? show_regs.cold.9+0x22/0x2f [ 2180.132728] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2180.132734] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2180.132737] ? watchdog+0x40/0x40 [ 2180.132742] ? hrtimer_interrupt+0x128/0x2c0 [ 2180.132746] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2180.132753] ? apic_timer_interrupt+0xf/0x20 [ 2180.132758] [ 2180.132762] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2180.132767] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2180.132772] do_raw_spin_lock+0x177/0x190 [ 2180.132777] _raw_spin_lock+0x21/0x30 [ 2180.132785] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2180.133091] obd_mod_free+0x46/0x140 [ptlrpc] [ 2180.133312] mdc_commit_open+0x91/0xb0 [mdc] [ 2180.133349] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2180.134015] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2180.134484] after_reply+0xe2c/0x1780 [ptlrpc] [ 2180.134687] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2180.135549] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2180.149015] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2180.149099] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2180.149106] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2180.149124] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2180.149660] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2180.149673] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2180.149709] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2180.149715] RAX: 0000000000005c3e RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2180.149717] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 2180.149719] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 2180.149721] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 2180.149723] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 2180.149726] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 2180.149729] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2180.149732] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 2180.149740] Call Trace: [ 2180.149746] [ 2180.149752] ? show_regs.cold.9+0x22/0x2f [ 2180.149763] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2180.149769] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2180.149774] ? watchdog+0x40/0x40 [ 2180.149778] ? hrtimer_interrupt+0x128/0x2c0 [ 2180.149782] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2180.149790] ? apic_timer_interrupt+0xf/0x20 [ 2180.149795] [ 2180.149798] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2180.149804] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2180.149809] do_raw_spin_lock+0x177/0x190 [ 2180.149813] _raw_spin_lock+0x21/0x30 [ 2180.149818] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2180.150531] process_one_work+0x2c8/0x700 [ 2180.150538] worker_thread+0x296/0x6e0 [ 2180.150542] ? rescuer_thread+0x570/0x570 [ 2180.150546] kthread+0x1d1/0x200 [ 2180.150550] ? set_kthread_struct+0x70/0x70 [ 2180.150555] ret_from_fork+0x1f/0x30 [ 2180.471568] ? do_wait_intr+0xf0/0xf0 [ 2180.475282] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2180.477556] kthread+0x1d1/0x200 [ 2180.480956] ? set_kthread_struct+0x70/0x70 [ 2180.482837] ret_from_fork+0x1f/0x30 [ 2184.159147] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 225s! [ 2184.165445] Showing busy workqueues and worker pools: [ 2184.168024] workqueue events: flags=0x0 [ 2184.170674] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2184.174544] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 2184.179966] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 2184.183506] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 2184.190338] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2184.194009] pending: vmstat_shepherd [ 2184.196289] workqueue events_long: flags=0x0 [ 2184.198680] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2184.202639] pending: lu_objects_flush [obdclass] [ 2184.205552] workqueue events_unbound: flags=0x2 [ 2184.208043] pwq 8: cpus=0-3 flags=0x4 nice=0 active=4/512 refcnt=6 [ 2184.211056] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2184.214760] workqueue events_power_efficient: flags=0x80 [ 2184.218013] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2184.221716] pending: neigh_periodic_work [ 2184.224230] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2184.228219] pending: do_cache_clean, check_lifetime [ 2184.231449] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2184.235491] pending: neigh_periodic_work [ 2184.237810] workqueue writeback: flags=0x4a [ 2184.240315] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2184.244557] pending: wb_workfn, wb_workfn [ 2184.247317] workqueue ipv6_addrconf: flags=0x40008 [ 2184.250264] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2184.254142] pending: addrconf_verify_work [ 2184.256520] workqueue knbd0-recv: flags=0x1a [ 2184.259041] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2184.262649] in-flight: 657:recv_work [ 2184.264488] workqueue ptlrpc_pinger: flags=0x2 [ 2184.266920] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2184.270165] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 2184.273527] pool 8: cpus=0-3 flags=0x4 nice=0 hung=225s workers=4 idle: 20831 12642 28859 [ 2184.278215] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2174s workers=2 idle: 659 [ 2193.844957] rcu: INFO: rcu_sched self-detected stall on CPU [ 2193.850371] rcu: 0-...!: (132714 ticks this GP) idle=98a/1/0x4000000000000002 softirq=125154/125154 fqs=959 [ 2193.855482] (t=240004 jiffies g=253585 q=2724) [ 2193.858384] rcu: rcu_sched kthread starved for 234964 jiffies! g253585 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [ 2193.863171] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 2193.867127] rcu: RCU grace-period kthread stack dump: [ 2193.868835] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 2193.875806] Call Trace: [ 2193.877237] __schedule+0x351/0xcb0 [ 2193.879199] schedule+0xc0/0x180 [ 2193.880568] schedule_timeout+0xb4/0x190 [ 2193.882541] ? do_raw_spin_unlock+0x75/0x190 [ 2193.884818] ? __next_timer_interrupt+0x160/0x160 [ 2193.886913] ? prepare_to_swait_event+0xf3/0x140 [ 2193.889247] rcu_gp_kthread+0x8dd/0x15b0 [ 2193.891097] ? force_qs_rnp+0x2c0/0x2c0 [ 2193.892951] kthread+0x1d1/0x200 [ 2193.894751] ? set_kthread_struct+0x70/0x70 [ 2193.896842] ret_from_fork+0x1f/0x30 [ 2193.898766] rcu: Stack dump where RCU GP kthread last ran: [ 2193.901327] Sending NMI from CPU 0 to CPUs 2: [ 2193.904360] NMI backtrace for cpu 2 [ 2193.904366] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2193.904368] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2193.904370] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2193.904376] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2193.904377] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2193.904379] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 [ 2193.904382] RAX: 0000000000002ead RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2193.904383] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 2193.904398] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 2193.904400] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 2193.904401] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 2193.904402] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 2193.904404] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2193.904405] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 2193.904407] Call Trace: [ 2193.904407] [ 2193.904409] ? show_regs.cold.9+0x22/0x2f [ 2193.904412] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 2193.904414] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 2193.904414] ? nmi_handle+0xbd/0x210 [ 2193.904415] ? default_do_nmi+0x4b/0x1b0 [ 2193.904416] ? do_nmi+0x280/0x380 [ 2193.904418] ? end_repeat_nmi+0x16/0x1f [ 2193.904419] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2193.904420] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2193.904422] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2193.904423] [ 2193.904423] do_raw_spin_lock+0x177/0x190 [ 2193.904424] _raw_spin_lock+0x21/0x30 [ 2193.904425] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2193.904429] process_one_work+0x2c8/0x700 [ 2193.904430] worker_thread+0x296/0x6e0 [ 2193.904431] ? rescuer_thread+0x570/0x570 [ 2193.904432] kthread+0x1d1/0x200 [ 2193.904432] ? set_kthread_struct+0x70/0x70 [ 2193.904433] ret_from_fork+0x1f/0x30 [ 2193.904521] NMI backtrace for cpu 0 [ 2193.996532] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2194.001466] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2194.005257] Call Trace: [ 2194.006537] [ 2194.007445] ? dump_stack+0xbb/0x10e [ 2194.008775] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 2194.011001] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 2194.013313] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 2194.015460] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 2194.017717] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 2194.019826] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 2194.021826] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 2194.023862] ? update_process_times+0x6d/0xc0 [ 2194.025804] ? tick_sched_handle.isra.18+0x27/0x90 [ 2194.027896] ? tick_sched_timer+0x53/0xb0 [ 2194.029329] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2194.031421] ? tick_sched_handle.isra.18+0x90/0x90 [ 2194.033541] ? hrtimer_interrupt+0x128/0x2c0 [ 2194.035323] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2194.037493] ? apic_timer_interrupt+0xf/0x20 [ 2194.039101] [ 2194.040258] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2194.042592] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2194.045066] ? do_raw_spin_lock+0x177/0x190 [ 2194.046882] ? _raw_spin_lock+0x21/0x30 [ 2194.048626] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2194.050550] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 2194.052482] ? mdc_commit_open+0x91/0xb0 [mdc] [ 2194.054039] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2194.055972] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2194.058415] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 2194.060842] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2194.063612] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2194.065378] ? do_wait_intr+0xf0/0xf0 [ 2194.066737] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2194.068890] ? kthread+0x1d1/0x200 [ 2194.070408] ? set_kthread_struct+0x70/0x70 [ 2194.072169] ? ret_from_fork+0x1f/0x30 [ 2204.058177] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 2204.060263] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 2204.064738] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 [ 2204.070719] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2204.070810] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2204.070820] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2204.070824] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2204.070841] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2204.070846] RSP: 0000:ffffaf9fc15234f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2204.071002] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2204.071004] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 2204.071007] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2204.071009] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 2204.071011] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 2204.071015] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 2204.071018] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2204.071021] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 2204.071031] Call Trace: [ 2204.071043] [ 2204.071051] ? show_regs.cold.9+0x22/0x2f [ 2204.071075] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2204.071085] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2204.071093] ? watchdog+0x40/0x40 [ 2204.071098] ? hrtimer_interrupt+0x128/0x2c0 [ 2204.071102] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2204.071110] ? apic_timer_interrupt+0xf/0x20 [ 2204.071115] [ 2204.071119] ? native_safe_halt+0x2e/0x50 [ 2204.071122] kvm_wait+0xcd/0xe0 [ 2204.071135] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2204.071146] do_raw_spin_lock+0x177/0x190 [ 2204.071151] _raw_spin_lock+0x21/0x30 [ 2204.071156] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2204.071513] ? do_raw_spin_unlock+0x75/0x190 [ 2204.071518] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2204.071712] ? do_wait_intr+0xf0/0xf0 [ 2204.071721] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2204.072012] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2204.072264] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2204.072491] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2204.072634] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2204.072682] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2204.072711] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2204.072736] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2204.072820] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2204.073237] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2204.073272] lmv_intent_open+0x338/0x1150 [lmv] [ 2204.073307] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2204.073400] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2204.073545] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2204.096414] sha512_generic [ 2204.096435] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2204.096444] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2204.096447] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2204.096535] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2204.096541] RSP: 0018:ffffaf9fc1543ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2204.096546] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2204.096549] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 2204.096551] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2204.096553] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 2204.096561] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 2204.096566] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 2204.096570] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2204.096572] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 2204.096588] Call Trace: [ 2204.096595] [ 2204.096603] ? show_regs.cold.9+0x22/0x2f [ 2204.096615] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2204.096624] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2204.096628] ? watchdog+0x40/0x40 [ 2204.096633] ? hrtimer_interrupt+0x128/0x2c0 [ 2204.096637] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2204.096644] ? apic_timer_interrupt+0xf/0x20 [ 2204.096649] [ 2204.096652] ? native_safe_halt+0x2e/0x50 [ 2204.096656] kvm_wait+0xcd/0xe0 [ 2204.096665] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2204.096678] do_raw_spin_lock+0x177/0x190 [ 2204.096682] _raw_spin_lock+0x21/0x30 [ 2204.096687] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2204.097103] ? do_raw_spin_unlock+0x75/0x190 [ 2204.097109] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2204.097368] ? do_wait_intr+0xf0/0xf0 [ 2204.097378] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2204.119831] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2204.120163] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2204.120272] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2204.120378] ? do_raw_spin_unlock+0x75/0x190 [ 2204.120394] ? _raw_spin_unlock+0x12/0x30 [ 2204.120403] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2204.120436] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2204.120457] ll_atomic_open+0x36a/0xd70 [lustre] [ 2204.120625] lookup_open+0x356/0xb40 [ 2204.120636] do_last+0x313/0xfc0 [ 2204.120642] ? nd_jump_root+0xe5/0x160 [ 2204.120647] ? path_init+0x437/0x520 [ 2204.120652] path_openat+0xf7/0x500 [ 2204.120657] ? do_raw_spin_unlock+0x75/0x190 [ 2204.120662] do_filp_open+0x99/0x140 [ 2204.120670] ? getname_flags+0x6e/0x330 [ 2204.120679] ? __check_object_size+0xff/0x256 [ 2204.120690] ? do_raw_spin_unlock+0x75/0x190 [ 2204.120695] ? _raw_spin_unlock+0x12/0x30 [ 2204.120699] do_sys_openat2+0x2b4/0x410 [ 2204.120704] do_sys_open+0x73/0xa0 [ 2204.120709] __x64_sys_openat+0x24/0x30 [ 2204.120716] do_syscall_64+0xc1/0x440 [ 2204.120724] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2204.120732] RIP: 0033:0x7ff54af45332 [ 2204.120744] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2204.120749] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2204.120758] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 2204.120761] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 2204.120763] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 2204.120765] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 2204.120768] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 2204.450321] mdc_close+0x53b/0xd70 [mdc] [ 2204.452286] lmv_close+0x18d/0x490 [lmv] [ 2204.454065] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 2204.457435] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2204.460551] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2204.463720] ll_md_close+0x274/0x800 [lustre] [ 2204.465764] ll_file_release+0x128/0xc50 [lustre] [ 2204.470011] __fput+0x10d/0x440 [ 2204.470982] ____fput+0x12/0x20 [ 2204.472553] task_work_run+0xc6/0x110 [ 2204.476010] exit_to_usermode_loop+0x1dd/0x1f0 [ 2204.479399] do_syscall_64+0x3d6/0x440 [ 2204.481116] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2204.485148] RIP: 0033:0x7ff54af45cd8 [ 2204.489646] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2204.504433] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2204.508359] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 2204.513596] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 2204.516253] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 2204.520146] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 2204.523144] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2214.367159] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 255s! [ 2214.370685] Showing busy workqueues and worker pools: [ 2214.375437] workqueue events: flags=0x0 [ 2214.379461] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2214.390322] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 2214.398475] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 2214.403650] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 2214.415630] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2214.421510] pending: vmstat_shepherd [ 2214.423174] workqueue events_long: flags=0x0 [ 2214.424703] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2214.427417] pending: lu_objects_flush [obdclass] [ 2214.434447] workqueue events_unbound: flags=0x2 [ 2214.436539] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 2214.439479] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2214.448933] workqueue events_power_efficient: flags=0x80 [ 2214.451650] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2214.459277] pending: neigh_periodic_work [ 2214.462967] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2214.468153] pending: do_cache_clean, check_lifetime [ 2214.472299] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2214.475115] pending: neigh_periodic_work [ 2214.477549] workqueue writeback: flags=0x4a [ 2214.482875] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2214.487578] pending: wb_workfn, wb_workfn [ 2214.494688] workqueue ipv6_addrconf: flags=0x40008 [ 2214.496885] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2214.500712] pending: addrconf_verify_work [ 2214.503247] workqueue knbd0-recv: flags=0x1a [ 2214.507021] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2214.511599] in-flight: 657:recv_work [ 2214.515516] workqueue ptlrpc_pinger: flags=0x2 [ 2214.519138] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2214.527426] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 2214.529534] pool 8: cpus=0-3 flags=0x4 nice=0 hung=255s workers=4 idle: 20831 12642 28859 [ 2214.532707] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2204s workers=2 idle: 659 [ 2220.059034] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 2220.064925] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2220.139608] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2220.160992] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2220.172174] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2220.179939] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2220.188959] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2220.216040] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2220.225358] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2220.233086] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 2220.247200] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 2220.255282] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 2220.268856] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 2220.277965] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 2220.288495] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2220.293939] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 2220.297707] Call Trace: [ 2220.298988] [ 2220.299693] ? show_regs.cold.9+0x22/0x2f [ 2220.302391] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2220.309509] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2220.311085] ? watchdog+0x40/0x40 [ 2220.313633] ? hrtimer_interrupt+0x128/0x2c0 [ 2220.315202] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2220.330960] ? apic_timer_interrupt+0xf/0x20 [ 2220.337116] [ 2220.341263] ? native_safe_halt+0x2e/0x50 [ 2220.347305] kvm_wait+0xcd/0xe0 [ 2220.350381] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2220.355441] do_raw_spin_lock+0x177/0x190 [ 2220.361402] _raw_spin_lock+0x21/0x30 [ 2220.363379] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2220.370186] process_one_work+0x2c8/0x700 [ 2220.375099] worker_thread+0x296/0x6e0 [ 2220.376995] ? rescuer_thread+0x570/0x570 [ 2220.380352] kthread+0x1d1/0x200 [ 2220.382082] ? set_kthread_struct+0x70/0x70 [ 2220.392980] ret_from_fork+0x1f/0x30 [ 2232.061321] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 2232.064968] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 2232.071419] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2232.071722] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2232.071729] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2232.071731] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2232.071745] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2232.071748] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2232.071753] RAX: 000000000000088a RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2232.071755] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 2232.071756] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2232.071759] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 2232.071761] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 2232.071764] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 2232.071767] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2232.071769] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 2232.071777] Call Trace: [ 2232.071782] [ 2232.071801] ? show_regs.cold.9+0x22/0x2f [ 2232.071811] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2232.071817] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2232.071821] ? watchdog+0x40/0x40 [ 2232.071825] ? hrtimer_interrupt+0x128/0x2c0 [ 2232.071829] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2232.071837] ? apic_timer_interrupt+0xf/0x20 [ 2232.072095] [ 2232.072100] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2232.072410] do_raw_spin_lock+0x177/0x190 [ 2232.072418] _raw_spin_lock+0x21/0x30 [ 2232.072422] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2232.073363] ? do_raw_spin_unlock+0x75/0x190 [ 2232.084457] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2232.084552] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2232.084559] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2232.084562] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2232.084576] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2232.084580] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2232.084585] RAX: 0000000000007b8b RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2232.084588] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 2232.084590] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2232.084592] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 2232.084594] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 2232.084598] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 2232.084600] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2232.084603] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 2232.084972] Call Trace: [ 2232.084979] [ 2232.085431] ? show_regs.cold.9+0x22/0x2f [ 2232.085446] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2232.085452] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2232.085456] ? watchdog+0x40/0x40 [ 2232.085460] ? hrtimer_interrupt+0x128/0x2c0 [ 2232.085464] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2232.085479] ? apic_timer_interrupt+0xf/0x20 [ 2232.085484] [ 2232.085491] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2232.085500] do_raw_spin_lock+0x177/0x190 [ 2232.085843] _raw_spin_lock+0x21/0x30 [ 2232.123240] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2232.124247] ? do_wait_intr+0xf0/0xf0 [ 2232.124260] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2232.124810] mdc_close+0x53b/0xd70 [mdc] [ 2232.124851] lmv_close+0x18d/0x490 [lmv] [ 2232.124887] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 2232.125059] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2232.125178] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2232.125260] ll_md_close+0x274/0x800 [lustre] [ 2232.125343] ll_file_release+0x128/0xc50 [lustre] [ 2232.126944] __fput+0x10d/0x440 [ 2232.126957] ____fput+0x12/0x20 [ 2232.126962] task_work_run+0xc6/0x110 [ 2232.126971] exit_to_usermode_loop+0x1dd/0x1f0 [ 2232.126979] do_syscall_64+0x3d6/0x440 [ 2232.126984] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2232.126993] RIP: 0033:0x7ff54af45cd8 [ 2232.127001] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2232.127006] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2232.127011] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 2232.127013] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 2232.127015] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 2232.127018] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 2232.127020] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2232.410980] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2232.413798] ? do_raw_spin_unlock+0x75/0x190 [ 2232.415855] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2232.418732] ? do_wait_intr+0xf0/0xf0 [ 2232.420055] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2232.422816] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2232.425208] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2232.428383] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2232.431164] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2232.433786] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2232.436074] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2232.438345] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2232.441063] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2232.444429] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2232.447705] lmv_intent_open+0x338/0x1150 [lmv] [ 2232.450378] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2232.453050] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2232.455752] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2232.457775] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2232.460171] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2232.462897] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2232.465353] ? do_raw_spin_unlock+0x75/0x190 [ 2232.467074] ? _raw_spin_unlock+0x12/0x30 [ 2232.468296] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2232.470055] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2232.471952] ll_atomic_open+0x36a/0xd70 [lustre] [ 2232.473517] lookup_open+0x356/0xb40 [ 2232.474763] do_last+0x313/0xfc0 [ 2232.475825] ? nd_jump_root+0xe5/0x160 [ 2232.477145] ? path_init+0x437/0x520 [ 2232.478847] path_openat+0xf7/0x500 [ 2232.480100] ? do_raw_spin_unlock+0x75/0x190 [ 2232.481432] do_filp_open+0x99/0x140 [ 2232.483145] ? getname_flags+0x6e/0x330 [ 2232.484418] ? __check_object_size+0xff/0x256 [ 2232.486094] ? do_raw_spin_unlock+0x75/0x190 [ 2232.487382] ? _raw_spin_unlock+0x12/0x30 [ 2232.489239] do_sys_openat2+0x2b4/0x410 [ 2232.490383] do_sys_open+0x73/0xa0 [ 2232.491981] __x64_sys_openat+0x24/0x30 [ 2232.493184] do_syscall_64+0xc1/0x440 [ 2232.494315] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2232.495950] RIP: 0033:0x7ff54af45332 [ 2232.497114] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2232.502849] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2232.510333] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 2232.514810] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 2232.517601] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 2232.521249] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 2232.524981] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 2240.062878] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [ 2240.069056] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2240.098158] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2240.106769] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2240.111079] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2240.115440] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2240.131123] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2240.134343] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2240.142012] RDX: 0000000000005748 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 2240.148924] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 2240.155414] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 2240.159941] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2240.170154] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 2240.182841] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2240.187379] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 2240.191336] Call Trace: [ 2240.192998] [ 2240.194013] ? show_regs.cold.9+0x22/0x2f [ 2240.198093] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2240.201860] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2240.206077] ? watchdog+0x40/0x40 [ 2240.208902] ? hrtimer_interrupt+0x128/0x2c0 [ 2240.212892] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2240.218879] ? apic_timer_interrupt+0xf/0x20 [ 2240.223296] [ 2240.225123] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2240.229136] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2240.231093] do_raw_spin_lock+0x177/0x190 [ 2240.243435] _raw_spin_lock+0x21/0x30 [ 2240.246209] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2240.252623] obd_mod_free+0x46/0x140 [ptlrpc] [ 2240.255860] mdc_commit_open+0x91/0xb0 [mdc] [ 2240.257300] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2240.261193] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2240.264611] after_reply+0xe2c/0x1780 [ptlrpc] [ 2240.271239] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2240.283342] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2240.289365] ? do_wait_intr+0xf0/0xf0 [ 2240.296976] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2240.303273] kthread+0x1d1/0x200 [ 2240.304500] ? set_kthread_struct+0x70/0x70 [ 2240.318985] ret_from_fork+0x1f/0x30 [ 2244.575138] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 285s! [ 2244.583131] Showing busy workqueues and worker pools: [ 2244.591364] workqueue events: flags=0x0 [ 2244.594843] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2244.601811] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 2244.612971] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 2244.619433] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 2244.631150] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2244.639131] pending: vmstat_shepherd [ 2244.642065] workqueue events_long: flags=0x0 [ 2244.648046] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2244.657419] pending: lu_objects_flush [obdclass] [ 2244.665165] workqueue events_unbound: flags=0x2 [ 2244.671177] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 2244.677191] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2244.688719] workqueue events_power_efficient: flags=0x80 [ 2244.690967] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2244.695463] pending: neigh_periodic_work [ 2244.699173] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2244.710081] pending: do_cache_clean, check_lifetime [ 2244.728841] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2244.744752] pending: neigh_periodic_work [ 2244.751224] workqueue writeback: flags=0x4a [ 2244.759326] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2244.772457] pending: wb_workfn, wb_workfn [ 2244.779746] workqueue ipv6_addrconf: flags=0x40008 [ 2244.786288] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2244.789370] pending: addrconf_verify_work [ 2244.794321] workqueue knbd0-recv: flags=0x1a [ 2244.797100] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2244.802225] in-flight: 657:recv_work [ 2244.804245] workqueue ptlrpc_pinger: flags=0x2 [ 2244.806486] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2244.809302] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 2244.812112] pool 8: cpus=0-3 flags=0x4 nice=0 hung=285s workers=4 idle: 20831 12642 28859 [ 2244.815707] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2235s workers=2 idle: 659 [ 2248.062838] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 2248.089995] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2248.153475] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2248.164083] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2248.169549] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2248.173757] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2248.177180] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2248.188400] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2248.192179] RAX: 0000000000007e4d RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2248.195542] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 2248.198249] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 2248.201199] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 2248.204158] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 2248.206937] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 2248.209909] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2248.211937] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 2248.215161] Call Trace: [ 2248.216449] [ 2248.217315] ? show_regs.cold.9+0x22/0x2f [ 2248.219208] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2248.221181] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2248.222879] ? watchdog+0x40/0x40 [ 2248.224389] ? hrtimer_interrupt+0x128/0x2c0 [ 2248.226200] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2248.228409] ? apic_timer_interrupt+0xf/0x20 [ 2248.230173] [ 2248.231134] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2248.234301] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2248.236350] do_raw_spin_lock+0x177/0x190 [ 2248.237791] _raw_spin_lock+0x21/0x30 [ 2248.239012] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2248.241326] process_one_work+0x2c8/0x700 [ 2248.242597] worker_thread+0x296/0x6e0 [ 2248.243939] ? rescuer_thread+0x570/0x570 [ 2248.245319] kthread+0x1d1/0x200 [ 2248.246403] ? set_kthread_struct+0x70/0x70 [ 2248.247861] ret_from_fork+0x1f/0x30 [ 2260.062029] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 2260.066651] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2260.071169] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 2260.120398] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2260.120413] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2260.120416] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2260.120564] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2260.120571] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2260.120577] RAX: 000000000000660c RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2260.120580] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 2260.120582] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2260.120584] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 2260.120586] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 2260.120589] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 2260.120597] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2260.120599] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 2260.120607] Call Trace: [ 2260.120613] [ 2260.120618] ? show_regs.cold.9+0x22/0x2f [ 2260.120628] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2260.120636] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2260.120646] ? watchdog+0x40/0x40 [ 2260.120650] ? hrtimer_interrupt+0x128/0x2c0 [ 2260.120654] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2260.120663] ? apic_timer_interrupt+0xf/0x20 [ 2260.120668] [ 2260.120671] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2260.120678] do_raw_spin_lock+0x177/0x190 [ 2260.120683] _raw_spin_lock+0x21/0x30 [ 2260.120687] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2260.123817] ? do_raw_spin_unlock+0x75/0x190 [ 2260.123826] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2260.124045] ? do_wait_intr+0xf0/0xf0 [ 2260.124056] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2260.124268] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2260.140238] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2260.140315] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2260.140323] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2260.140326] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2260.140351] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2260.140355] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2260.140360] RAX: 0000000000007b83 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2260.140362] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 2260.140364] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2260.140366] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 2260.140368] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 2260.140371] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 2260.140374] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2260.140376] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 2260.140384] Call Trace: [ 2260.140389] [ 2260.140395] ? show_regs.cold.9+0x22/0x2f [ 2260.140403] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2260.140409] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2260.140413] ? watchdog+0x40/0x40 [ 2260.140417] ? hrtimer_interrupt+0x128/0x2c0 [ 2260.140420] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2260.140428] ? apic_timer_interrupt+0xf/0x20 [ 2260.140433] [ 2260.140435] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2260.140441] do_raw_spin_lock+0x177/0x190 [ 2260.140445] _raw_spin_lock+0x21/0x30 [ 2260.140450] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2260.141460] ? do_raw_spin_unlock+0x75/0x190 [ 2260.141466] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2260.141704] ? do_wait_intr+0xf0/0xf0 [ 2260.141712] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2260.158035] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2260.158375] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2260.167529] mdc_close+0x53b/0xd70 [mdc] [ 2260.167585] lmv_close+0x18d/0x490 [lmv] [ 2260.167615] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 2260.175287] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2260.175398] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2260.175430] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2260.175459] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2260.175593] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2260.177217] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2260.177261] lmv_intent_open+0x338/0x1150 [lmv] [ 2260.177293] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2260.200026] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2260.200166] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2260.200247] ll_md_close+0x274/0x800 [lustre] [ 2260.200331] ll_file_release+0x128/0xc50 [lustre] [ 2260.200965] __fput+0x10d/0x440 [ 2260.200976] ____fput+0x12/0x20 [ 2260.200980] task_work_run+0xc6/0x110 [ 2260.200988] exit_to_usermode_loop+0x1dd/0x1f0 [ 2260.200994] do_syscall_64+0x3d6/0x440 [ 2260.200999] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2260.201007] RIP: 0033:0x7ff54af45cd8 [ 2260.201015] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2260.201019] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2260.201024] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 2260.201026] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 2260.201028] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 2260.201031] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 2260.201033] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2261.022738] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2261.024717] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2261.032912] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2261.038223] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2261.044778] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2261.060138] ? do_raw_spin_unlock+0x75/0x190 [ 2261.067537] ? _raw_spin_unlock+0x12/0x30 [ 2261.078169] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2261.089038] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2261.097630] ll_atomic_open+0x36a/0xd70 [lustre] [ 2261.104892] lookup_open+0x356/0xb40 [ 2261.108007] do_last+0x313/0xfc0 [ 2261.113403] ? nd_jump_root+0xe5/0x160 [ 2261.116373] ? path_init+0x437/0x520 [ 2261.123043] path_openat+0xf7/0x500 [ 2261.130125] ? do_raw_spin_unlock+0x75/0x190 [ 2261.132889] do_filp_open+0x99/0x140 [ 2261.138875] ? getname_flags+0x6e/0x330 [ 2261.142839] ? __check_object_size+0xff/0x256 [ 2261.148799] ? do_raw_spin_unlock+0x75/0x190 [ 2261.150710] ? _raw_spin_unlock+0x12/0x30 [ 2261.154603] do_sys_openat2+0x2b4/0x410 [ 2261.156962] do_sys_open+0x73/0xa0 [ 2261.160581] __x64_sys_openat+0x24/0x30 [ 2261.163155] do_syscall_64+0xc1/0x440 [ 2261.165222] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2261.168568] RIP: 0033:0x7ff54af45332 [ 2261.170619] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2261.181138] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2261.185150] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 2261.190995] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 2261.194849] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 2261.199239] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 2261.213034] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 2272.055058] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [ 2272.067991] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2272.119785] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2272.124665] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2272.139967] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2272.144617] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2272.162529] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2272.168258] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2272.175073] RDX: 0000000000003251 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 2272.178497] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 2272.183698] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 2272.186848] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2272.192156] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 2272.203024] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2272.204878] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 2272.218141] Call Trace: [ 2272.219049] [ 2272.219798] ? show_regs.cold.9+0x22/0x2f [ 2272.222572] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2272.229185] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2272.231848] ? watchdog+0x40/0x40 [ 2272.234900] ? hrtimer_interrupt+0x128/0x2c0 [ 2272.237198] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2272.239680] ? apic_timer_interrupt+0xf/0x20 [ 2272.242738] [ 2272.243829] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2272.246047] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2272.250436] do_raw_spin_lock+0x177/0x190 [ 2272.251770] _raw_spin_lock+0x21/0x30 [ 2272.258820] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2272.262011] obd_mod_free+0x46/0x140 [ptlrpc] [ 2272.271693] mdc_commit_open+0x91/0xb0 [mdc] [ 2272.276842] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2272.281199] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2272.288920] after_reply+0xe2c/0x1780 [ptlrpc] [ 2272.293204] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2272.304039] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2272.305871] ? do_wait_intr+0xf0/0xf0 [ 2272.310664] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2272.312580] kthread+0x1d1/0x200 [ 2272.320391] ? set_kthread_struct+0x70/0x70 [ 2272.322626] ret_from_fork+0x1f/0x30 [ 2275.295052] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 316s! [ 2275.298035] Showing busy workqueues and worker pools: [ 2275.300255] workqueue events: flags=0x0 [ 2275.302276] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2275.304929] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 2275.308477] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 2275.311105] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 2275.315637] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2275.318268] pending: vmstat_shepherd [ 2275.319887] workqueue events_long: flags=0x0 [ 2275.321556] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2275.324177] pending: lu_objects_flush [obdclass] [ 2275.326334] workqueue events_unbound: flags=0x2 [ 2275.327985] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 2275.330422] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2275.339680] workqueue events_power_efficient: flags=0x80 [ 2275.345916] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2275.348071] pending: neigh_periodic_work [ 2275.367409] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2275.388691] pending: do_cache_clean, check_lifetime [ 2275.405926] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2275.413647] pending: neigh_periodic_work [ 2275.425145] workqueue writeback: flags=0x4a [ 2275.429698] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2275.438627] pending: wb_workfn, wb_workfn [ 2275.443531] workqueue xprtiod: flags=0xa [ 2275.446148] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 2275.450336] pending: xprt_autoclose [ 2275.454090] workqueue ipv6_addrconf: flags=0x40008 [ 2275.460850] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2275.463194] pending: addrconf_verify_work [ 2275.467619] workqueue knbd0-recv: flags=0x1a [ 2275.470881] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2275.476269] in-flight: 657:recv_work [ 2275.478277] workqueue ptlrpc_pinger: flags=0x2 [ 2275.482438] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2275.486813] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 2275.491174] pool 8: cpus=0-3 flags=0x4 nice=0 hung=316s workers=4 idle: 20831 12642 28859 [ 2275.496787] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2265s workers=2 idle: 659 [ 2276.064900] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 2276.079978] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2276.115566] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2276.121747] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2276.125751] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2276.129308] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2276.131900] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2276.140434] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2276.143515] RAX: 0000000000007107 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2276.146629] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 2276.149498] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 2276.152405] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 2276.159903] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 2276.164610] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 2276.168935] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2276.170993] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 2276.178276] Call Trace: [ 2276.180272] [ 2276.181093] ? show_regs.cold.9+0x22/0x2f [ 2276.182513] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2276.187854] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2276.190730] ? watchdog+0x40/0x40 [ 2276.192722] ? hrtimer_interrupt+0x128/0x2c0 [ 2276.195657] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2276.197903] ? apic_timer_interrupt+0xf/0x20 [ 2276.200121] [ 2276.200963] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2276.204176] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2276.211163] do_raw_spin_lock+0x177/0x190 [ 2276.216920] _raw_spin_lock+0x21/0x30 [ 2276.218250] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2276.222770] process_one_work+0x2c8/0x700 [ 2276.224227] worker_thread+0x296/0x6e0 [ 2276.232778] ? rescuer_thread+0x570/0x570 [ 2276.236026] kthread+0x1d1/0x200 [ 2276.237956] ? set_kthread_struct+0x70/0x70 [ 2276.241412] ret_from_fork+0x1f/0x30 [ 2288.058168] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 2288.061293] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 2288.061363] Modules linked in: [ 2288.078487] Modules linked in: lustre(O) [ 2288.079824] lustre(O) osp(O) ofd(O) lod(O) [ 2288.083834] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) [ 2288.085857] mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2288.085924] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2288.085931] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2288.085934] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2288.085948] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2288.085952] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2288.085957] RAX: 0000000000007210 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2288.085959] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 2288.085961] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2288.085963] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 2288.085965] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 2288.085968] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 2288.085971] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2288.085973] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 2288.085981] Call Trace: [ 2288.085986] [ 2288.085991] ? show_regs.cold.9+0x22/0x2f [ 2288.086001] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2288.086006] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2288.086010] ? watchdog+0x40/0x40 [ 2288.086014] ? hrtimer_interrupt+0x128/0x2c0 [ 2288.086017] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2288.086025] ? apic_timer_interrupt+0xf/0x20 [ 2288.086029] [ 2288.086032] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2288.086038] do_raw_spin_lock+0x177/0x190 [ 2288.088974] obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2288.089034] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2288.089040] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2288.089042] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2288.089054] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2288.089058] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2288.089062] RAX: 0000000000000e94 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2288.089269] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 2288.089274] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2288.089276] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 2288.089278] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 2288.089281] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 2288.089284] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2288.089287] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 2288.089294] Call Trace: [ 2288.089299] [ 2288.089350] ? show_regs.cold.9+0x22/0x2f [ 2288.089358] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2288.089364] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2288.089367] ? watchdog+0x40/0x40 [ 2288.089371] ? hrtimer_interrupt+0x128/0x2c0 [ 2288.089375] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2288.089382] ? apic_timer_interrupt+0xf/0x20 [ 2288.089386] [ 2288.089388] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2288.089394] do_raw_spin_lock+0x177/0x190 [ 2288.089398] _raw_spin_lock+0x21/0x30 [ 2288.089402] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2288.089695] ? do_raw_spin_unlock+0x75/0x190 [ 2288.089701] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2288.089903] ? do_wait_intr+0xf0/0xf0 [ 2288.089911] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2288.090352] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2288.090570] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2288.090794] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2288.090904] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2288.090941] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2288.090969] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2288.090997] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2288.091313] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2288.091531] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2288.091560] lmv_intent_open+0x338/0x1150 [lmv] [ 2288.091589] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2288.091695] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2288.091780] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2288.091805] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2288.091897] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2288.091983] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2288.092228] ? do_raw_spin_unlock+0x75/0x190 [ 2288.092237] ? _raw_spin_unlock+0x12/0x30 [ 2288.092252] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2288.092279] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2288.092298] ll_atomic_open+0x36a/0xd70 [lustre] [ 2288.092380] lookup_open+0x356/0xb40 [ 2288.092388] do_last+0x313/0xfc0 [ 2288.092393] ? nd_jump_root+0xe5/0x160 [ 2288.092397] ? path_init+0x437/0x520 [ 2288.092401] path_openat+0xf7/0x500 [ 2288.092405] ? do_raw_spin_unlock+0x75/0x190 [ 2288.092409] do_filp_open+0x99/0x140 [ 2288.092416] ? getname_flags+0x6e/0x330 [ 2288.092420] ? __check_object_size+0xff/0x256 [ 2288.092426] ? do_raw_spin_unlock+0x75/0x190 [ 2288.092430] ? _raw_spin_unlock+0x12/0x30 [ 2288.092435] do_sys_openat2+0x2b4/0x410 [ 2288.092439] do_sys_open+0x73/0xa0 [ 2288.092443] __x64_sys_openat+0x24/0x30 [ 2288.092446] do_syscall_64+0xc1/0x440 [ 2288.092451] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2288.092457] RIP: 0033:0x7ff54af45332 [ 2288.092463] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2288.092467] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2288.092472] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 2288.092475] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 2288.092477] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 2288.092479] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 2288.092481] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 2288.383506] _raw_spin_lock+0x21/0x30 [ 2288.384550] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2288.385785] ? do_raw_spin_unlock+0x75/0x190 [ 2288.387144] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2288.388441] ? do_wait_intr+0xf0/0xf0 [ 2288.389308] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2288.390767] mdc_close+0x53b/0xd70 [mdc] [ 2288.392004] lmv_close+0x18d/0x490 [lmv] [ 2288.393412] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 2288.395275] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2288.396814] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2288.398439] ll_md_close+0x274/0x800 [lustre] [ 2288.400141] ll_file_release+0x128/0xc50 [lustre] [ 2288.401876] __fput+0x10d/0x440 [ 2288.403328] ____fput+0x12/0x20 [ 2288.404484] task_work_run+0xc6/0x110 [ 2288.405858] exit_to_usermode_loop+0x1dd/0x1f0 [ 2288.407118] do_syscall_64+0x3d6/0x440 [ 2288.407886] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2288.409255] RIP: 0033:0x7ff54af45cd8 [ 2288.410571] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2288.414925] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2288.416627] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 2288.418177] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 2288.419493] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 2288.420953] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 2288.422696] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2300.057237] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [ 2300.062733] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2300.108461] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2300.121286] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2300.132290] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2300.139376] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2300.155607] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2300.165388] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2300.171603] RDX: 000000000000130e RSI: 0000000000000003 RDI: ffff96848811cae0 [ 2300.176898] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 2300.179781] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 2300.182798] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2300.186618] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 2300.190548] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2300.192883] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 2300.196063] Call Trace: [ 2300.197324] [ 2300.198062] ? show_regs.cold.9+0x22/0x2f [ 2300.199683] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2300.203213] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2300.205437] ? watchdog+0x40/0x40 [ 2300.206489] ? hrtimer_interrupt+0x128/0x2c0 [ 2300.207648] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2300.209522] ? apic_timer_interrupt+0xf/0x20 [ 2300.217406] [ 2300.220419] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2300.227329] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2300.232893] do_raw_spin_lock+0x177/0x190 [ 2300.234200] _raw_spin_lock+0x21/0x30 [ 2300.235406] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2300.244055] obd_mod_free+0x46/0x140 [ptlrpc] [ 2300.248700] mdc_commit_open+0x91/0xb0 [mdc] [ 2300.252765] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2300.259492] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2300.262855] after_reply+0xe2c/0x1780 [ptlrpc] [ 2300.265701] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2300.269368] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2300.276774] ? do_wait_intr+0xf0/0xf0 [ 2300.278064] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2300.285196] kthread+0x1d1/0x200 [ 2300.286718] ? set_kthread_struct+0x70/0x70 [ 2300.290326] ret_from_fork+0x1f/0x30 [ 2304.059116] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 2304.062863] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2304.074843] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2304.077496] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2304.079422] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2304.080952] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2304.082187] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2304.086032] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2304.087506] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2304.088970] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 2304.090358] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 2304.091860] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 2304.093385] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 2304.094977] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 2304.096674] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2304.098153] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 2304.099634] Call Trace: [ 2304.100354] [ 2304.100760] ? show_regs.cold.9+0x22/0x2f [ 2304.101567] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2304.102567] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2304.103608] ? watchdog+0x40/0x40 [ 2304.104395] ? hrtimer_interrupt+0x128/0x2c0 [ 2304.105359] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2304.106363] ? apic_timer_interrupt+0xf/0x20 [ 2304.107522] [ 2304.108018] ? native_safe_halt+0x2e/0x50 [ 2304.108797] kvm_wait+0xcd/0xe0 [ 2304.109569] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2304.110647] do_raw_spin_lock+0x177/0x190 [ 2304.111546] _raw_spin_lock+0x21/0x30 [ 2304.112350] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2304.113790] process_one_work+0x2c8/0x700 [ 2304.114764] worker_thread+0x296/0x6e0 [ 2304.115592] ? rescuer_thread+0x570/0x570 [ 2304.116535] kthread+0x1d1/0x200 [ 2304.117311] ? set_kthread_struct+0x70/0x70 [ 2304.118225] ret_from_fork+0x1f/0x30 [ 2305.503176] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 346s! [ 2305.505787] Showing busy workqueues and worker pools: [ 2305.508356] workqueue events: flags=0x0 [ 2305.510399] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2305.514087] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 2305.518947] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 2305.522340] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 2305.527576] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2305.530826] pending: vmstat_shepherd [ 2305.532807] workqueue events_long: flags=0x0 [ 2305.534852] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2305.537887] pending: lu_objects_flush [obdclass] [ 2305.540804] workqueue events_unbound: flags=0x2 [ 2305.542737] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 2305.545320] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2305.550128] workqueue events_power_efficient: flags=0x80 [ 2305.552698] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2305.555822] pending: neigh_periodic_work [ 2305.557833] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2305.561053] pending: do_cache_clean, check_lifetime [ 2305.563281] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2305.566304] pending: neigh_periodic_work [ 2305.568366] workqueue writeback: flags=0x4a [ 2305.570282] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2305.573161] pending: wb_workfn, wb_workfn [ 2305.575058] workqueue xprtiod: flags=0xa [ 2305.577167] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 2305.580337] pending: xprt_autoclose [ 2305.582295] workqueue ipv6_addrconf: flags=0x40008 [ 2305.584493] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2305.587439] pending: addrconf_verify_work [ 2305.589699] workqueue knbd0-recv: flags=0x1a [ 2305.591417] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2305.594433] in-flight: 657:recv_work [ 2305.596314] workqueue ptlrpc_pinger: flags=0x2 [ 2305.598440] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2305.601343] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 2305.604464] pool 8: cpus=0-3 flags=0x4 nice=0 hung=346s workers=4 idle: 20831 12642 28859 [ 2305.608449] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2295s workers=2 idle: 659 [ 2316.060026] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 2316.064144] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2316.066401] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [ 2316.088597] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2316.088610] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2316.088613] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2316.088626] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2316.088630] RSP: 0000:ffffaf9fc15234f0 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 2316.088636] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2316.088638] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 2316.088640] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2316.088643] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 2316.088645] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 2316.088648] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 2316.088651] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2316.088653] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 2316.088662] Call Trace: [ 2316.088667] [ 2316.088674] ? show_regs.cold.9+0x22/0x2f [ 2316.088685] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2316.088692] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2316.088696] ? watchdog+0x40/0x40 [ 2316.088701] ? hrtimer_interrupt+0x128/0x2c0 [ 2316.088705] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2316.088712] ? apic_timer_interrupt+0xf/0x20 [ 2316.088716] [ 2316.088719] ? native_safe_halt+0x2e/0x50 [ 2316.088723] kvm_wait+0xcd/0xe0 [ 2316.088731] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2316.088741] do_raw_spin_lock+0x177/0x190 [ 2316.088746] _raw_spin_lock+0x21/0x30 [ 2316.088750] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2316.089237] ? do_raw_spin_unlock+0x75/0x190 [ 2316.089243] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2316.089522] ? do_wait_intr+0xf0/0xf0 [ 2316.089532] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2316.089745] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2316.090122] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2316.090366] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2316.090554] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2316.090594] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2316.090626] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2316.090654] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2316.090739] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2316.091091] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2316.091122] lmv_intent_open+0x338/0x1150 [lmv] [ 2316.091154] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2316.091240] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2316.094709] Modules linked in: [ 2316.099582] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2316.099632] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2316.099756] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2316.099912] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2316.100011] ? do_raw_spin_unlock+0x75/0x190 [ 2316.100020] ? _raw_spin_unlock+0x12/0x30 [ 2316.100028] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2316.100058] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2316.100080] ll_atomic_open+0x36a/0xd70 [lustre] [ 2316.100168] lookup_open+0x356/0xb40 [ 2316.100178] do_last+0x313/0xfc0 [ 2316.100183] ? nd_jump_root+0xe5/0x160 [ 2316.100188] ? path_init+0x437/0x520 [ 2316.100193] path_openat+0xf7/0x500 [ 2316.100198] ? do_raw_spin_unlock+0x75/0x190 [ 2316.100202] do_filp_open+0x99/0x140 [ 2316.100209] ? getname_flags+0x6e/0x330 [ 2316.100214] ? __check_object_size+0xff/0x256 [ 2316.100222] ? do_raw_spin_unlock+0x75/0x190 [ 2316.100227] ? _raw_spin_unlock+0x12/0x30 [ 2316.100232] do_sys_openat2+0x2b4/0x410 [ 2316.100239] do_sys_open+0x73/0xa0 [ 2316.100243] __x64_sys_openat+0x24/0x30 [ 2316.100248] do_syscall_64+0xc1/0x440 [ 2316.100255] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2316.100262] RIP: 0033:0x7ff54af45332 [ 2316.100271] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2316.100276] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2316.100281] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 2316.100284] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 2316.100286] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 2316.100288] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 2316.100290] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 2316.267658] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2316.281745] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2316.284275] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2316.286431] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2316.287923] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2316.294205] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2316.296288] RAX: 0000000000004a13 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2316.298053] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 2316.299812] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2316.309956] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 2316.315297] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 2316.317222] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 2316.319215] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2316.321486] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 2316.323769] Call Trace: [ 2316.324626] [ 2316.325270] ? show_regs.cold.9+0x22/0x2f [ 2316.329134] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2316.330753] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2316.332476] ? watchdog+0x40/0x40 [ 2316.336983] ? hrtimer_interrupt+0x128/0x2c0 [ 2316.339686] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2316.346322] ? apic_timer_interrupt+0xf/0x20 [ 2316.353304] [ 2316.354500] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2316.356462] do_raw_spin_lock+0x177/0x190 [ 2316.363351] _raw_spin_lock+0x21/0x30 [ 2316.368454] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2316.374946] ? do_raw_spin_unlock+0x75/0x190 [ 2316.380336] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2316.382360] ? do_wait_intr+0xf0/0xf0 [ 2316.383501] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2316.389442] mdc_close+0x53b/0xd70 [mdc] [ 2316.390526] lmv_close+0x18d/0x490 [lmv] [ 2316.391603] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 2316.395239] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2316.397113] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2316.402783] ll_md_close+0x274/0x800 [lustre] [ 2316.404197] ll_file_release+0x128/0xc50 [lustre] [ 2316.406214] __fput+0x10d/0x440 [ 2316.407896] ____fput+0x12/0x20 [ 2316.409203] task_work_run+0xc6/0x110 [ 2316.410703] exit_to_usermode_loop+0x1dd/0x1f0 [ 2316.412631] do_syscall_64+0x3d6/0x440 [ 2316.413981] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2316.415854] RIP: 0033:0x7ff54af45cd8 [ 2316.417043] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2316.423824] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2316.426445] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 2316.428384] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 2316.430210] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 2316.432632] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 2316.434542] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2332.059068] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 2332.069259] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 2332.069404] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2332.069483] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2332.069490] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2332.069496] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2332.069914] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2332.069925] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2332.069930] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2332.069935] RAX: 0000000000000339 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2332.069938] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 2332.069940] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 2332.069943] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 2332.069945] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 2332.069949] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 2332.069952] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2332.069954] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 2332.069963] Call Trace: [ 2332.070094] [ 2332.070099] ? show_regs.cold.9+0x22/0x2f [ 2332.070111] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2332.070117] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2332.070121] ? watchdog+0x40/0x40 [ 2332.070126] ? hrtimer_interrupt+0x128/0x2c0 [ 2332.070130] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2332.070146] ? apic_timer_interrupt+0xf/0x20 [ 2332.070151] [ 2332.070154] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2332.070160] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2332.070166] do_raw_spin_lock+0x177/0x190 [ 2332.070171] _raw_spin_lock+0x21/0x30 [ 2332.070180] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2332.070440] process_one_work+0x2c8/0x700 [ 2332.070454] worker_thread+0x296/0x6e0 [ 2332.070459] ? rescuer_thread+0x570/0x570 [ 2332.070463] kthread+0x1d1/0x200 [ 2332.070468] ? set_kthread_struct+0x70/0x70 [ 2332.070472] ret_from_fork+0x1f/0x30 [ 2332.586893] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2332.667896] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2332.677731] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2332.686049] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2332.695559] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2332.731475] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2332.744315] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2332.747548] RDX: 0000000000007448 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 2332.754618] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 2332.758263] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 2332.770700] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2332.776712] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 2332.781271] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2332.793717] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 2332.802732] Call Trace: [ 2332.803652] [ 2332.804431] ? show_regs.cold.9+0x22/0x2f [ 2332.808795] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2332.816185] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2332.818042] ? watchdog+0x40/0x40 [ 2332.821007] ? hrtimer_interrupt+0x128/0x2c0 [ 2332.824585] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2332.826402] ? apic_timer_interrupt+0xf/0x20 [ 2332.836178] [ 2332.837817] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2332.845356] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2332.853024] do_raw_spin_lock+0x177/0x190 [ 2332.855992] _raw_spin_lock+0x21/0x30 [ 2332.866060] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2332.875129] obd_mod_free+0x46/0x140 [ptlrpc] [ 2332.884831] mdc_commit_open+0x91/0xb0 [mdc] [ 2332.887862] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2332.897525] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2332.911469] after_reply+0xe2c/0x1780 [ptlrpc] [ 2332.918561] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2332.923954] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2332.926252] ? do_wait_intr+0xf0/0xf0 [ 2332.928103] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2332.931081] kthread+0x1d1/0x200 [ 2332.932834] ? set_kthread_struct+0x70/0x70 [ 2332.934759] ret_from_fork+0x1f/0x30 [ 2335.713678] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 376s! [ 2335.726105] Showing busy workqueues and worker pools: [ 2335.731640] workqueue events: flags=0x0 [ 2335.734128] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2335.736890] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 2335.749242] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 2335.751634] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 2335.774141] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2335.778630] pending: vmstat_shepherd [ 2335.785433] workqueue events_long: flags=0x0 [ 2335.791400] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2335.797264] pending: lu_objects_flush [obdclass] [ 2335.801346] workqueue events_unbound: flags=0x2 [ 2335.810907] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 2335.820020] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2335.828759] workqueue events_power_efficient: flags=0x80 [ 2335.832598] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2335.840997] pending: neigh_periodic_work [ 2335.844127] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2335.859527] pending: do_cache_clean, check_lifetime [ 2335.865929] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2335.872040] pending: neigh_periodic_work [ 2335.875017] workqueue writeback: flags=0x4a [ 2335.884336] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2335.891356] pending: wb_workfn, wb_workfn [ 2335.893922] workqueue xprtiod: flags=0xa [ 2335.902905] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 2335.905301] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 2335.918100] workqueue ipv6_addrconf: flags=0x40008 [ 2335.925515] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2335.930566] pending: addrconf_verify_work [ 2335.942238] workqueue knbd0-recv: flags=0x1a [ 2335.945970] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2335.955219] in-flight: 657:recv_work [ 2335.958107] workqueue ptlrpc_pinger: flags=0x2 [ 2335.966853] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2335.975298] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 2335.979263] pool 8: cpus=0-3 flags=0x4 nice=0 hung=377s workers=4 idle: 20831 12642 28859 [ 2335.987925] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2326s workers=2 idle: 659 [ 2344.058055] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [ 2344.061203] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [ 2344.062369] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2344.062442] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2344.062450] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2344.062453] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2344.062501] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2344.062505] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2344.062509] RAX: 0000000000006288 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2344.062511] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 2344.062513] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2344.062515] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 2344.062516] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 2344.062519] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 2344.062522] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2344.062524] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 2344.062531] Call Trace: [ 2344.062547] [ 2344.062554] ? show_regs.cold.9+0x22/0x2f [ 2344.062564] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2344.062571] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2344.062574] ? watchdog+0x40/0x40 [ 2344.062578] ? hrtimer_interrupt+0x128/0x2c0 [ 2344.062582] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2344.062591] ? apic_timer_interrupt+0xf/0x20 [ 2344.062807] [ 2344.062813] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2344.062822] do_raw_spin_lock+0x177/0x190 [ 2344.062827] _raw_spin_lock+0x21/0x30 [ 2344.062838] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2344.063596] ? do_raw_spin_unlock+0x75/0x190 [ 2344.063602] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2344.063946] ? do_wait_intr+0xf0/0xf0 [ 2344.063956] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2344.068072] mdc_close+0x53b/0xd70 [mdc] [ 2344.068115] lmv_close+0x18d/0x490 [lmv] [ 2344.068154] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 2344.068339] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2344.068490] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2344.070123] ll_md_close+0x274/0x800 [lustre] [ 2344.070211] ll_file_release+0x128/0xc50 [lustre] [ 2344.070292] __fput+0x10d/0x440 [ 2344.070303] ____fput+0x12/0x20 [ 2344.070308] task_work_run+0xc6/0x110 [ 2344.070316] exit_to_usermode_loop+0x1dd/0x1f0 [ 2344.070325] do_syscall_64+0x3d6/0x440 [ 2344.070401] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2344.070412] RIP: 0033:0x7ff54af45cd8 [ 2344.070421] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2344.070425] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2344.070431] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 2344.070433] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 2344.070435] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 2344.070437] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 2344.070440] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2344.578309] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2344.634191] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2344.640952] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2344.646654] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2344.650514] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2344.668066] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2344.671744] RAX: 000000000000108c RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2344.677474] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 2344.683498] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2344.691887] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 2344.700975] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 2344.709623] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 2344.715198] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2344.719756] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 2344.730223] Call Trace: [ 2344.733468] [ 2344.734247] ? show_regs.cold.9+0x22/0x2f [ 2344.735839] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2344.737507] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2344.740206] ? watchdog+0x40/0x40 [ 2344.746743] ? hrtimer_interrupt+0x128/0x2c0 [ 2344.750064] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2344.762367] ? apic_timer_interrupt+0xf/0x20 [ 2344.771150] [ 2344.776090] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2344.781400] do_raw_spin_lock+0x177/0x190 [ 2344.786014] _raw_spin_lock+0x21/0x30 [ 2344.794133] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2344.811124] ? do_raw_spin_unlock+0x75/0x190 [ 2344.819894] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2344.829406] ? do_wait_intr+0xf0/0xf0 [ 2344.830785] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2344.832610] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2344.848731] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2344.852965] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2344.859933] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2344.864154] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2344.868008] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2344.869641] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2344.874777] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2344.878708] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2344.883145] lmv_intent_open+0x338/0x1150 [lmv] [ 2344.884643] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2344.886206] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2344.904119] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2344.909761] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2344.915176] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2344.918107] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2344.921123] ? do_raw_spin_unlock+0x75/0x190 [ 2344.923704] ? _raw_spin_unlock+0x12/0x30 [ 2344.926379] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2344.928080] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2344.934466] ll_atomic_open+0x36a/0xd70 [lustre] [ 2344.939708] lookup_open+0x356/0xb40 [ 2344.941242] do_last+0x313/0xfc0 [ 2344.942616] ? nd_jump_root+0xe5/0x160 [ 2344.945101] ? path_init+0x437/0x520 [ 2344.948537] path_openat+0xf7/0x500 [ 2344.955327] ? do_raw_spin_unlock+0x75/0x190 [ 2344.962261] do_filp_open+0x99/0x140 [ 2344.971048] ? getname_flags+0x6e/0x330 [ 2344.972380] ? __check_object_size+0xff/0x256 [ 2344.980361] ? do_raw_spin_unlock+0x75/0x190 [ 2344.983752] ? _raw_spin_unlock+0x12/0x30 [ 2344.987206] do_sys_openat2+0x2b4/0x410 [ 2344.988797] do_sys_open+0x73/0xa0 [ 2344.990117] __x64_sys_openat+0x24/0x30 [ 2344.991768] do_syscall_64+0xc1/0x440 [ 2344.992936] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2344.998088] RIP: 0033:0x7ff54af45332 [ 2345.002831] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2345.017015] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2345.022221] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 2345.026918] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 2345.030118] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 2345.032674] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 2345.035662] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 2360.060398] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 2360.062710] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 2360.066584] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2360.066749] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2360.066756] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2360.066760] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2360.066774] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2360.066777] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2360.066782] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2360.066784] RDX: 00000000000076a2 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 2360.066786] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 2360.066789] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 2360.066791] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2360.066794] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 2360.066796] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2360.066799] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 2360.066805] Call Trace: [ 2360.067220] [ 2360.067227] ? show_regs.cold.9+0x22/0x2f [ 2360.067236] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2360.067242] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2360.067246] ? watchdog+0x40/0x40 [ 2360.067250] ? hrtimer_interrupt+0x128/0x2c0 [ 2360.067254] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2360.067260] ? apic_timer_interrupt+0xf/0x20 [ 2360.067264] [ 2360.067303] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2360.067310] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2360.067314] do_raw_spin_lock+0x177/0x190 [ 2360.067322] _raw_spin_lock+0x21/0x30 [ 2360.067328] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2360.068262] obd_mod_free+0x46/0x140 [ptlrpc] [ 2360.068953] mdc_commit_open+0x91/0xb0 [mdc] [ 2360.068996] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2360.069198] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2360.069519] after_reply+0xe2c/0x1780 [ptlrpc] [ 2360.071003] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2360.071223] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2360.071802] ? do_wait_intr+0xf0/0xf0 [ 2360.071812] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2360.083264] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2360.083353] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2360.083361] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2360.083368] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2360.083798] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2360.083810] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2360.083814] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2360.083819] RAX: 0000000000007984 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2360.083821] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 2360.083824] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 2360.083826] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 2360.083828] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 2360.083832] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 2360.083836] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2360.083838] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 2360.083845] Call Trace: [ 2360.084110] [ 2360.084117] ? show_regs.cold.9+0x22/0x2f [ 2360.084126] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2360.084133] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2360.084137] ? watchdog+0x40/0x40 [ 2360.084142] ? hrtimer_interrupt+0x128/0x2c0 [ 2360.084145] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2360.084154] ? apic_timer_interrupt+0xf/0x20 [ 2360.084158] [ 2360.084161] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2360.084166] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2360.084171] do_raw_spin_lock+0x177/0x190 [ 2360.084176] _raw_spin_lock+0x21/0x30 [ 2360.084180] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2360.084576] process_one_work+0x2c8/0x700 [ 2360.084583] worker_thread+0x296/0x6e0 [ 2360.084587] ? rescuer_thread+0x570/0x570 [ 2360.084591] kthread+0x1d1/0x200 [ 2360.084594] ? set_kthread_struct+0x70/0x70 [ 2360.084598] ret_from_fork+0x1f/0x30 [ 2360.533479] kthread+0x1d1/0x200 [ 2360.534660] ? set_kthread_struct+0x70/0x70 [ 2360.537962] ret_from_fork+0x1f/0x30 [ 2366.431159] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 407s! [ 2366.436966] Showing busy workqueues and worker pools: [ 2366.440231] workqueue events: flags=0x0 [ 2366.442600] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2366.446699] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 2366.453094] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 2366.455543] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 2366.463118] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2366.469871] pending: vmstat_shepherd [ 2366.471883] workqueue events_long: flags=0x0 [ 2366.477271] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2366.484660] pending: lu_objects_flush [obdclass] [ 2366.486992] workqueue events_unbound: flags=0x2 [ 2366.489855] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 2366.492832] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2366.512152] workqueue events_power_efficient: flags=0x80 [ 2366.520938] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2366.526077] pending: neigh_periodic_work [ 2366.527608] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2366.532862] pending: do_cache_clean, check_lifetime [ 2366.538287] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2366.542851] pending: neigh_periodic_work [ 2366.545886] workqueue writeback: flags=0x4a [ 2366.553090] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2366.557339] pending: wb_workfn, wb_workfn [ 2366.559213] workqueue xprtiod: flags=0xa [ 2366.562152] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 2366.564880] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 2366.570136] workqueue ipv6_addrconf: flags=0x40008 [ 2366.572508] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2366.576437] pending: addrconf_verify_work [ 2366.580108] workqueue knbd0-recv: flags=0x1a [ 2366.583240] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2366.589039] in-flight: 657:recv_work [ 2366.591875] workqueue ptlrpc_pinger: flags=0x2 [ 2366.600961] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2366.606418] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 2366.611904] pool 8: cpus=0-3 flags=0x4 nice=0 hung=407s workers=4 idle: 20831 12642 28859 [ 2366.616296] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2356s workers=2 idle: 659 [ 2372.060237] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [ 2372.064446] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [ 2372.070565] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2372.070643] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2372.070650] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2372.070653] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2372.070668] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2372.070673] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2372.070678] RAX: 0000000000007e12 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2372.070681] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 2372.070683] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2372.070684] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 2372.070687] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 2372.070691] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 2372.070693] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2372.070696] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 2372.070705] Call Trace: [ 2372.070711] [ 2372.070718] ? show_regs.cold.9+0x22/0x2f [ 2372.070727] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2372.070733] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2372.070737] ? watchdog+0x40/0x40 [ 2372.070741] ? hrtimer_interrupt+0x128/0x2c0 [ 2372.070745] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2372.070754] ? apic_timer_interrupt+0xf/0x20 [ 2372.070758] [ 2372.070761] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2372.070767] do_raw_spin_lock+0x177/0x190 [ 2372.070772] _raw_spin_lock+0x21/0x30 [ 2372.070777] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2372.073300] ? do_raw_spin_unlock+0x75/0x190 [ 2372.073310] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2372.079741] Modules linked in: [ 2372.131757] ? do_wait_intr+0xf0/0xf0 [ 2372.131778] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2372.136871] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) [ 2372.140952] mdc_close+0x53b/0xd70 [mdc] [ 2372.156406] obdecho(O) [ 2372.167959] lmv_close+0x18d/0x490 [lmv] [ 2372.171624] mgc(O) [ 2372.174770] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 2372.175005] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2372.175114] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2372.175199] ll_md_close+0x274/0x800 [lustre] [ 2372.175408] ll_file_release+0x128/0xc50 [lustre] [ 2372.175500] __fput+0x10d/0x440 [ 2372.175511] ____fput+0x12/0x20 [ 2372.175515] task_work_run+0xc6/0x110 [ 2372.175523] exit_to_usermode_loop+0x1dd/0x1f0 [ 2372.175531] do_syscall_64+0x3d6/0x440 [ 2372.175536] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2372.175546] RIP: 0033:0x7ff54af45cd8 [ 2372.175719] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2372.175725] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2372.175731] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 2372.175734] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 2372.175736] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 2372.175738] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 2372.175740] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2372.382432] mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2372.401022] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2372.406205] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2372.409228] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2372.411752] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2372.419398] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2372.421019] RAX: 0000000000005106 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2372.424143] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 2372.427397] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2372.432154] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 2372.434795] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 2372.437912] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 2372.441821] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2372.444007] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 2372.446159] Call Trace: [ 2372.449043] [ 2372.449835] ? show_regs.cold.9+0x22/0x2f [ 2372.451450] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2372.453436] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2372.455067] ? watchdog+0x40/0x40 [ 2372.456698] ? hrtimer_interrupt+0x128/0x2c0 [ 2372.458560] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2372.461537] ? apic_timer_interrupt+0xf/0x20 [ 2372.463505] [ 2372.464820] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2372.469031] do_raw_spin_lock+0x177/0x190 [ 2372.472056] _raw_spin_lock+0x21/0x30 [ 2372.473592] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2372.478765] ? do_raw_spin_unlock+0x75/0x190 [ 2372.479909] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2372.481543] ? do_wait_intr+0xf0/0xf0 [ 2372.482592] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2372.484590] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2372.486600] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2372.488792] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2372.491196] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2372.492943] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2372.494245] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2372.495987] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2372.498678] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2372.500899] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2372.502792] lmv_intent_open+0x338/0x1150 [lmv] [ 2372.504525] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2372.506396] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2372.508368] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2372.509994] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2372.511798] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2372.513798] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2372.515548] ? do_raw_spin_unlock+0x75/0x190 [ 2372.517070] ? _raw_spin_unlock+0x12/0x30 [ 2372.517836] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2372.519216] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2372.521159] ll_atomic_open+0x36a/0xd70 [lustre] [ 2372.522808] lookup_open+0x356/0xb40 [ 2372.524490] do_last+0x313/0xfc0 [ 2372.525654] ? nd_jump_root+0xe5/0x160 [ 2372.527026] ? path_init+0x437/0x520 [ 2372.528731] path_openat+0xf7/0x500 [ 2372.529778] ? do_raw_spin_unlock+0x75/0x190 [ 2372.531315] do_filp_open+0x99/0x140 [ 2372.533239] ? getname_flags+0x6e/0x330 [ 2372.534280] ? __check_object_size+0xff/0x256 [ 2372.535469] ? do_raw_spin_unlock+0x75/0x190 [ 2372.536533] ? _raw_spin_unlock+0x12/0x30 [ 2372.537386] do_sys_openat2+0x2b4/0x410 [ 2372.538206] do_sys_open+0x73/0xa0 [ 2372.539954] __x64_sys_openat+0x24/0x30 [ 2372.540930] do_syscall_64+0xc1/0x440 [ 2372.543271] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2372.544625] RIP: 0033:0x7ff54af45332 [ 2372.546028] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2372.553966] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2372.556928] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 2372.559893] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 2372.562185] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 2372.566981] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 2372.570070] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 2373.851617] rcu: INFO: rcu_sched self-detected stall on CPU [ 2373.860817] rcu: 0-...!: (218991 ticks this GP) idle=98a/1/0x4000000000000002 softirq=125154/125154 fqs=959 [ 2373.871619] (t=420011 jiffies g=253585 q=2808) [ 2373.875719] rcu: rcu_sched kthread starved for 414971 jiffies! g253585 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [ 2373.888160] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 2373.897947] rcu: RCU grace-period kthread stack dump: [ 2373.900165] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 2373.911988] Call Trace: [ 2373.912818] __schedule+0x351/0xcb0 [ 2373.913986] schedule+0xc0/0x180 [ 2373.924858] schedule_timeout+0xb4/0x190 [ 2373.932472] ? do_raw_spin_unlock+0x75/0x190 [ 2373.939669] ? __next_timer_interrupt+0x160/0x160 [ 2373.942118] ? prepare_to_swait_event+0xf3/0x140 [ 2373.945419] rcu_gp_kthread+0x8dd/0x15b0 [ 2373.947261] ? force_qs_rnp+0x2c0/0x2c0 [ 2373.951114] kthread+0x1d1/0x200 [ 2373.952227] ? set_kthread_struct+0x70/0x70 [ 2373.956208] ret_from_fork+0x1f/0x30 [ 2373.960166] rcu: Stack dump where RCU GP kthread last ran: [ 2373.964548] Sending NMI from CPU 0 to CPUs 2: [ 2373.976396] INFO: NMI handler (nmi_cpu_backtrace_handler) took too long to run: 1.924 msecs [ 2373.978861] NMI backtrace for cpu 2 [ 2373.978870] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2373.978872] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2373.978873] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2373.978876] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2373.978878] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2373.978880] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 [ 2373.978882] RAX: 0000000000004d08 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2373.978883] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 2373.978885] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 2373.978886] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 2373.978888] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 2373.978889] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 2373.978890] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2373.978892] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 2373.978893] Call Trace: [ 2373.978894] [ 2373.978895] ? show_regs.cold.9+0x22/0x2f [ 2373.978896] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 2373.978897] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 2373.978898] ? nmi_handle+0xbd/0x210 [ 2373.978899] ? default_do_nmi+0x4b/0x1b0 [ 2373.978900] ? do_nmi+0x280/0x380 [ 2373.978901] ? end_repeat_nmi+0x16/0x1f [ 2373.978902] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2373.978904] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2373.978905] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2373.978906] [ 2373.978907] do_raw_spin_lock+0x177/0x190 [ 2373.978908] _raw_spin_lock+0x21/0x30 [ 2373.978909] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2373.978910] process_one_work+0x2c8/0x700 [ 2373.978911] worker_thread+0x296/0x6e0 [ 2373.978912] ? rescuer_thread+0x570/0x570 [ 2373.978913] kthread+0x1d1/0x200 [ 2373.978914] ? set_kthread_struct+0x70/0x70 [ 2373.978915] ret_from_fork+0x1f/0x30 [ 2373.978920] NMI backtrace for cpu 0 [ 2374.187481] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2374.191193] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2374.194890] Call Trace: [ 2374.196003] [ 2374.196817] ? dump_stack+0xbb/0x10e [ 2374.197881] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 2374.199898] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 2374.201850] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 2374.204435] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 2374.206694] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 2374.209053] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 2374.211557] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 2374.213954] ? update_process_times+0x6d/0xc0 [ 2374.216003] ? tick_sched_handle.isra.18+0x27/0x90 [ 2374.218227] ? tick_sched_timer+0x53/0xb0 [ 2374.219839] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2374.221452] ? tick_sched_handle.isra.18+0x90/0x90 [ 2374.222888] ? hrtimer_interrupt+0x128/0x2c0 [ 2374.224970] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2374.227665] ? apic_timer_interrupt+0xf/0x20 [ 2374.229673] [ 2374.230847] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2374.232992] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2374.235645] ? do_raw_spin_lock+0x177/0x190 [ 2374.239535] ? _raw_spin_lock+0x21/0x30 [ 2374.241320] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2374.245972] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 2374.247831] ? mdc_commit_open+0x91/0xb0 [mdc] [ 2374.251824] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2374.257738] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2374.262823] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 2374.266482] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2374.270961] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2374.280850] ? do_wait_intr+0xf0/0xf0 [ 2374.284926] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2374.299498] ? kthread+0x1d1/0x200 [ 2374.300710] ? set_kthread_struct+0x70/0x70 [ 2374.306711] ? ret_from_fork+0x1f/0x30 [ 2396.639042] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 437s! [ 2396.642893] Showing busy workqueues and worker pools: [ 2396.645592] workqueue events: flags=0x0 [ 2396.647536] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2396.650938] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 2396.654823] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 2396.659752] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 2396.665094] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2396.668329] pending: vmstat_shepherd [ 2396.670286] workqueue events_long: flags=0x0 [ 2396.672378] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2396.675603] pending: lu_objects_flush [obdclass] [ 2396.678352] workqueue events_unbound: flags=0x2 [ 2396.680888] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 2396.686223] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2396.698407] workqueue events_power_efficient: flags=0x80 [ 2396.704887] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2396.713217] pending: neigh_periodic_work [ 2396.714714] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2396.722936] pending: do_cache_clean, check_lifetime [ 2396.725560] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2396.735859] pending: neigh_periodic_work [ 2396.742419] workqueue writeback: flags=0x4a [ 2396.747452] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2396.755882] pending: wb_workfn, wb_workfn [ 2396.760808] workqueue xprtiod: flags=0xa [ 2396.762952] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 2396.767205] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 2396.771583] workqueue ipv6_addrconf: flags=0x40008 [ 2396.773740] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2396.776933] pending: addrconf_verify_work [ 2396.779806] workqueue knbd0-recv: flags=0x1a [ 2396.785376] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2396.792911] in-flight: 657:recv_work [ 2396.798157] workqueue ptlrpc_pinger: flags=0x2 [ 2396.804137] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2396.815483] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 2396.823219] pool 8: cpus=0-3 flags=0x4 nice=0 hung=437s workers=4 idle: 20831 12642 28859 [ 2396.835816] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2387s workers=2 idle: 659 [ 2400.060116] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [ 2400.064675] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 2400.064815] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2400.065042] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2400.065049] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2400.065051] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2400.065066] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2400.065070] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2400.065075] RAX: 0000000000006185 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2400.065077] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 2400.065080] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2400.065082] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 2400.065084] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 2400.065087] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 2400.065089] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2400.065091] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 2400.065100] Call Trace: [ 2400.065106] [ 2400.065112] ? show_regs.cold.9+0x22/0x2f [ 2400.065122] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2400.065128] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2400.065133] ? watchdog+0x40/0x40 [ 2400.065137] ? hrtimer_interrupt+0x128/0x2c0 [ 2400.065142] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2400.065217] ? apic_timer_interrupt+0xf/0x20 [ 2400.065225] [ 2400.065228] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2400.065234] do_raw_spin_lock+0x177/0x190 [ 2400.065239] _raw_spin_lock+0x21/0x30 [ 2400.065244] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2400.065543] ? do_raw_spin_unlock+0x75/0x190 [ 2400.065549] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2400.065741] ? do_wait_intr+0xf0/0xf0 [ 2400.065750] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2400.066867] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [ 2400.066874] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2400.066963] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2400.066968] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2400.066787] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2400.066972] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2400.066984] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2400.066988] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2400.066993] RAX: 0000000000004182 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2400.066995] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 2400.066997] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2400.066999] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 2400.067001] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 2400.067005] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 2400.067008] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2400.067010] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 2400.067306] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2400.067687] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2400.067816] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2400.067853] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2400.068062] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2400.068091] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2400.068613] Call Trace: [ 2400.068618] [ 2400.068620] ? show_regs.cold.9+0x22/0x2f [ 2400.068632] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2400.068639] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2400.068643] ? watchdog+0x40/0x40 [ 2400.068649] ? hrtimer_interrupt+0x128/0x2c0 [ 2400.068652] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2400.068662] ? apic_timer_interrupt+0xf/0x20 [ 2400.068666] [ 2400.068669] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2400.068678] do_raw_spin_lock+0x177/0x190 [ 2400.068683] _raw_spin_lock+0x21/0x30 [ 2400.068261] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2400.068688] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2400.069221] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2400.069489] lmv_intent_open+0x338/0x1150 [lmv] [ 2400.069612] ? do_raw_spin_unlock+0x75/0x190 [ 2400.069622] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2400.070358] ? do_wait_intr+0xf0/0xf0 [ 2400.070384] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2400.070598] mdc_close+0x53b/0xd70 [mdc] [ 2400.070636] lmv_close+0x18d/0x490 [lmv] [ 2400.070665] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 2400.070851] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2400.070978] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2400.071217] ll_md_close+0x274/0x800 [lustre] [ 2400.071299] ll_file_release+0x128/0xc50 [lustre] [ 2400.071391] __fput+0x10d/0x440 [ 2400.071400] ____fput+0x12/0x20 [ 2400.071404] task_work_run+0xc6/0x110 [ 2400.071411] exit_to_usermode_loop+0x1dd/0x1f0 [ 2400.071419] do_syscall_64+0x3d6/0x440 [ 2400.071424] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2400.071432] RIP: 0033:0x7ff54af45cd8 [ 2400.071440] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2400.071443] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2400.071448] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 2400.071450] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 2400.071452] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 2400.071455] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 2400.071457] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2400.069610] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2400.077346] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) [ 2400.126727] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2400.147675] rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2400.147721] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2400.147729] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2400.147735] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2400.149194] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30f/0x6d0 [ 2400.149210] Code: c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 <0f> 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 05 18 d4 [ 2400.149215] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2400.149219] RAX: 0000000000006b44 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2400.149222] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 2400.149224] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 2400.149226] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 2400.149229] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 2400.149232] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 2400.149235] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2400.149238] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 2400.149247] Call Trace: [ 2400.149253] [ 2400.149259] ? show_regs.cold.9+0x22/0x2f [ 2400.149269] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2400.149276] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2400.149280] ? watchdog+0x40/0x40 [ 2400.149284] ? hrtimer_interrupt+0x128/0x2c0 [ 2400.149288] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2400.149298] ? apic_timer_interrupt+0xf/0x20 [ 2400.149303] [ 2400.149306] ? __pv_queued_spin_lock_slowpath+0x30f/0x6d0 [ 2400.149311] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2400.149316] do_raw_spin_lock+0x177/0x190 [ 2400.149322] _raw_spin_lock+0x21/0x30 [ 2400.149327] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2400.150794] process_one_work+0x2c8/0x700 [ 2400.150802] worker_thread+0x296/0x6e0 [ 2400.150807] ? rescuer_thread+0x570/0x570 [ 2400.150811] kthread+0x1d1/0x200 [ 2400.150815] ? set_kthread_struct+0x70/0x70 [ 2400.150819] ret_from_fork+0x1f/0x30 [ 2400.787689] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2400.792168] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2400.794574] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2400.805549] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2400.813314] ? do_raw_spin_unlock+0x75/0x190 [ 2400.818807] ? _raw_spin_unlock+0x12/0x30 [ 2400.822786] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2400.825298] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2400.829579] ll_atomic_open+0x36a/0xd70 [lustre] [ 2400.834218] lookup_open+0x356/0xb40 [ 2400.835857] do_last+0x313/0xfc0 [ 2400.838833] ? nd_jump_root+0xe5/0x160 [ 2400.840643] ? path_init+0x437/0x520 [ 2400.845429] path_openat+0xf7/0x500 [ 2400.848027] ? do_raw_spin_unlock+0x75/0x190 [ 2400.853337] do_filp_open+0x99/0x140 [ 2400.856786] ? getname_flags+0x6e/0x330 [ 2400.858356] ? __check_object_size+0xff/0x256 [ 2400.864801] ? do_raw_spin_unlock+0x75/0x190 [ 2400.871190] ? _raw_spin_unlock+0x12/0x30 [ 2400.875129] do_sys_openat2+0x2b4/0x410 [ 2400.878855] do_sys_open+0x73/0xa0 [ 2400.881257] __x64_sys_openat+0x24/0x30 [ 2400.883010] do_syscall_64+0xc1/0x440 [ 2400.887398] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2400.890395] RIP: 0033:0x7ff54af45332 [ 2400.898219] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2400.915823] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2400.919746] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 2400.929229] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 2400.937204] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 2400.942589] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 2400.948417] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 2424.056950] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 2424.061737] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2424.088086] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2424.094047] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2424.105124] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2424.111268] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2424.131393] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2424.140681] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2424.147200] RDX: 00000000000042bf RSI: 0000000000000003 RDI: ffff96848811cae0 [ 2424.153743] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 2424.161572] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 2424.170549] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2424.180617] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 2424.188583] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2424.190491] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 2424.200961] Call Trace: [ 2424.203891] [ 2424.206389] ? show_regs.cold.9+0x22/0x2f [ 2424.211832] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2424.213819] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2424.217705] ? watchdog+0x40/0x40 [ 2424.219804] ? hrtimer_interrupt+0x128/0x2c0 [ 2424.222417] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2424.227698] ? apic_timer_interrupt+0xf/0x20 [ 2424.230366] [ 2424.231522] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2424.236341] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2424.239401] do_raw_spin_lock+0x177/0x190 [ 2424.249042] _raw_spin_lock+0x21/0x30 [ 2424.255036] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2424.256849] obd_mod_free+0x46/0x140 [ptlrpc] [ 2424.272649] mdc_commit_open+0x91/0xb0 [mdc] [ 2424.275499] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2424.284593] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2424.301221] after_reply+0xe2c/0x1780 [ptlrpc] [ 2424.306324] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2424.316829] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2424.325062] ? do_wait_intr+0xf0/0xf0 [ 2424.327197] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2424.344168] kthread+0x1d1/0x200 [ 2424.345219] ? set_kthread_struct+0x70/0x70 [ 2424.352982] ret_from_fork+0x1f/0x30 [ 2426.847075] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 467s! [ 2426.851750] Showing busy workqueues and worker pools: [ 2426.854595] workqueue events: flags=0x0 [ 2426.856813] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2426.863278] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 2426.874302] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 2426.878782] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 2426.886330] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2426.891981] pending: vmstat_shepherd [ 2426.895115] workqueue events_long: flags=0x0 [ 2426.898118] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2426.906181] pending: lu_objects_flush [obdclass] [ 2426.912087] workqueue events_unbound: flags=0x2 [ 2426.915316] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 2426.924003] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2426.937612] workqueue events_power_efficient: flags=0x80 [ 2426.939984] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2426.945717] pending: neigh_periodic_work [ 2426.950154] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2426.953987] pending: do_cache_clean, check_lifetime [ 2426.961403] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2426.968193] pending: neigh_periodic_work [ 2426.971043] workqueue writeback: flags=0x4a [ 2426.976224] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2426.982987] pending: wb_workfn, wb_workfn [ 2426.986015] workqueue xprtiod: flags=0xa [ 2426.991768] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 2426.999776] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 2427.013955] workqueue ipv6_addrconf: flags=0x40008 [ 2427.015635] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2427.022959] pending: addrconf_verify_work [ 2427.027938] workqueue knbd0-recv: flags=0x1a [ 2427.033259] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2427.041590] in-flight: 657:recv_work [ 2427.047308] workqueue ptlrpc_pinger: flags=0x2 [ 2427.057977] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2427.066791] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 2427.074752] pool 8: cpus=0-3 flags=0x4 nice=0 hung=468s workers=4 idle: 20831 12642 28859 [ 2427.084916] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2417s workers=2 idle: 659 [ 2428.059179] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [ 2428.061769] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [ 2428.061820] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 2428.061827] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2428.061911] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2428.061916] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2428.061923] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2428.062773] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2428.062785] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2428.062789] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2428.062793] RAX: 0000000000005987 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2428.062796] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 2428.062799] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 2428.062801] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 2428.062803] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 2428.062807] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 2428.062809] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2428.062812] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 2428.062820] Call Trace: [ 2428.062825] [ 2428.062830] ? show_regs.cold.9+0x22/0x2f [ 2428.062840] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2428.062846] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2428.062850] ? watchdog+0x40/0x40 [ 2428.062854] ? hrtimer_interrupt+0x128/0x2c0 [ 2428.062858] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2428.062867] ? apic_timer_interrupt+0xf/0x20 [ 2428.062872] [ 2428.062875] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2428.062880] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2428.062946] do_raw_spin_lock+0x177/0x190 [ 2428.062952] _raw_spin_lock+0x21/0x30 [ 2428.062957] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2428.063241] process_one_work+0x2c8/0x700 [ 2428.063248] worker_thread+0x296/0x6e0 [ 2428.063253] ? rescuer_thread+0x570/0x570 [ 2428.063256] kthread+0x1d1/0x200 [ 2428.063260] ? set_kthread_struct+0x70/0x70 [ 2428.063264] ret_from_fork+0x1f/0x30 [ 2428.077478] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2428.077554] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2428.077562] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2428.077564] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2428.077578] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2428.077582] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2428.077586] RAX: 0000000000006688 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2428.077588] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 2428.077590] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2428.077593] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 2428.077595] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 2428.077598] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 2428.077600] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2428.077603] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 2428.077620] Call Trace: [ 2428.077626] [ 2428.077631] ? show_regs.cold.9+0x22/0x2f [ 2428.077641] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2428.077647] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2428.077651] ? watchdog+0x40/0x40 [ 2428.077655] ? hrtimer_interrupt+0x128/0x2c0 [ 2428.077659] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2428.077666] ? apic_timer_interrupt+0xf/0x20 [ 2428.077670] [ 2428.077674] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2428.077680] do_raw_spin_lock+0x177/0x190 [ 2428.077685] _raw_spin_lock+0x21/0x30 [ 2428.077690] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2428.084545] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2428.084615] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2428.084622] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2428.084625] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2428.084639] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2428.084644] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2428.084649] RAX: 0000000000005184 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2428.084651] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 2428.084654] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2428.084656] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 2428.084658] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 2428.084661] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 2428.084664] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2428.084666] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 2428.084674] Call Trace: [ 2428.084679] [ 2428.084686] ? show_regs.cold.9+0x22/0x2f [ 2428.084695] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2428.084700] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2428.084704] ? watchdog+0x40/0x40 [ 2428.084708] ? hrtimer_interrupt+0x128/0x2c0 [ 2428.084713] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2428.084721] ? apic_timer_interrupt+0xf/0x20 [ 2428.084725] [ 2428.084727] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2428.084733] do_raw_spin_lock+0x177/0x190 [ 2428.084738] _raw_spin_lock+0x21/0x30 [ 2428.084742] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2428.106307] ? do_raw_spin_unlock+0x75/0x190 [ 2428.106328] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2428.106587] ? do_wait_intr+0xf0/0xf0 [ 2428.106595] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2428.157172] ? do_raw_spin_unlock+0x75/0x190 [ 2428.157193] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2428.159839] ? do_wait_intr+0xf0/0xf0 [ 2428.159852] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2428.161153] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2428.161395] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2428.163806] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2428.163929] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2428.163967] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2428.163996] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2428.164025] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2428.164120] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2428.164347] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2428.164377] lmv_intent_open+0x338/0x1150 [lmv] [ 2428.164409] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2428.164494] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2428.186456] mdc_close+0x53b/0xd70 [mdc] [ 2428.186612] lmv_close+0x18d/0x490 [lmv] [ 2428.186641] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 2428.186803] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2428.188043] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2428.188642] ll_md_close+0x274/0x800 [lustre] [ 2428.189629] ll_file_release+0x128/0xc50 [lustre] [ 2428.190630] __fput+0x10d/0x440 [ 2428.190641] ____fput+0x12/0x20 [ 2428.190645] task_work_run+0xc6/0x110 [ 2428.190653] exit_to_usermode_loop+0x1dd/0x1f0 [ 2428.190660] do_syscall_64+0x3d6/0x440 [ 2428.190664] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2428.190672] RIP: 0033:0x7ff54af45cd8 [ 2428.190679] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2428.190684] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2428.190689] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 2428.190692] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 2428.190694] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 2428.190696] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 2428.190699] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2428.986755] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2428.988820] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2428.993192] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2428.996901] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2429.000417] ? do_raw_spin_unlock+0x75/0x190 [ 2429.003610] ? _raw_spin_unlock+0x12/0x30 [ 2429.011867] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2429.016517] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2429.027734] ll_atomic_open+0x36a/0xd70 [lustre] [ 2429.035664] lookup_open+0x356/0xb40 [ 2429.042150] do_last+0x313/0xfc0 [ 2429.048036] ? nd_jump_root+0xe5/0x160 [ 2429.055106] ? path_init+0x437/0x520 [ 2429.061504] path_openat+0xf7/0x500 [ 2429.063661] ? do_raw_spin_unlock+0x75/0x190 [ 2429.065683] do_filp_open+0x99/0x140 [ 2429.070616] ? getname_flags+0x6e/0x330 [ 2429.072610] ? __check_object_size+0xff/0x256 [ 2429.079829] ? do_raw_spin_unlock+0x75/0x190 [ 2429.083666] ? _raw_spin_unlock+0x12/0x30 [ 2429.086972] do_sys_openat2+0x2b4/0x410 [ 2429.089927] do_sys_open+0x73/0xa0 [ 2429.092465] __x64_sys_openat+0x24/0x30 [ 2429.094544] do_syscall_64+0xc1/0x440 [ 2429.099107] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2429.101405] RIP: 0033:0x7ff54af45332 [ 2429.103166] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2429.119925] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2429.123458] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 2429.129462] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 2429.136797] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 2429.140334] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 2429.146788] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 2452.055024] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 2452.058066] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2452.083883] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2452.089719] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2452.093663] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2452.095923] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2452.102703] RSP: 0000:ffffaf9fc1b97bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2452.106032] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2452.108733] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 2452.111210] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 2452.114261] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 2452.116772] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2452.119881] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 2452.122914] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2452.125380] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 2452.128551] Call Trace: [ 2452.129464] [ 2452.130430] ? show_regs.cold.9+0x22/0x2f [ 2452.132281] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2452.134443] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2452.136183] ? watchdog+0x40/0x40 [ 2452.137289] ? hrtimer_interrupt+0x128/0x2c0 [ 2452.138788] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2452.140892] ? apic_timer_interrupt+0xf/0x20 [ 2452.142801] [ 2452.143829] ? native_safe_halt+0x2e/0x50 [ 2452.146014] kvm_wait+0xcd/0xe0 [ 2452.147438] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2452.149354] do_raw_spin_lock+0x177/0x190 [ 2452.151005] _raw_spin_lock+0x21/0x30 [ 2452.152697] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2452.155211] obd_mod_free+0x46/0x140 [ptlrpc] [ 2452.157415] mdc_commit_open+0x91/0xb0 [mdc] [ 2452.159894] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2452.162940] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2452.165947] after_reply+0xe2c/0x1780 [ptlrpc] [ 2452.168577] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2452.171006] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2452.173323] ? do_wait_intr+0xf0/0xf0 [ 2452.196399] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2452.235724] kthread+0x1d1/0x200 [ 2452.237214] ? set_kthread_struct+0x70/0x70 [ 2452.239302] ret_from_fork+0x1f/0x30 [ 2456.060259] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 2456.060847] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 2456.064818] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 2456.064833] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2456.064903] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2456.064909] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2456.064912] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2456.064929] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2456.064933] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2456.064938] RAX: 0000000000007f05 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2456.064940] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 2456.064942] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2456.064944] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 2456.064947] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 2456.064950] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 2456.064953] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2456.064955] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 2456.064964] Call Trace: [ 2456.064970] [ 2456.064977] ? show_regs.cold.9+0x22/0x2f [ 2456.064987] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2456.064994] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2456.064998] ? watchdog+0x40/0x40 [ 2456.065003] ? hrtimer_interrupt+0x128/0x2c0 [ 2456.065007] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2456.065016] ? apic_timer_interrupt+0xf/0x20 [ 2456.065021] [ 2456.065024] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2456.065030] do_raw_spin_lock+0x177/0x190 [ 2456.065035] _raw_spin_lock+0x21/0x30 [ 2456.065040] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2456.065315] ? do_raw_spin_unlock+0x75/0x190 [ 2456.065321] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2456.066575] ? do_wait_intr+0xf0/0xf0 [ 2456.066586] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2456.066804] mdc_close+0x53b/0xd70 [mdc] [ 2456.066846] lmv_close+0x18d/0x490 [lmv] [ 2456.066878] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 2456.067035] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2456.067145] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2456.067244] ll_md_close+0x274/0x800 [lustre] [ 2456.068252] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2456.068318] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2456.068324] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2456.067327] ll_file_release+0x128/0xc50 [lustre] [ 2456.068331] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2456.072096] __fput+0x10d/0x440 [ 2456.072107] ____fput+0x12/0x20 [ 2456.072112] task_work_run+0xc6/0x110 [ 2456.072120] exit_to_usermode_loop+0x1dd/0x1f0 [ 2456.072128] do_syscall_64+0x3d6/0x440 [ 2456.072133] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2456.072142] RIP: 0033:0x7ff54af45cd8 [ 2456.072225] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2456.072230] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2456.072236] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 2456.072238] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 2456.072247] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 2456.072250] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 2456.072252] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2456.073745] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2456.073891] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2456.073898] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2456.073900] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2456.073913] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2456.073917] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2456.073921] RAX: 0000000000007f8a RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2456.073923] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 2456.073926] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2456.073927] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 2456.073929] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 2456.073932] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 2456.073935] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2456.073937] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 2456.073945] Call Trace: [ 2456.073950] [ 2456.073955] ? show_regs.cold.9+0x22/0x2f [ 2456.073964] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2456.073970] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2456.073974] ? watchdog+0x40/0x40 [ 2456.073978] ? hrtimer_interrupt+0x128/0x2c0 [ 2456.073981] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2456.073989] ? apic_timer_interrupt+0xf/0x20 [ 2456.073993] [ 2456.073996] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2456.074002] do_raw_spin_lock+0x177/0x190 [ 2456.074007] _raw_spin_lock+0x21/0x30 [ 2456.074011] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2456.083960] [ 2456.083969] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2456.100747] ? do_raw_spin_unlock+0x75/0x190 [ 2456.100840] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2456.101149] ? do_wait_intr+0xf0/0xf0 [ 2456.101158] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2456.101357] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2456.101657] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2456.102017] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2456.102128] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2456.102168] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2456.102200] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2456.102230] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2456.102318] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2456.102631] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2456.111433] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2456.111444] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2456.111450] RAX: 0000000000006616 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2456.111452] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 2456.111455] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 2456.111457] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 2456.111459] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 2456.111462] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 2456.111464] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2456.111466] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 2456.111473] Call Trace: [ 2456.111478] [ 2456.111485] ? show_regs.cold.9+0x22/0x2f [ 2456.115193] lmv_intent_open+0x338/0x1150 [lmv] [ 2456.115244] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2456.115357] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2456.115596] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2456.115631] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2456.115720] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2456.115916] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2456.115999] ? do_raw_spin_unlock+0x75/0x190 [ 2456.116007] ? _raw_spin_unlock+0x12/0x30 [ 2456.116014] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2456.116040] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2456.116058] ll_atomic_open+0x36a/0xd70 [lustre] [ 2456.116143] lookup_open+0x356/0xb40 [ 2456.116152] do_last+0x313/0xfc0 [ 2456.116156] ? nd_jump_root+0xe5/0x160 [ 2456.116160] ? path_init+0x437/0x520 [ 2456.116164] path_openat+0xf7/0x500 [ 2456.116169] ? do_raw_spin_unlock+0x75/0x190 [ 2456.116173] do_filp_open+0x99/0x140 [ 2456.116179] ? getname_flags+0x6e/0x330 [ 2456.116184] ? __check_object_size+0xff/0x256 [ 2456.116191] ? do_raw_spin_unlock+0x75/0x190 [ 2456.116194] ? _raw_spin_unlock+0x12/0x30 [ 2456.116198] do_sys_openat2+0x2b4/0x410 [ 2456.116203] do_sys_open+0x73/0xa0 [ 2456.116207] __x64_sys_openat+0x24/0x30 [ 2456.116211] do_syscall_64+0xc1/0x440 [ 2456.116218] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2456.116224] RIP: 0033:0x7ff54af45332 [ 2456.116232] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2456.116236] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2456.116241] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 2456.116244] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 2456.116246] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 2456.116247] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 2456.116250] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 2456.887295] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2456.894846] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2456.901338] ? watchdog+0x40/0x40 [ 2456.903765] ? hrtimer_interrupt+0x128/0x2c0 [ 2456.905941] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2456.908285] ? apic_timer_interrupt+0xf/0x20 [ 2456.913924] [ 2456.914992] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2456.919051] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2456.923397] do_raw_spin_lock+0x177/0x190 [ 2456.928662] _raw_spin_lock+0x21/0x30 [ 2456.930078] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2456.933184] process_one_work+0x2c8/0x700 [ 2456.935030] worker_thread+0x296/0x6e0 [ 2456.936858] ? rescuer_thread+0x570/0x570 [ 2456.938822] kthread+0x1d1/0x200 [ 2456.940819] ? set_kthread_struct+0x70/0x70 [ 2456.942732] ret_from_fork+0x1f/0x30 [ 2457.571630] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 498s! [ 2457.574955] Showing busy workqueues and worker pools: [ 2457.577617] workqueue events: flags=0x0 [ 2457.579608] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2457.582545] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 2457.586058] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 2457.589607] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 2457.595119] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2457.598781] pending: vmstat_shepherd [ 2457.600643] workqueue events_long: flags=0x0 [ 2457.602565] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2457.605565] pending: lu_objects_flush [obdclass] [ 2457.608533] workqueue events_unbound: flags=0x2 [ 2457.610645] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 2457.614323] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2457.620503] workqueue events_power_efficient: flags=0x80 [ 2457.625513] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2457.636090] pending: neigh_periodic_work [ 2457.641848] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2457.649960] pending: do_cache_clean, check_lifetime [ 2457.654031] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2457.661145] pending: neigh_periodic_work [ 2457.669286] workqueue writeback: flags=0x4a [ 2457.674662] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2457.678443] pending: wb_workfn, wb_workfn [ 2457.682701] workqueue xprtiod: flags=0xa [ 2457.685350] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 2457.687710] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 2457.691592] workqueue ipv6_addrconf: flags=0x40008 [ 2457.694656] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2457.698142] pending: addrconf_verify_work [ 2457.704441] workqueue knbd0-recv: flags=0x1a [ 2457.710815] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2457.719479] in-flight: 657:recv_work [ 2457.721653] workqueue ptlrpc_pinger: flags=0x2 [ 2457.724108] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2457.727275] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 2457.732201] pool 8: cpus=0-3 flags=0x4 nice=0 hung=498s workers=4 idle: 20831 12642 28859 [ 2457.739798] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2447s workers=2 idle: 659 [ 2484.056843] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 2484.059999] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 2484.061317] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 2484.061329] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2484.061411] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2484.061416] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2484.061419] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2484.061433] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2484.061437] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2484.061441] RAX: 0000000000005f92 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2484.061443] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 2484.061446] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2484.061448] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 2484.061450] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 2484.061453] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 2484.061456] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2484.061458] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 2484.061465] Call Trace: [ 2484.061471] [ 2484.061477] ? show_regs.cold.9+0x22/0x2f [ 2484.061488] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2484.061494] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2484.061498] ? watchdog+0x40/0x40 [ 2484.061503] ? hrtimer_interrupt+0x128/0x2c0 [ 2484.061507] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2484.061515] ? apic_timer_interrupt+0xf/0x20 [ 2484.061520] [ 2484.061522] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2484.061528] do_raw_spin_lock+0x177/0x190 [ 2484.061533] _raw_spin_lock+0x21/0x30 [ 2484.061538] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2484.062030] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2484.062088] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2484.063164] ? do_raw_spin_unlock+0x75/0x190 [ 2484.063223] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2484.063229] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2484.063239] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2484.063243] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2484.063248] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2484.063250] RDX: 000000000000349d RSI: 0000000000000003 RDI: ffff96848811cae0 [ 2484.063253] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 2484.063255] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 2484.063257] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2484.063260] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 2484.063263] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2484.063265] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 2484.063272] Call Trace: [ 2484.063276] [ 2484.063279] ? show_regs.cold.9+0x22/0x2f [ 2484.063288] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2484.063294] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2484.063176] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2484.063300] ? watchdog+0x40/0x40 [ 2484.063306] ? hrtimer_interrupt+0x128/0x2c0 [ 2484.063310] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2484.063318] ? apic_timer_interrupt+0xf/0x20 [ 2484.063323] [ 2484.063325] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2484.063329] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2484.063333] do_raw_spin_lock+0x177/0x190 [ 2484.063350] _raw_spin_lock+0x21/0x30 [ 2484.063356] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2484.063892] obd_mod_free+0x46/0x140 [ptlrpc] [ 2484.064383] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 2484.064830] ? do_wait_intr+0xf0/0xf0 [ 2484.064840] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2484.065203] mdc_close+0x53b/0xd70 [mdc] [ 2484.065362] mdc_commit_open+0x91/0xb0 [mdc] [ 2484.065489] lmv_close+0x18d/0x490 [lmv] [ 2484.065401] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2484.065864] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2484.065604] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 2484.066287] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2484.066870] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2484.066075] after_reply+0xe2c/0x1780 [ptlrpc] [ 2484.067464] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2484.067677] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2484.068029] ? do_wait_intr+0xf0/0xf0 [ 2484.066957] ll_md_close+0x274/0x800 [lustre] [ 2484.068137] ll_file_release+0x128/0xc50 [lustre] [ 2484.068232] __fput+0x10d/0x440 [ 2484.068242] ____fput+0x12/0x20 [ 2484.068247] task_work_run+0xc6/0x110 [ 2484.068254] exit_to_usermode_loop+0x1dd/0x1f0 [ 2484.068262] do_syscall_64+0x3d6/0x440 [ 2484.068267] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2484.068276] RIP: 0033:0x7ff54af45cd8 [ 2484.068283] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2484.068287] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2484.068293] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 2484.068295] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 2484.068298] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 2484.068300] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 2484.068302] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2484.068040] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2484.069438] kthread+0x1d1/0x200 [ 2484.069445] ? set_kthread_struct+0x70/0x70 [ 2484.069448] ret_from_fork+0x1f/0x30 [ 2484.070065] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2484.070134] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2484.070141] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2484.070148] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2484.070581] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2484.070591] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2484.070596] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2484.070600] RAX: 0000000000005f2b RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2484.070602] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 2484.070604] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 2484.070606] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 2484.070608] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 2484.070612] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 2484.070614] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2484.070616] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 2484.070624] Call Trace: [ 2484.070629] [ 2484.070634] ? show_regs.cold.9+0x22/0x2f [ 2484.087119] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2484.087264] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2484.087348] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2484.087352] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2484.087367] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2484.087372] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2484.087376] RAX: 000000000000758e RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2484.087379] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 2484.087381] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2484.087382] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 2484.087385] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 2484.087388] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 2484.087391] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2484.087393] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 2484.087401] Call Trace: [ 2484.087407] [ 2484.087413] ? show_regs.cold.9+0x22/0x2f [ 2484.087422] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2484.087428] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2484.087432] ? watchdog+0x40/0x40 [ 2484.087437] ? hrtimer_interrupt+0x128/0x2c0 [ 2484.087440] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2484.087449] ? apic_timer_interrupt+0xf/0x20 [ 2484.087453] [ 2484.087456] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2484.087464] do_raw_spin_lock+0x177/0x190 [ 2484.087469] _raw_spin_lock+0x21/0x30 [ 2484.087474] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2484.087752] ? do_raw_spin_unlock+0x75/0x190 [ 2484.087758] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2484.089088] ? do_wait_intr+0xf0/0xf0 [ 2484.089099] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2484.089561] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2484.089791] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2484.092245] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2484.168424] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2484.168445] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2484.168450] ? watchdog+0x40/0x40 [ 2484.168455] ? hrtimer_interrupt+0x128/0x2c0 [ 2484.168458] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2484.168467] ? apic_timer_interrupt+0xf/0x20 [ 2484.168472] [ 2484.168477] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2484.168483] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2484.168490] do_raw_spin_lock+0x177/0x190 [ 2484.168495] _raw_spin_lock+0x21/0x30 [ 2484.168499] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2484.168783] process_one_work+0x2c8/0x700 [ 2484.168791] worker_thread+0x296/0x6e0 [ 2484.168795] ? rescuer_thread+0x570/0x570 [ 2484.168798] kthread+0x1d1/0x200 [ 2484.168802] ? set_kthread_struct+0x70/0x70 [ 2484.168805] ret_from_fork+0x1f/0x30 [ 2485.507024] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2485.509773] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2485.512218] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2485.514672] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2485.517777] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2485.520827] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2485.524090] lmv_intent_open+0x338/0x1150 [lmv] [ 2485.526798] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2485.529747] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2485.532871] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2485.536729] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2485.539058] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2485.546078] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2485.548235] ? do_raw_spin_unlock+0x75/0x190 [ 2485.549878] ? _raw_spin_unlock+0x12/0x30 [ 2485.558542] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2485.564479] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2485.569085] ll_atomic_open+0x36a/0xd70 [lustre] [ 2485.573297] lookup_open+0x356/0xb40 [ 2485.574845] do_last+0x313/0xfc0 [ 2485.580766] ? nd_jump_root+0xe5/0x160 [ 2485.582001] ? path_init+0x437/0x520 [ 2485.587517] path_openat+0xf7/0x500 [ 2485.589675] ? do_raw_spin_unlock+0x75/0x190 [ 2485.601072] do_filp_open+0x99/0x140 [ 2485.603036] ? getname_flags+0x6e/0x330 [ 2485.609370] ? __check_object_size+0xff/0x256 [ 2485.611606] ? do_raw_spin_unlock+0x75/0x190 [ 2485.619416] ? _raw_spin_unlock+0x12/0x30 [ 2485.620976] do_sys_openat2+0x2b4/0x410 [ 2485.624743] do_sys_open+0x73/0xa0 [ 2485.627528] __x64_sys_openat+0x24/0x30 [ 2485.636732] do_syscall_64+0xc1/0x440 [ 2485.638154] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2485.648773] RIP: 0033:0x7ff54af45332 [ 2485.655186] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2485.687568] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2485.695925] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 2485.702362] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 2485.706454] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 2485.711771] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 2485.718786] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 2487.777554] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 528s! [ 2487.788032] Showing busy workqueues and worker pools: [ 2487.793986] workqueue events: flags=0x0 [ 2487.796430] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2487.803860] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 2487.811119] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 2487.815036] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 2487.838700] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2487.842465] pending: vmstat_shepherd [ 2487.843946] workqueue events_long: flags=0x0 [ 2487.850648] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2487.858681] pending: lu_objects_flush [obdclass] [ 2487.866891] workqueue events_unbound: flags=0x2 [ 2487.869869] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 2487.874672] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2487.882598] workqueue events_power_efficient: flags=0x80 [ 2487.884250] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2487.890399] pending: neigh_periodic_work [ 2487.891986] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2487.895699] pending: do_cache_clean, check_lifetime [ 2487.897664] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2487.899877] pending: neigh_periodic_work [ 2487.901601] workqueue writeback: flags=0x4a [ 2487.904691] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2487.906728] pending: wb_workfn, wb_workfn [ 2487.909741] workqueue xprtiod: flags=0xa [ 2487.911829] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 2487.914812] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 2487.919033] workqueue ipv6_addrconf: flags=0x40008 [ 2487.922476] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2487.925888] pending: addrconf_verify_work [ 2487.929839] workqueue knbd0-recv: flags=0x1a [ 2487.931915] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2487.934717] in-flight: 657:recv_work [ 2487.940879] workqueue ptlrpc_pinger: flags=0x2 [ 2487.943087] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2487.948571] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 2487.955199] pool 8: cpus=0-3 flags=0x4 nice=0 hung=529s workers=4 idle: 20831 12642 28859 [ 2487.960760] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2478s workers=2 idle: 659 [ 2512.056490] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 2512.058119] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 2512.059019] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 2512.059027] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2512.059106] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2512.059111] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2512.059117] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2512.059462] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2512.059474] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2512.059486] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2512.059591] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2512.059594] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 2512.059596] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 2512.059597] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 2512.059600] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 2512.059603] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 2512.059605] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2512.059607] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 2512.059616] Call Trace: [ 2512.059623] [ 2512.059630] ? show_regs.cold.9+0x22/0x2f [ 2512.059641] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2512.059648] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2512.059666] ? watchdog+0x40/0x40 [ 2512.059670] ? hrtimer_interrupt+0x128/0x2c0 [ 2512.059674] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2512.059682] ? apic_timer_interrupt+0xf/0x20 [ 2512.059686] [ 2512.059689] ? native_safe_halt+0x2e/0x50 [ 2512.059693] kvm_wait+0xcd/0xe0 [ 2512.059700] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2512.059707] do_raw_spin_lock+0x177/0x190 [ 2512.059720] _raw_spin_lock+0x21/0x30 [ 2512.059724] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2512.059933] process_one_work+0x2c8/0x700 [ 2512.059940] worker_thread+0x296/0x6e0 [ 2512.059944] ? rescuer_thread+0x570/0x570 [ 2512.059947] kthread+0x1d1/0x200 [ 2512.059951] ? set_kthread_struct+0x70/0x70 [ 2512.059955] ret_from_fork+0x1f/0x30 [ 2512.061054] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 [ 2512.061117] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 2512.061123] squashfs [ 2512.061128] Modules linked in: [ 2512.061128] ata_generic [ 2512.061132] lustre(O) [ 2512.061132] crct10dif_pclmul crc32_pclmul [ 2512.061138] osp(O) [ 2512.061141] crc32c_intel [ 2512.061144] ofd(O) [ 2512.061145] ata_piix [ 2512.061149] lod(O) [ 2512.061150] ghash_clmulni_intel [ 2512.061155] mdt(O) [ 2512.061156] serio_raw [ 2512.061160] mdd(O) [ 2512.061161] libata [ 2512.061165] mgs(O) [ 2512.061167] dm_mirror [ 2512.061170] lquota(O) [ 2512.061172] dm_region_hash [ 2512.061176] lfsck(O) [ 2512.061177] dm_log [ 2512.061181] obdecho(O) [ 2512.061182] dm_mod [ 2512.061187] mgc(O) [ 2512.061187] sha512_ssse3 [ 2512.061192] mdc(O) [ 2512.061192] sha512_generic [ 2512.061199] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2512.061205] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2512.061209] lov(O) osc(O) lmv(O) [ 2512.061210] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2512.061218] ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) [ 2512.061227] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2512.061230] obdclass(O) ksocklnd(O) [ 2512.061236] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 [ 2512.061237] lnet(O) [ 2512.061241] ORIG_RAX: ffffffffffffff13 [ 2512.061245] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2512.061247] RDX: 0000000000001516 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 2512.061250] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 2512.061252] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 2512.061254] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2512.061257] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 2512.061260] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2512.061263] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 2512.061271] Call Trace: [ 2512.061275] [ 2512.061280] ? show_regs.cold.9+0x22/0x2f [ 2512.061289] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2512.061295] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2512.061299] ? watchdog+0x40/0x40 [ 2512.061303] ? hrtimer_interrupt+0x128/0x2c0 [ 2512.061307] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2512.061314] ? apic_timer_interrupt+0xf/0x20 [ 2512.061319] [ 2512.061322] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2512.061328] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2512.061333] do_raw_spin_lock+0x177/0x190 [ 2512.061337] _raw_spin_lock+0x21/0x30 [ 2512.061415] libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2512.061446] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2512.061455] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2512.061457] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2512.061464] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2512.061470] RSP: 0000:ffffaf9fc15234f0 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 2512.061474] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2512.061477] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 2512.061479] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2512.061481] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 2512.061483] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 2512.061486] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 2512.061489] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2512.061491] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 2512.061496] Call Trace: [ 2512.061499] [ 2512.061501] ? show_regs.cold.9+0x22/0x2f [ 2512.061507] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2512.061512] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2512.061515] ? watchdog+0x40/0x40 [ 2512.061519] ? hrtimer_interrupt+0x128/0x2c0 [ 2512.061523] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2512.061528] ? apic_timer_interrupt+0xf/0x20 [ 2512.061532] [ 2512.061535] ? native_safe_halt+0x2e/0x50 [ 2512.061538] kvm_wait+0xcd/0xe0 [ 2512.061545] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2512.061550] do_raw_spin_lock+0x177/0x190 [ 2512.061554] _raw_spin_lock+0x21/0x30 [ 2512.061557] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2512.061342] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2512.062412] ? do_raw_spin_unlock+0x75/0x190 [ 2512.062416] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2512.062541] ? do_wait_intr+0xf0/0xf0 [ 2512.062548] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2512.062671] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2512.063152] obd_mod_free+0x46/0x140 [ptlrpc] [ 2512.062835] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2512.063588] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2512.063462] mdc_commit_open+0x91/0xb0 [mdc] [ 2512.064499] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2512.064841] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2512.065385] after_reply+0xe2c/0x1780 [ptlrpc] [ 2512.066756] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2512.068057] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2512.068291] ? do_wait_intr+0xf0/0xf0 [ 2512.068302] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2512.068524] kthread+0x1d1/0x200 [ 2512.068530] ? set_kthread_struct+0x70/0x70 [ 2512.069651] ret_from_fork+0x1f/0x30 [ 2512.071147] Modules linked in: lustre(O) [ 2512.077077] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2512.077129] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2512.077161] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2512.077192] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2512.077366] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2512.077651] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2512.078491] lmv_intent_open+0x338/0x1150 [lmv] [ 2512.078525] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2512.078620] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2512.103304] osp(O) [ 2512.109100] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2512.109147] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2512.117575] ofd(O) [ 2512.122468] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2512.127231] lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2512.127309] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2512.127317] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2512.127320] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2512.127336] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2512.127340] RSP: 0018:ffffaf9fc1543ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2512.127345] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2512.127348] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 2512.127350] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2512.127351] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 2512.127353] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 2512.127356] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 2512.127359] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2512.127361] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 2512.127370] Call Trace: [ 2512.127377] [ 2512.127383] ? show_regs.cold.9+0x22/0x2f [ 2512.127394] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2512.127401] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2512.127407] ? watchdog+0x40/0x40 [ 2512.127411] ? hrtimer_interrupt+0x128/0x2c0 [ 2512.127415] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2512.127422] ? apic_timer_interrupt+0xf/0x20 [ 2512.127426] [ 2512.127428] ? native_safe_halt+0x2e/0x50 [ 2512.127432] kvm_wait+0xcd/0xe0 [ 2512.127440] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2512.127449] do_raw_spin_lock+0x177/0x190 [ 2512.127454] _raw_spin_lock+0x21/0x30 [ 2512.127458] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2512.140111] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2512.145422] ? do_raw_spin_unlock+0x75/0x190 [ 2512.145446] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2512.155756] ? do_raw_spin_unlock+0x75/0x190 [ 2512.155779] ? _raw_spin_unlock+0x12/0x30 [ 2512.155787] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2512.155825] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2512.155843] ll_atomic_open+0x36a/0xd70 [lustre] [ 2512.156141] lookup_open+0x356/0xb40 [ 2512.156150] do_last+0x313/0xfc0 [ 2512.156155] ? nd_jump_root+0xe5/0x160 [ 2512.156160] ? path_init+0x437/0x520 [ 2512.156165] path_openat+0xf7/0x500 [ 2512.156170] ? do_raw_spin_unlock+0x75/0x190 [ 2512.156176] do_filp_open+0x99/0x140 [ 2512.156183] ? getname_flags+0x6e/0x330 [ 2512.156187] ? __check_object_size+0xff/0x256 [ 2512.156195] ? do_raw_spin_unlock+0x75/0x190 [ 2512.156256] ? _raw_spin_unlock+0x12/0x30 [ 2512.156260] do_sys_openat2+0x2b4/0x410 [ 2512.156265] do_sys_open+0x73/0xa0 [ 2512.156269] __x64_sys_openat+0x24/0x30 [ 2512.156272] do_syscall_64+0xc1/0x440 [ 2512.156279] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2512.156285] RIP: 0033:0x7ff54af45332 [ 2512.156293] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2512.156297] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2512.156303] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 2512.156305] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 2512.156307] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 2512.156309] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 2512.156311] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 2513.360446] ? do_wait_intr+0xf0/0xf0 [ 2513.364651] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2513.367443] mdc_close+0x53b/0xd70 [mdc] [ 2513.370802] lmv_close+0x18d/0x490 [lmv] [ 2513.375064] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 2513.377459] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2513.382836] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2513.386233] ll_md_close+0x274/0x800 [lustre] [ 2513.388592] ll_file_release+0x128/0xc50 [lustre] [ 2513.392233] __fput+0x10d/0x440 [ 2513.393850] ____fput+0x12/0x20 [ 2513.395740] task_work_run+0xc6/0x110 [ 2513.398336] exit_to_usermode_loop+0x1dd/0x1f0 [ 2513.400821] do_syscall_64+0x3d6/0x440 [ 2513.402919] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2513.405183] RIP: 0033:0x7ff54af45cd8 [ 2513.406842] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2513.416226] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2513.422135] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 2513.432275] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 2513.437994] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 2513.442369] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 2513.457626] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2517.983308] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 559s! [ 2517.990276] Showing busy workqueues and worker pools: [ 2517.998742] workqueue events: flags=0x0 [ 2518.002050] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2518.005769] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 2518.013789] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 2518.018585] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 2518.024737] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2518.028438] pending: vmstat_shepherd [ 2518.030910] workqueue events_long: flags=0x0 [ 2518.033142] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2518.039533] pending: lu_objects_flush [obdclass] [ 2518.044724] workqueue events_unbound: flags=0x2 [ 2518.048295] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 2518.051981] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2518.057871] workqueue events_power_efficient: flags=0x80 [ 2518.062639] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2518.067177] pending: neigh_periodic_work [ 2518.069461] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2518.074391] pending: do_cache_clean, check_lifetime [ 2518.076972] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2518.080580] pending: neigh_periodic_work [ 2518.081982] workqueue writeback: flags=0x4a [ 2518.091121] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2518.098939] pending: wb_workfn, wb_workfn [ 2518.101425] workqueue xprtiod: flags=0xa [ 2518.104678] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 2518.110199] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 2518.118692] workqueue ipv6_addrconf: flags=0x40008 [ 2518.125764] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2518.130699] pending: addrconf_verify_work [ 2518.138439] workqueue knbd0-recv: flags=0x1a [ 2518.141573] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2518.146780] in-flight: 657:recv_work [ 2518.148766] workqueue ptlrpc_pinger: flags=0x2 [ 2518.151794] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2518.156511] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 2518.164648] pool 8: cpus=0-3 flags=0x4 nice=0 hung=559s workers=4 idle: 20831 12642 28859 [ 2518.173822] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2508s workers=2 idle: 659 [ 2540.059338] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 2540.062900] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 2540.068086] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 2540.068109] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2540.068197] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2540.068202] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2540.068204] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2540.068217] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2540.068226] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2540.068231] RAX: 0000000000004983 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2540.068233] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 2540.068234] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2540.068236] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 2540.068239] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 2540.068241] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 2540.068244] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2540.068246] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 2540.068260] Call Trace: [ 2540.068266] [ 2540.068272] ? show_regs.cold.9+0x22/0x2f [ 2540.068287] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2540.068294] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2540.068298] ? watchdog+0x40/0x40 [ 2540.068303] ? hrtimer_interrupt+0x128/0x2c0 [ 2540.068318] Modules linked in: [ 2540.068312] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2540.068321] lustre(O) osp(O) [ 2540.068328] ? apic_timer_interrupt+0xf/0x20 [ 2540.068330] ofd(O) lod(O) mdt(O) [ 2540.068339] [ 2540.068340] mdd(O) mgs(O) lquota(O) [ 2540.068346] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2540.068350] lfsck(O) obdecho(O) mgc(O) mdc(O) [ 2540.068358] do_raw_spin_lock+0x177/0x190 [ 2540.068360] lov(O) osc(O) lmv(O) ec(O) [ 2540.068369] _raw_spin_lock+0x21/0x30 [ 2540.068375] fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2540.068417] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2540.068421] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2540.068426] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2540.068381] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2540.069668] ? do_raw_spin_unlock+0x75/0x190 [ 2540.069675] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2540.071084] [ 2540.071088] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2540.071100] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2540.071105] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2540.071110] RAX: 0000000000005807 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2540.071112] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 2540.071114] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 2540.071116] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 2540.071119] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 2540.071122] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 2540.071125] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2540.071127] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 2540.071134] Call Trace: [ 2540.071138] [ 2540.071142] ? show_regs.cold.9+0x22/0x2f [ 2540.071151] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2540.071156] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2540.071161] ? watchdog+0x40/0x40 [ 2540.071165] ? hrtimer_interrupt+0x128/0x2c0 [ 2540.071169] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2540.071177] ? apic_timer_interrupt+0xf/0x20 [ 2540.071182] [ 2540.071184] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2540.071189] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2540.071194] do_raw_spin_lock+0x177/0x190 [ 2540.071198] _raw_spin_lock+0x21/0x30 [ 2540.071202] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2540.071424] process_one_work+0x2c8/0x700 [ 2540.071431] worker_thread+0x296/0x6e0 [ 2540.071436] ? rescuer_thread+0x570/0x570 [ 2540.071440] kthread+0x1d1/0x200 [ 2540.071444] ? set_kthread_struct+0x70/0x70 [ 2540.071447] ret_from_fork+0x1f/0x30 [ 2540.073189] ? do_wait_intr+0xf0/0xf0 [ 2540.073203] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2540.073412] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2540.075989] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2540.076242] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2540.076348] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2540.076384] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2540.076415] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2540.076443] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2540.089270] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2540.089348] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2540.089356] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2540.089359] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2540.089372] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2540.089377] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2540.089382] RAX: 0000000000002c1d RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2540.089384] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 2540.089386] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2540.089388] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 2540.089391] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 2540.089394] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 2540.089397] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2540.089399] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 2540.089407] Call Trace: [ 2540.089414] [ 2540.089419] ? show_regs.cold.9+0x22/0x2f [ 2540.089429] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2540.089436] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2540.089440] ? watchdog+0x40/0x40 [ 2540.089444] ? hrtimer_interrupt+0x128/0x2c0 [ 2540.089448] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2540.089457] ? apic_timer_interrupt+0xf/0x20 [ 2540.089461] [ 2540.089464] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2540.089470] do_raw_spin_lock+0x177/0x190 [ 2540.089475] _raw_spin_lock+0x21/0x30 [ 2540.089479] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2540.089754] ? do_raw_spin_unlock+0x75/0x190 [ 2540.089760] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2540.174071] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2540.174555] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2540.174597] lmv_intent_open+0x338/0x1150 [lmv] [ 2540.175680] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2540.175871] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2540.175955] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2540.175977] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2540.176060] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2540.176142] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2540.185571] ? do_wait_intr+0xf0/0xf0 [ 2540.185592] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2540.186070] mdc_close+0x53b/0xd70 [mdc] [ 2540.186121] lmv_close+0x18d/0x490 [lmv] [ 2540.186151] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 2540.186316] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2540.186442] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2540.186530] ll_md_close+0x274/0x800 [lustre] [ 2540.186625] ll_file_release+0x128/0xc50 [lustre] [ 2540.188543] __fput+0x10d/0x440 [ 2540.188553] ____fput+0x12/0x20 [ 2540.188558] task_work_run+0xc6/0x110 [ 2540.188566] exit_to_usermode_loop+0x1dd/0x1f0 [ 2540.188573] do_syscall_64+0x3d6/0x440 [ 2540.188578] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2540.188587] RIP: 0033:0x7ff54af45cd8 [ 2540.188595] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2540.188599] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2540.188605] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 2540.188608] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 2540.188610] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 2540.188612] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 2540.188614] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2541.179527] ? do_raw_spin_unlock+0x75/0x190 [ 2541.189406] ? _raw_spin_unlock+0x12/0x30 [ 2541.197885] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2541.199591] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2541.208366] ll_atomic_open+0x36a/0xd70 [lustre] [ 2541.216551] lookup_open+0x356/0xb40 [ 2541.224175] do_last+0x313/0xfc0 [ 2541.232277] ? nd_jump_root+0xe5/0x160 [ 2541.233160] ? path_init+0x437/0x520 [ 2541.233968] path_openat+0xf7/0x500 [ 2541.234691] ? do_raw_spin_unlock+0x75/0x190 [ 2541.247415] do_filp_open+0x99/0x140 [ 2541.254212] ? getname_flags+0x6e/0x330 [ 2541.257910] ? __check_object_size+0xff/0x256 [ 2541.261435] ? do_raw_spin_unlock+0x75/0x190 [ 2541.267038] ? _raw_spin_unlock+0x12/0x30 [ 2541.273003] do_sys_openat2+0x2b4/0x410 [ 2541.282640] do_sys_open+0x73/0xa0 [ 2541.283746] __x64_sys_openat+0x24/0x30 [ 2541.289204] do_syscall_64+0xc1/0x440 [ 2541.290869] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2541.297275] RIP: 0033:0x7ff54af45332 [ 2541.302230] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2541.326163] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2541.335251] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 2541.349079] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 2541.356501] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 2541.369276] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 2541.377102] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 2544.055055] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [ 2544.058220] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2544.076549] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2544.081088] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2544.084567] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2544.086433] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2544.094318] RSP: 0000:ffffaf9fc1b97bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2544.097324] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2544.100165] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 2544.102901] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 2544.105544] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 2544.108284] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2544.111115] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 2544.114447] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2544.116691] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 2544.119441] Call Trace: [ 2544.120358] [ 2544.121220] ? show_regs.cold.9+0x22/0x2f [ 2544.122782] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2544.124720] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2544.126521] ? watchdog+0x40/0x40 [ 2544.127785] ? hrtimer_interrupt+0x128/0x2c0 [ 2544.128844] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2544.130089] ? apic_timer_interrupt+0xf/0x20 [ 2544.131374] [ 2544.132265] ? native_safe_halt+0x2e/0x50 [ 2544.133885] kvm_wait+0xcd/0xe0 [ 2544.135181] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2544.136717] do_raw_spin_lock+0x177/0x190 [ 2544.138160] _raw_spin_lock+0x21/0x30 [ 2544.139478] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2544.141926] obd_mod_free+0x46/0x140 [ptlrpc] [ 2544.144320] mdc_commit_open+0x91/0xb0 [mdc] [ 2544.146154] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2544.148467] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2544.150926] after_reply+0xe2c/0x1780 [ptlrpc] [ 2544.153103] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2544.155288] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2544.157268] ? do_wait_intr+0xf0/0xf0 [ 2544.158731] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2544.161053] kthread+0x1d1/0x200 [ 2544.162375] ? set_kthread_struct+0x70/0x70 [ 2544.164164] ret_from_fork+0x1f/0x30 [ 2548.198924] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 589s! [ 2548.204112] Showing busy workqueues and worker pools: [ 2548.205604] workqueue events: flags=0x0 [ 2548.207100] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2548.209279] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 2548.213014] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 2548.215137] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 2548.218530] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2548.220446] pending: vmstat_shepherd [ 2548.221507] workqueue events_long: flags=0x0 [ 2548.222635] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2548.224439] pending: lu_objects_flush [obdclass] [ 2548.226975] workqueue events_unbound: flags=0x2 [ 2548.228525] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 2548.230783] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2548.234377] workqueue events_power_efficient: flags=0x80 [ 2548.236051] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2548.237831] pending: neigh_periodic_work [ 2548.239009] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2548.240865] pending: do_cache_clean, check_lifetime [ 2548.242765] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2548.244505] pending: neigh_periodic_work [ 2548.245795] workqueue writeback: flags=0x4a [ 2548.246934] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2548.248640] pending: wb_workfn, wb_workfn [ 2548.249809] workqueue xprtiod: flags=0xa [ 2548.251753] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 2548.253410] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 2548.256226] workqueue ipv6_addrconf: flags=0x40008 [ 2548.257468] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2548.259501] pending: addrconf_verify_work [ 2548.260802] workqueue knbd0-recv: flags=0x1a [ 2548.262270] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2548.263878] in-flight: 657:recv_work [ 2548.265020] workqueue ptlrpc_pinger: flags=0x2 [ 2548.266577] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2548.268818] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 2548.270779] pool 8: cpus=0-3 flags=0x4 nice=0 hung=589s workers=4 idle: 20831 12642 28859 [ 2548.273530] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2538s workers=2 idle: 659 [ 2553.859351] rcu: INFO: rcu_sched self-detected stall on CPU [ 2553.870607] rcu: 0-...!: (310397 ticks this GP) idle=98a/1/0x4000000000000002 softirq=125154/125154 fqs=959 [ 2553.884516] (t=600019 jiffies g=253585 q=2871) [ 2553.899223] rcu: rcu_sched kthread starved for 594979 jiffies! g253585 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [ 2553.908426] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 2553.913484] rcu: RCU grace-period kthread stack dump: [ 2553.920118] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 2553.936697] Call Trace: [ 2553.944171] __schedule+0x351/0xcb0 [ 2553.945553] schedule+0xc0/0x180 [ 2553.952368] schedule_timeout+0xb4/0x190 [ 2553.965284] ? do_raw_spin_unlock+0x75/0x190 [ 2553.971652] ? __next_timer_interrupt+0x160/0x160 [ 2553.977318] ? prepare_to_swait_event+0xf3/0x140 [ 2553.983228] rcu_gp_kthread+0x8dd/0x15b0 [ 2553.985301] ? force_qs_rnp+0x2c0/0x2c0 [ 2553.987194] kthread+0x1d1/0x200 [ 2553.995269] ? set_kthread_struct+0x70/0x70 [ 2553.997403] ret_from_fork+0x1f/0x30 [ 2554.004751] rcu: Stack dump where RCU GP kthread last ran: [ 2554.010202] NMI backtrace for cpu 0 [ 2554.012427] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2554.019073] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2554.039816] Call Trace: [ 2554.041147] [ 2554.042699] ? dump_stack+0xbb/0x10e [ 2554.044437] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 2554.049416] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 2554.058379] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 2554.068310] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 2554.070458] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 2554.094942] ? rcu_check_gp_kthread_starvation+0x16f/0x188 [ 2554.102594] ? rcu_sched_clock_irq.cold.51+0xce/0x307 [ 2554.109131] ? update_process_times+0x6d/0xc0 [ 2554.112421] ? tick_sched_handle.isra.18+0x27/0x90 [ 2554.116995] ? tick_sched_timer+0x53/0xb0 [ 2554.122175] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2554.128891] ? tick_sched_handle.isra.18+0x90/0x90 [ 2554.131598] ? hrtimer_interrupt+0x128/0x2c0 [ 2554.135463] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2554.137961] ? apic_timer_interrupt+0xf/0x20 [ 2554.145081] [ 2554.153084] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2554.158450] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2554.168732] ? do_raw_spin_lock+0x177/0x190 [ 2554.174126] ? _raw_spin_lock+0x21/0x30 [ 2554.179470] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2554.190385] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 2554.198512] ? mdc_commit_open+0x91/0xb0 [mdc] [ 2554.204103] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2554.219054] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2554.222396] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 2554.232314] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2554.238797] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2554.247064] ? do_wait_intr+0xf0/0xf0 [ 2554.250544] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2554.257425] ? kthread+0x1d1/0x200 [ 2554.262964] ? set_kthread_struct+0x70/0x70 [ 2554.268547] ? ret_from_fork+0x1f/0x30 [ 2554.276968] NMI backtrace for cpu 0 [ 2554.282447] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2554.293981] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2554.302469] Call Trace: [ 2554.303901] [ 2554.308091] ? dump_stack+0xbb/0x10e [ 2554.311986] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 2554.315745] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 2554.326159] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 2554.329386] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 2554.338320] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 2554.343030] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 2554.345358] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 2554.350773] ? update_process_times+0x6d/0xc0 [ 2554.358974] ? tick_sched_handle.isra.18+0x27/0x90 [ 2554.363063] ? tick_sched_timer+0x53/0xb0 [ 2554.364902] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2554.371490] ? tick_sched_handle.isra.18+0x90/0x90 [ 2554.377308] ? hrtimer_interrupt+0x128/0x2c0 [ 2554.379093] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2554.381987] ? apic_timer_interrupt+0xf/0x20 [ 2554.383799] [ 2554.388187] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2554.393419] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2554.398980] ? do_raw_spin_lock+0x177/0x190 [ 2554.404789] ? _raw_spin_lock+0x21/0x30 [ 2554.407463] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2554.411652] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 2554.419978] ? mdc_commit_open+0x91/0xb0 [mdc] [ 2554.426304] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2554.440190] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2554.450132] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 2554.458626] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2554.462319] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2554.468754] ? do_wait_intr+0xf0/0xf0 [ 2554.471556] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2554.479085] ? kthread+0x1d1/0x200 [ 2554.482653] ? set_kthread_struct+0x70/0x70 [ 2554.484760] ? ret_from_fork+0x1f/0x30 [ 2568.058083] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 2568.060114] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 2568.062421] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2568.062519] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2568.062538] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2568.062542] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2568.062559] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2568.062564] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2568.062570] RAX: 0000000000006963 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2568.062572] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 2568.062574] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2568.062577] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 2568.062579] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 2568.062585] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 2568.064402] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 2568.064415] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2568.064498] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2568.064504] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2568.064511] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2568.065817] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2568.065831] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2568.065835] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2568.065841] RAX: 0000000000004f75 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2568.065843] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 2568.065846] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 2568.065848] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 2568.065851] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 2568.065855] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 2568.065858] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2568.065860] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 2568.065869] Call Trace: [ 2568.065874] [ 2568.065881] ? show_regs.cold.9+0x22/0x2f [ 2568.065890] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2568.065898] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2568.065903] ? watchdog+0x40/0x40 [ 2568.065908] ? hrtimer_interrupt+0x128/0x2c0 [ 2568.065913] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2568.065921] ? apic_timer_interrupt+0xf/0x20 [ 2568.065926] [ 2568.065929] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2568.065934] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2568.065940] do_raw_spin_lock+0x177/0x190 [ 2568.065945] _raw_spin_lock+0x21/0x30 [ 2568.065950] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2568.067890] process_one_work+0x2c8/0x700 [ 2568.067907] worker_thread+0x296/0x6e0 [ 2568.067912] ? rescuer_thread+0x570/0x570 [ 2568.067916] kthread+0x1d1/0x200 [ 2568.067920] ? set_kthread_struct+0x70/0x70 [ 2568.067924] ret_from_fork+0x1f/0x30 [ 2568.070763] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2568.070840] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2568.070846] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2568.070849] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2568.070862] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2568.070867] RSP: 0000:ffffaf9fc15234f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2568.070871] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2568.070873] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 2568.070876] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2568.070878] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 2568.070880] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 2568.070883] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 2568.070885] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2568.070888] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 2568.070895] Call Trace: [ 2568.070900] [ 2568.070905] ? show_regs.cold.9+0x22/0x2f [ 2568.119607] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2568.128042] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2568.128117] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2568.128122] ? watchdog+0x40/0x40 [ 2568.128126] ? hrtimer_interrupt+0x128/0x2c0 [ 2568.128131] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2568.128139] ? apic_timer_interrupt+0xf/0x20 [ 2568.128145] [ 2568.128149] ? native_safe_halt+0x2e/0x50 [ 2568.132833] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 2568.141814] kvm_wait+0xcd/0xe0 [ 2568.141838] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2568.141849] do_raw_spin_lock+0x177/0x190 [ 2568.141854] _raw_spin_lock+0x21/0x30 [ 2568.141861] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2568.142687] ? do_raw_spin_unlock+0x75/0x190 [ 2568.142694] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2568.143090] ? do_wait_intr+0xf0/0xf0 [ 2568.143100] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2568.165015] Call Trace: [ 2568.165034] [ 2568.165043] ? show_regs.cold.9+0x22/0x2f [ 2568.165064] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2568.165071] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2568.165075] ? watchdog+0x40/0x40 [ 2568.165080] ? hrtimer_interrupt+0x128/0x2c0 [ 2568.165407] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2568.165416] ? apic_timer_interrupt+0xf/0x20 [ 2568.165421] [ 2568.165424] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2568.165436] do_raw_spin_lock+0x177/0x190 [ 2568.165441] _raw_spin_lock+0x21/0x30 [ 2568.165446] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2568.166487] ? do_raw_spin_unlock+0x75/0x190 [ 2568.166495] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2568.166892] ? do_wait_intr+0xf0/0xf0 [ 2568.166903] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2568.174643] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2568.175106] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2568.176822] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2568.177057] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2568.177097] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2568.187983] mdc_close+0x53b/0xd70 [mdc] [ 2568.193856] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2568.197672] lmv_close+0x18d/0x490 [lmv] [ 2568.204052] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2568.208249] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 2568.208688] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2568.208802] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2568.208884] ll_md_close+0x274/0x800 [lustre] [ 2568.208978] ll_file_release+0x128/0xc50 [lustre] [ 2568.209060] __fput+0x10d/0x440 [ 2568.209070] ____fput+0x12/0x20 [ 2568.209074] task_work_run+0xc6/0x110 [ 2568.209081] exit_to_usermode_loop+0x1dd/0x1f0 [ 2568.209087] do_syscall_64+0x3d6/0x440 [ 2568.209091] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2568.209100] RIP: 0033:0x7ff54af45cd8 [ 2568.209106] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2568.209110] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2568.209115] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 2568.209117] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 2568.209118] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 2568.209120] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 2568.209122] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2568.642970] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2568.645722] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2568.647259] lmv_intent_open+0x338/0x1150 [lmv] [ 2568.649511] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2568.651959] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2568.653581] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2568.654791] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2568.655956] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2568.657323] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2568.658598] ? do_raw_spin_unlock+0x75/0x190 [ 2568.660754] ? _raw_spin_unlock+0x12/0x30 [ 2568.662566] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2568.664332] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2568.666963] ll_atomic_open+0x36a/0xd70 [lustre] [ 2568.668911] lookup_open+0x356/0xb40 [ 2568.669877] do_last+0x313/0xfc0 [ 2568.671535] ? nd_jump_root+0xe5/0x160 [ 2568.672871] ? path_init+0x437/0x520 [ 2568.674267] path_openat+0xf7/0x500 [ 2568.675414] ? do_raw_spin_unlock+0x75/0x190 [ 2568.677072] do_filp_open+0x99/0x140 [ 2568.678636] ? getname_flags+0x6e/0x330 [ 2568.683031] ? __check_object_size+0xff/0x256 [ 2568.684332] ? do_raw_spin_unlock+0x75/0x190 [ 2568.686300] ? _raw_spin_unlock+0x12/0x30 [ 2568.687817] do_sys_openat2+0x2b4/0x410 [ 2568.689810] do_sys_open+0x73/0xa0 [ 2568.691491] __x64_sys_openat+0x24/0x30 [ 2568.693330] do_syscall_64+0xc1/0x440 [ 2568.695284] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2568.697731] RIP: 0033:0x7ff54af45332 [ 2568.699605] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2568.706669] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2568.710622] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 2568.714471] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 2568.717911] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 2568.721188] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 2568.725225] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 2578.399244] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 619s! [ 2578.403882] Showing busy workqueues and worker pools: [ 2578.406467] workqueue events: flags=0x0 [ 2578.408403] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2578.413326] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 2578.418676] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 2578.421682] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 2578.426612] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2578.430778] pending: vmstat_shepherd [ 2578.434136] workqueue events_long: flags=0x0 [ 2578.436099] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2578.440847] pending: lu_objects_flush [obdclass] [ 2578.445723] workqueue events_unbound: flags=0x2 [ 2578.447180] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 2578.451225] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2578.455801] workqueue events_power_efficient: flags=0x80 [ 2578.459754] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2578.463066] pending: neigh_periodic_work [ 2578.465701] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2578.469508] pending: do_cache_clean, check_lifetime [ 2578.471412] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2578.474567] pending: neigh_periodic_work [ 2578.476605] workqueue writeback: flags=0x4a [ 2578.479147] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2578.482675] pending: wb_workfn, wb_workfn [ 2578.485972] workqueue xprtiod: flags=0xa [ 2578.489163] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 2578.491673] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 2578.496554] workqueue ipv6_addrconf: flags=0x40008 [ 2578.498838] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2578.502182] pending: addrconf_verify_work [ 2578.503964] workqueue knbd0-recv: flags=0x1a [ 2578.507796] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2578.513653] in-flight: 657:recv_work [ 2578.517794] workqueue ptlrpc_pinger: flags=0x2 [ 2578.522462] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2578.526781] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 2578.531302] pool 8: cpus=0-3 flags=0x4 nice=0 hung=619s workers=4 idle: 20831 12642 28859 [ 2578.534445] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2568s workers=2 idle: 659 [ 2596.058010] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 2596.062328] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2596.062950] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 2596.063235] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 2596.063240] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2596.063320] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2596.063326] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2596.063329] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2596.063380] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2596.063385] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2596.063390] RAX: 0000000000001c0c RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2596.063392] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 2596.063394] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2596.063397] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 2596.063399] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 2596.063402] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 2596.063405] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2596.063408] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 2596.063416] Call Trace: [ 2596.063430] [ 2596.063436] ? show_regs.cold.9+0x22/0x2f [ 2596.063446] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2596.063453] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2596.066267] ? watchdog+0x40/0x40 [ 2596.066274] ? hrtimer_interrupt+0x128/0x2c0 [ 2596.066279] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2596.066287] ? apic_timer_interrupt+0xf/0x20 [ 2596.066292] [ 2596.066297] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2596.066307] do_raw_spin_lock+0x177/0x190 [ 2596.066313] _raw_spin_lock+0x21/0x30 [ 2596.066317] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2596.066695] ? do_raw_spin_unlock+0x75/0x190 [ 2596.066702] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2596.066910] ? do_wait_intr+0xf0/0xf0 [ 2596.066920] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2596.069699] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2596.071187] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2596.073085] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2596.073218] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2596.073272] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2596.073304] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2596.073333] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2596.073419] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2596.075652] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2596.075690] lmv_intent_open+0x338/0x1150 [lmv] [ 2596.075720] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2596.075824] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2596.075909] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2596.075932] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2596.077285] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2596.077368] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2596.077452] ? do_raw_spin_unlock+0x75/0x190 [ 2596.078191] ? _raw_spin_unlock+0x12/0x30 [ 2596.078199] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2596.078229] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2596.078248] ll_atomic_open+0x36a/0xd70 [lustre] [ 2596.078348] lookup_open+0x356/0xb40 [ 2596.078357] do_last+0x313/0xfc0 [ 2596.078361] ? nd_jump_root+0xe5/0x160 [ 2596.078366] ? path_init+0x437/0x520 [ 2596.078371] path_openat+0xf7/0x500 [ 2596.078376] ? do_raw_spin_unlock+0x75/0x190 [ 2596.078381] do_filp_open+0x99/0x140 [ 2596.078387] ? getname_flags+0x6e/0x330 [ 2596.078392] ? __check_object_size+0xff/0x256 [ 2596.078398] ? do_raw_spin_unlock+0x75/0x190 [ 2596.078402] ? _raw_spin_unlock+0x12/0x30 [ 2596.078406] do_sys_openat2+0x2b4/0x410 [ 2596.078410] do_sys_open+0x73/0xa0 [ 2596.078415] __x64_sys_openat+0x24/0x30 [ 2596.078418] do_syscall_64+0xc1/0x440 [ 2596.078424] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2596.078430] RIP: 0033:0x7ff54af45332 [ 2596.078437] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2596.078441] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2596.078447] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 2596.078449] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 2596.078451] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 2596.078453] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 2596.078456] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 2596.085927] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2596.085940] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2596.085942] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2596.085957] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2596.085961] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2596.085966] RAX: 0000000000001b82 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2596.085969] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 2596.085971] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2596.085974] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 2596.085976] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 2596.085979] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 2596.085982] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2596.085985] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 2596.085994] Call Trace: [ 2596.086000] [ 2596.086006] ? show_regs.cold.9+0x22/0x2f [ 2596.086016] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2596.086022] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2596.086027] ? watchdog+0x40/0x40 [ 2596.086032] ? hrtimer_interrupt+0x128/0x2c0 [ 2596.086036] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2596.086047] ? apic_timer_interrupt+0xf/0x20 [ 2596.086051] [ 2596.086054] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2596.086062] do_raw_spin_lock+0x177/0x190 [ 2596.086067] _raw_spin_lock+0x21/0x30 [ 2596.086071] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2596.086577] ? do_raw_spin_unlock+0x75/0x190 [ 2596.086584] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2596.086868] ? do_wait_intr+0xf0/0xf0 [ 2596.086876] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2596.087073] mdc_close+0x53b/0xd70 [mdc] [ 2596.087360] lmv_close+0x18d/0x490 [lmv] [ 2596.087389] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 2596.087560] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2596.087686] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2596.091928] Modules linked in: [ 2596.095538] ll_md_close+0x274/0x800 [lustre] [ 2596.118551] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2596.119320] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2596.119328] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2596.119335] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2596.119643] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2596.119653] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2596.119657] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2596.119661] RAX: 0000000000006a94 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2596.119663] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 2596.119665] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 2596.119667] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 2596.119670] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 2596.119672] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 2596.119675] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2596.119677] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 2596.119686] Call Trace: [ 2596.119691] [ 2596.119696] ? show_regs.cold.9+0x22/0x2f [ 2596.119705] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2596.119711] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2596.119715] ? watchdog+0x40/0x40 [ 2596.119719] ? hrtimer_interrupt+0x128/0x2c0 [ 2596.119723] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2596.119732] ? apic_timer_interrupt+0xf/0x20 [ 2596.119736] [ 2596.119739] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2596.119744] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2596.119748] do_raw_spin_lock+0x177/0x190 [ 2596.119752] _raw_spin_lock+0x21/0x30 [ 2596.119758] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2596.119970] process_one_work+0x2c8/0x700 [ 2596.119977] worker_thread+0x296/0x6e0 [ 2596.119980] ? rescuer_thread+0x570/0x570 [ 2596.119983] kthread+0x1d1/0x200 [ 2596.119988] ? set_kthread_struct+0x70/0x70 [ 2596.119991] ret_from_fork+0x1f/0x30 [ 2596.924267] ll_file_release+0x128/0xc50 [lustre] [ 2596.926318] __fput+0x10d/0x440 [ 2596.927894] ____fput+0x12/0x20 [ 2596.929299] task_work_run+0xc6/0x110 [ 2596.931002] exit_to_usermode_loop+0x1dd/0x1f0 [ 2596.933880] do_syscall_64+0x3d6/0x440 [ 2596.935170] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2596.938978] RIP: 0033:0x7ff54af45cd8 [ 2596.940977] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2596.959300] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2596.964882] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 2596.973318] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 2596.980945] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 2596.982519] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 2596.989921] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2604.056572] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [ 2604.060328] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2604.096512] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2604.104650] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2604.111762] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2604.113900] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2604.133481] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2604.136663] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2604.140414] RDX: 0000000000006476 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 2604.143927] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 2604.150354] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 2604.157744] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2604.165247] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 2604.170485] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2604.172979] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 2604.186529] Call Trace: [ 2604.188204] [ 2604.189097] ? show_regs.cold.9+0x22/0x2f [ 2604.191944] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2604.196343] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2604.199327] ? watchdog+0x40/0x40 [ 2604.203776] ? hrtimer_interrupt+0x128/0x2c0 [ 2604.206017] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2604.209477] ? apic_timer_interrupt+0xf/0x20 [ 2604.214461] [ 2604.215306] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2604.218491] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2604.220776] do_raw_spin_lock+0x177/0x190 [ 2604.223624] _raw_spin_lock+0x21/0x30 [ 2604.225324] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2604.227822] obd_mod_free+0x46/0x140 [ptlrpc] [ 2604.229769] mdc_commit_open+0x91/0xb0 [mdc] [ 2604.232879] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2604.235767] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2604.238554] after_reply+0xe2c/0x1780 [ptlrpc] [ 2604.241796] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2604.244606] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2604.246419] ? do_wait_intr+0xf0/0xf0 [ 2604.250198] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2604.252805] kthread+0x1d1/0x200 [ 2604.254292] ? set_kthread_struct+0x70/0x70 [ 2604.256366] ret_from_fork+0x1f/0x30 [ 2608.607309] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 649s! [ 2608.614350] Showing busy workqueues and worker pools: [ 2608.619116] workqueue events: flags=0x0 [ 2608.624373] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2608.632185] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 2608.641707] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 2608.649629] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 2608.663613] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2608.667955] pending: vmstat_shepherd [ 2608.671835] workqueue events_long: flags=0x0 [ 2608.675033] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2608.679566] pending: lu_objects_flush [obdclass] [ 2608.685703] workqueue events_unbound: flags=0x2 [ 2608.687638] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 2608.695673] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2608.717304] workqueue events_power_efficient: flags=0x80 [ 2608.722745] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2608.729580] pending: neigh_periodic_work [ 2608.734267] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2608.741262] pending: do_cache_clean, check_lifetime [ 2608.746419] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2608.753547] pending: neigh_periodic_work [ 2608.758496] workqueue writeback: flags=0x4a [ 2608.762208] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2608.768521] pending: wb_workfn, wb_workfn [ 2608.769962] workqueue xprtiod: flags=0xa [ 2608.774510] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 2608.780000] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 2608.785706] workqueue ipv6_addrconf: flags=0x40008 [ 2608.789240] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2608.795644] pending: addrconf_verify_work [ 2608.801402] workqueue knbd0-recv: flags=0x1a [ 2608.805512] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2608.812212] in-flight: 657:recv_work [ 2608.817073] workqueue ptlrpc_pinger: flags=0x2 [ 2608.823173] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2608.828079] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 2608.833188] pool 8: cpus=0-3 flags=0x4 nice=0 hung=649s workers=4 idle: 20831 12642 28859 [ 2608.842096] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2599s workers=2 idle: 659 [ 2624.058281] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 2624.064102] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 2624.065531] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 2624.065543] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2624.065659] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2624.065665] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2624.065668] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2624.065682] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2624.065687] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2624.065691] RAX: 0000000000007f16 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2624.065694] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 2624.065696] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2624.065698] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 2624.065701] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 2624.065705] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 2624.065707] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2624.065710] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 2624.065718] Call Trace: [ 2624.065724] [ 2624.065729] ? show_regs.cold.9+0x22/0x2f [ 2624.065740] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2624.065747] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2624.065751] ? watchdog+0x40/0x40 [ 2624.065756] ? hrtimer_interrupt+0x128/0x2c0 [ 2624.065760] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2624.065771] ? apic_timer_interrupt+0xf/0x20 [ 2624.065776] [ 2624.065779] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2624.065786] do_raw_spin_lock+0x177/0x190 [ 2624.065790] _raw_spin_lock+0x21/0x30 [ 2624.065795] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2624.066077] ? do_raw_spin_unlock+0x75/0x190 [ 2624.066083] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2624.072900] ? do_wait_intr+0xf0/0xf0 [ 2624.072915] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2624.074114] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2624.074188] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2624.074194] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2624.074197] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2624.076705] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2624.077016] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2624.077244] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2624.077376] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2624.077596] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2624.077626] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2624.077651] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2624.077737] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2624.077955] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2624.077995] lmv_intent_open+0x338/0x1150 [lmv] [ 2624.078031] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2624.078113] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2624.078193] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2624.078214] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2624.078295] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2624.078642] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2624.078724] ? do_raw_spin_unlock+0x75/0x190 [ 2624.078731] ? _raw_spin_unlock+0x12/0x30 [ 2624.078738] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2624.078765] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2624.078786] ll_atomic_open+0x36a/0xd70 [lustre] [ 2624.078869] lookup_open+0x356/0xb40 [ 2624.078879] do_last+0x313/0xfc0 [ 2624.078883] ? nd_jump_root+0xe5/0x160 [ 2624.078887] ? path_init+0x437/0x520 [ 2624.078891] path_openat+0xf7/0x500 [ 2624.078896] ? do_raw_spin_unlock+0x75/0x190 [ 2624.078900] do_filp_open+0x99/0x140 [ 2624.078906] ? getname_flags+0x6e/0x330 [ 2624.078910] ? __check_object_size+0xff/0x256 [ 2624.078917] ? do_raw_spin_unlock+0x75/0x190 [ 2624.078921] ? _raw_spin_unlock+0x12/0x30 [ 2624.078925] do_sys_openat2+0x2b4/0x410 [ 2624.078930] do_sys_open+0x73/0xa0 [ 2624.078934] __x64_sys_openat+0x24/0x30 [ 2624.078938] do_syscall_64+0xc1/0x440 [ 2624.078945] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2624.078951] RIP: 0033:0x7ff54af45332 [ 2624.079889] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2624.079897] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2624.079903] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 2624.079905] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 2624.079908] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 2624.079910] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 2624.079912] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 2624.084443] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2624.084525] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2624.084533] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2624.084541] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2624.084854] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2624.084866] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2624.084870] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2624.084875] RAX: 0000000000007f83 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2624.084877] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 2624.084879] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 2624.084881] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 2624.084883] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 2624.084887] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 2624.084889] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2624.084892] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 2624.084901] Call Trace: [ 2624.084906] [ 2624.084912] ? show_regs.cold.9+0x22/0x2f [ 2624.084921] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2624.084928] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2624.084932] ? watchdog+0x40/0x40 [ 2624.084936] ? hrtimer_interrupt+0x128/0x2c0 [ 2624.084941] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2624.084949] ? apic_timer_interrupt+0xf/0x20 [ 2624.084954] [ 2624.084957] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2624.084962] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2624.084967] do_raw_spin_lock+0x177/0x190 [ 2624.084972] _raw_spin_lock+0x21/0x30 [ 2624.084976] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2624.088156] process_one_work+0x2c8/0x700 [ 2624.088167] worker_thread+0x296/0x6e0 [ 2624.088172] ? rescuer_thread+0x570/0x570 [ 2624.088176] kthread+0x1d1/0x200 [ 2624.088180] ? set_kthread_struct+0x70/0x70 [ 2624.088184] ret_from_fork+0x1f/0x30 [ 2624.839421] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2624.856065] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2624.858748] RAX: 0000000000007e2a RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2624.863733] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 2624.871145] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2624.885163] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 2624.894605] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 2624.904215] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 2624.910429] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2624.913525] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 2624.919672] Call Trace: [ 2624.920655] [ 2624.921640] ? show_regs.cold.9+0x22/0x2f [ 2624.924622] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2624.926970] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2624.929207] ? watchdog+0x40/0x40 [ 2624.932664] ? hrtimer_interrupt+0x128/0x2c0 [ 2624.936644] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2624.938587] ? apic_timer_interrupt+0xf/0x20 [ 2624.942582] [ 2624.943320] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2624.950539] do_raw_spin_lock+0x177/0x190 [ 2624.951897] _raw_spin_lock+0x21/0x30 [ 2624.955603] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2624.961636] ? do_raw_spin_unlock+0x75/0x190 [ 2624.962930] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2624.971377] ? do_wait_intr+0xf0/0xf0 [ 2624.972791] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2624.979737] mdc_close+0x53b/0xd70 [mdc] [ 2624.983933] lmv_close+0x18d/0x490 [lmv] [ 2624.988985] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 2624.994626] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2625.001317] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2625.010303] ll_md_close+0x274/0x800 [lustre] [ 2625.013299] ll_file_release+0x128/0xc50 [lustre] [ 2625.017227] __fput+0x10d/0x440 [ 2625.019814] ____fput+0x12/0x20 [ 2625.023342] task_work_run+0xc6/0x110 [ 2625.025083] exit_to_usermode_loop+0x1dd/0x1f0 [ 2625.027992] do_syscall_64+0x3d6/0x440 [ 2625.030148] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2625.038730] RIP: 0033:0x7ff54af45cd8 [ 2625.041140] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2625.058861] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2625.074515] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 2625.085129] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 2625.091183] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 2625.099252] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 2625.105909] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2636.055277] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [ 2636.059951] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2636.085470] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2636.090890] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2636.101929] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2636.105059] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2636.118323] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2636.125005] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2636.128104] RDX: 0000000000006eb3 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 2636.132492] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 2636.137929] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 2636.147237] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2636.150007] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 2636.155623] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2636.157078] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 2636.159875] Call Trace: [ 2636.160526] [ 2636.161152] ? show_regs.cold.9+0x22/0x2f [ 2636.162450] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2636.164024] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2636.165619] ? watchdog+0x40/0x40 [ 2636.167015] ? hrtimer_interrupt+0x128/0x2c0 [ 2636.168329] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2636.169879] ? apic_timer_interrupt+0xf/0x20 [ 2636.171408] [ 2636.172305] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2636.175438] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2636.177073] do_raw_spin_lock+0x177/0x190 [ 2636.178948] _raw_spin_lock+0x21/0x30 [ 2636.180225] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2636.182955] obd_mod_free+0x46/0x140 [ptlrpc] [ 2636.184240] mdc_commit_open+0x91/0xb0 [mdc] [ 2636.185406] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2636.186933] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2636.188428] after_reply+0xe2c/0x1780 [ptlrpc] [ 2636.189785] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2636.191536] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2636.193150] ? do_wait_intr+0xf0/0xf0 [ 2636.194607] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2636.196598] kthread+0x1d1/0x200 [ 2636.197989] ? set_kthread_struct+0x70/0x70 [ 2636.199450] ret_from_fork+0x1f/0x30 [ 2639.327078] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 680s! [ 2639.329586] Showing busy workqueues and worker pools: [ 2639.331848] workqueue events: flags=0x0 [ 2639.333423] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2639.336358] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 2639.346919] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 2639.350299] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 2639.355452] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2639.358514] pending: vmstat_shepherd [ 2639.360390] workqueue events_long: flags=0x0 [ 2639.362332] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2639.365307] pending: lu_objects_flush [obdclass] [ 2639.367930] workqueue events_unbound: flags=0x2 [ 2639.369964] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 2639.373161] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2639.378601] workqueue events_power_efficient: flags=0x80 [ 2639.381171] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2639.383713] pending: neigh_periodic_work [ 2639.385345] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2639.387876] pending: do_cache_clean, check_lifetime [ 2639.389810] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2639.392105] pending: neigh_periodic_work [ 2639.393766] workqueue writeback: flags=0x4a [ 2639.395117] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2639.397577] pending: wb_workfn, wb_workfn [ 2639.399203] workqueue xprtiod: flags=0xa [ 2639.400572] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 2639.403030] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 2639.405925] workqueue ipv6_addrconf: flags=0x40008 [ 2639.407423] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2639.409792] pending: addrconf_verify_work [ 2639.411335] workqueue knbd0-recv: flags=0x1a [ 2639.412962] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2639.415278] in-flight: 657:recv_work [ 2639.416853] workqueue ptlrpc_pinger: flags=0x2 [ 2639.418317] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2639.420255] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 2639.422522] pool 8: cpus=0-3 flags=0x4 nice=0 hung=680s workers=4 idle: 20831 12642 28859 [ 2639.425449] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2629s workers=2 idle: 659 [ 2652.062756] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 2652.065035] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2652.074342] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 2652.074532] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 2652.074548] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2652.074615] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2652.074620] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2652.074626] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2652.074921] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2652.074932] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2652.074937] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2652.074941] RAX: 0000000000007e5f RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2652.074954] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 2652.074956] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 2652.074959] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 2652.074961] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 2652.074964] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 2652.074967] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2652.074969] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 2652.074977] Call Trace: [ 2652.074982] [ 2652.074988] ? show_regs.cold.9+0x22/0x2f [ 2652.074998] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2652.075004] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2652.075007] ? watchdog+0x40/0x40 [ 2652.075027] ? hrtimer_interrupt+0x128/0x2c0 [ 2652.075032] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2652.075040] ? apic_timer_interrupt+0xf/0x20 [ 2652.075044] [ 2652.075047] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2652.075052] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2652.075056] do_raw_spin_lock+0x177/0x190 [ 2652.075061] _raw_spin_lock+0x21/0x30 [ 2652.075066] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2652.075297] process_one_work+0x2c8/0x700 [ 2652.075305] worker_thread+0x296/0x6e0 [ 2652.075310] ? rescuer_thread+0x570/0x570 [ 2652.075313] kthread+0x1d1/0x200 [ 2652.075318] ? set_kthread_struct+0x70/0x70 [ 2652.075321] ret_from_fork+0x1f/0x30 [ 2652.158432] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2652.158445] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2652.158449] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2652.158461] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2652.158465] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2652.158470] RAX: 0000000000007686 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2652.158472] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 2652.158474] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2652.158476] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 2652.158478] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 2652.158481] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 2652.158483] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2652.158485] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 2652.158493] Call Trace: [ 2652.158498] [ 2652.158503] ? show_regs.cold.9+0x22/0x2f [ 2652.158511] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2652.158517] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2652.158521] ? watchdog+0x40/0x40 [ 2652.158525] ? hrtimer_interrupt+0x128/0x2c0 [ 2652.158528] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2652.158793] ? apic_timer_interrupt+0xf/0x20 [ 2652.158799] [ 2652.158802] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2652.158924] do_raw_spin_lock+0x177/0x190 [ 2652.158930] _raw_spin_lock+0x21/0x30 [ 2652.158934] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2652.159505] ? do_raw_spin_unlock+0x75/0x190 [ 2652.159511] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2652.160278] ? do_wait_intr+0xf0/0xf0 [ 2652.160288] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2652.161145] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2652.161547] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2652.164650] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2652.164776] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2652.165003] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2652.165574] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2652.165603] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2652.178108] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2652.178180] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2652.178187] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2652.178189] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2652.178203] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2652.178208] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2652.178212] RAX: 0000000000007d85 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2652.178215] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 2652.178217] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2652.178219] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 2652.178221] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 2652.178225] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 2652.178228] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2652.178230] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 2652.178239] Call Trace: [ 2652.178244] [ 2652.178250] ? show_regs.cold.9+0x22/0x2f [ 2652.189480] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2652.271091] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2652.271110] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2652.271114] ? watchdog+0x40/0x40 [ 2652.271119] ? hrtimer_interrupt+0x128/0x2c0 [ 2652.271124] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2652.271131] ? apic_timer_interrupt+0xf/0x20 [ 2652.271135] [ 2652.271140] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2652.271151] do_raw_spin_lock+0x177/0x190 [ 2652.271157] _raw_spin_lock+0x21/0x30 [ 2652.271162] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2652.271902] ? do_raw_spin_unlock+0x75/0x190 [ 2652.271908] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2652.272173] ? do_wait_intr+0xf0/0xf0 [ 2652.272181] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2652.272402] mdc_close+0x53b/0xd70 [mdc] [ 2652.272895] lmv_close+0x18d/0x490 [lmv] [ 2652.272920] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 2652.275353] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2652.288250] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2652.288303] lmv_intent_open+0x338/0x1150 [lmv] [ 2652.288333] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2652.288441] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2652.288524] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2652.288607] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2652.288693] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2652.288789] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2652.288985] ? do_raw_spin_unlock+0x75/0x190 [ 2652.288993] ? _raw_spin_unlock+0x12/0x30 [ 2652.289001] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2652.289093] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2652.289113] ll_atomic_open+0x36a/0xd70 [lustre] [ 2652.289201] lookup_open+0x356/0xb40 [ 2652.289210] do_last+0x313/0xfc0 [ 2652.289215] ? nd_jump_root+0xe5/0x160 [ 2652.289220] ? path_init+0x437/0x520 [ 2652.289225] path_openat+0xf7/0x500 [ 2652.289230] ? do_raw_spin_unlock+0x75/0x190 [ 2652.289234] do_filp_open+0x99/0x140 [ 2652.289241] ? getname_flags+0x6e/0x330 [ 2652.289246] ? __check_object_size+0xff/0x256 [ 2652.289254] ? do_raw_spin_unlock+0x75/0x190 [ 2652.289259] ? _raw_spin_unlock+0x12/0x30 [ 2652.289264] do_sys_openat2+0x2b4/0x410 [ 2652.289269] do_sys_open+0x73/0xa0 [ 2652.289273] __x64_sys_openat+0x24/0x30 [ 2652.289276] do_syscall_64+0xc1/0x440 [ 2652.289283] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2652.289289] RIP: 0033:0x7ff54af45332 [ 2652.289297] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2652.289301] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2652.289306] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 2652.289309] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 2652.289311] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 2652.289314] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 2652.289316] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 2653.556437] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2653.558148] ll_md_close+0x274/0x800 [lustre] [ 2653.559369] ll_file_release+0x128/0xc50 [lustre] [ 2653.583470] __fput+0x10d/0x440 [ 2653.591201] ____fput+0x12/0x20 [ 2653.601390] task_work_run+0xc6/0x110 [ 2653.603961] exit_to_usermode_loop+0x1dd/0x1f0 [ 2653.611448] do_syscall_64+0x3d6/0x440 [ 2653.619236] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2653.632918] RIP: 0033:0x7ff54af45cd8 [ 2653.646195] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2653.675390] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2653.685553] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 2653.693902] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 2653.706217] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 2653.721686] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 2653.734118] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2664.063436] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [ 2664.075305] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2664.120182] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2664.130891] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2664.138532] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2664.141789] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2664.153524] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2664.158191] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2664.173621] RDX: 000000000000263e RSI: 0000000000000003 RDI: ffff96848811cae0 [ 2664.180329] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 2664.190712] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 2664.193276] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2664.210251] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 2664.217111] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2664.221388] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 2664.226716] Call Trace: [ 2664.228895] [ 2664.235157] ? show_regs.cold.9+0x22/0x2f [ 2664.237722] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2664.241256] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2664.244547] ? watchdog+0x40/0x40 [ 2664.250402] ? hrtimer_interrupt+0x128/0x2c0 [ 2664.254439] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2664.267797] ? apic_timer_interrupt+0xf/0x20 [ 2664.272791] [ 2664.274745] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2664.288521] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2664.294274] do_raw_spin_lock+0x177/0x190 [ 2664.296790] _raw_spin_lock+0x21/0x30 [ 2664.299613] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2664.303346] obd_mod_free+0x46/0x140 [ptlrpc] [ 2664.309536] mdc_commit_open+0x91/0xb0 [mdc] [ 2664.318203] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2664.320266] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2664.338192] after_reply+0xe2c/0x1780 [ptlrpc] [ 2664.341638] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2664.347709] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2664.351193] ? do_wait_intr+0xf0/0xf0 [ 2664.358768] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2664.362579] kthread+0x1d1/0x200 [ 2664.364448] ? set_kthread_struct+0x70/0x70 [ 2664.365960] ret_from_fork+0x1f/0x30 [ 2669.535046] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 710s! [ 2669.538138] Showing busy workqueues and worker pools: [ 2669.540239] workqueue events: flags=0x0 [ 2669.542505] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2669.546524] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 2669.552185] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 2669.558133] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 2669.567530] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2669.571149] pending: vmstat_shepherd [ 2669.573171] workqueue events_long: flags=0x0 [ 2669.575289] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2669.578756] pending: lu_objects_flush [obdclass] [ 2669.581474] workqueue events_unbound: flags=0x2 [ 2669.583741] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 2669.586559] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2669.591621] workqueue events_power_efficient: flags=0x80 [ 2669.595334] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2669.600020] pending: neigh_periodic_work [ 2669.602549] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2669.607745] pending: do_cache_clean, check_lifetime [ 2669.609653] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2669.625021] pending: neigh_periodic_work [ 2669.627597] workqueue writeback: flags=0x4a [ 2669.631341] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2669.641925] pending: wb_workfn, wb_workfn [ 2669.649584] workqueue xprtiod: flags=0xa [ 2669.651154] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 2669.659201] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 2669.672172] workqueue ipv6_addrconf: flags=0x40008 [ 2669.678948] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2669.691105] pending: addrconf_verify_work [ 2669.702353] workqueue knbd0-recv: flags=0x1a [ 2669.710170] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2669.721347] in-flight: 657:recv_work [ 2669.725302] workqueue ptlrpc_pinger: flags=0x2 [ 2669.728970] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2669.736331] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 2669.745232] pool 8: cpus=0-3 flags=0x4 nice=0 hung=710s workers=4 idle: 20831 12642 28859 [ 2669.754841] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2659s workers=2 idle: 659 [ 2680.058121] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 2680.062406] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2680.063195] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 2680.069307] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [ 2680.069321] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2680.069391] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2680.069397] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2680.069400] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2680.069415] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2680.069419] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2680.069423] RAX: 0000000000007836 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2680.069425] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 2680.069427] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2680.069429] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 2680.069431] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 2680.069435] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 2680.069448] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2680.069450] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 2680.069459] Call Trace: [ 2680.069463] [ 2680.069469] ? show_regs.cold.9+0x22/0x2f [ 2680.069478] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2680.069485] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2680.069489] ? watchdog+0x40/0x40 [ 2680.069494] ? hrtimer_interrupt+0x128/0x2c0 [ 2680.069498] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2680.069507] ? apic_timer_interrupt+0xf/0x20 [ 2680.069511] [ 2680.069514] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2680.069521] do_raw_spin_lock+0x177/0x190 [ 2680.069526] _raw_spin_lock+0x21/0x30 [ 2680.069531] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2680.069966] ? do_raw_spin_unlock+0x75/0x190 [ 2680.069972] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2680.070235] ? do_wait_intr+0xf0/0xf0 [ 2680.070244] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2680.070435] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2680.070656] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2680.071031] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2680.071165] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2680.071202] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2680.071250] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2680.071279] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2680.071368] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2680.071612] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2680.071640] lmv_intent_open+0x338/0x1150 [lmv] [ 2680.071674] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2680.071830] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2680.071917] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2680.072042] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2680.072132] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2680.072209] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2680.072287] ? do_raw_spin_unlock+0x75/0x190 [ 2680.072294] ? _raw_spin_unlock+0x12/0x30 [ 2680.072300] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2680.072339] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2680.072355] ll_atomic_open+0x36a/0xd70 [lustre] [ 2680.072468] lookup_open+0x356/0xb40 [ 2680.072477] do_last+0x313/0xfc0 [ 2680.072482] ? nd_jump_root+0xe5/0x160 [ 2680.072486] ? path_init+0x437/0x520 [ 2680.072491] path_openat+0xf7/0x500 [ 2680.072496] ? do_raw_spin_unlock+0x75/0x190 [ 2680.072500] do_filp_open+0x99/0x140 [ 2680.072506] ? getname_flags+0x6e/0x330 [ 2680.072511] ? __check_object_size+0xff/0x256 [ 2680.072518] ? do_raw_spin_unlock+0x75/0x190 [ 2680.072522] ? _raw_spin_unlock+0x12/0x30 [ 2680.072526] do_sys_openat2+0x2b4/0x410 [ 2680.072531] do_sys_open+0x73/0xa0 [ 2680.072536] __x64_sys_openat+0x24/0x30 [ 2680.072539] do_syscall_64+0xc1/0x440 [ 2680.072546] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2680.072553] RIP: 0033:0x7ff54af45332 [ 2680.072561] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2680.072565] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2680.072571] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 2680.072573] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 2680.072575] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 2680.072578] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 2680.072580] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 2680.094336] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2680.094352] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2680.094355] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2680.094370] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2680.094375] RSP: 0018:ffffaf9fc1543ae0 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 2680.094382] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2680.094384] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 2680.094386] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2680.094388] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 2680.098149] Modules linked in: [ 2680.116399] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 2680.129880] lustre(O) [ 2680.142493] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 2680.142504] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2680.142507] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 2680.142516] Call Trace: [ 2680.142525] [ 2680.142532] ? show_regs.cold.9+0x22/0x2f [ 2680.146684] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2680.146774] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2680.146783] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2680.146791] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2680.147146] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2680.147157] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2680.147161] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2680.147167] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2680.147169] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 2680.147171] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 2680.147173] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 2680.147176] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 2680.147180] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 2680.147182] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2680.147185] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 2680.147194] Call Trace: [ 2680.147201] [ 2680.147208] ? show_regs.cold.9+0x22/0x2f [ 2680.147222] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2680.147230] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2680.147237] ? watchdog+0x40/0x40 [ 2680.147243] ? hrtimer_interrupt+0x128/0x2c0 [ 2680.147247] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2680.147256] ? apic_timer_interrupt+0xf/0x20 [ 2680.147260] [ 2680.147397] ? native_safe_halt+0x2e/0x50 [ 2680.147404] kvm_wait+0xcd/0xe0 [ 2680.147415] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2680.147425] do_raw_spin_lock+0x177/0x190 [ 2680.147430] _raw_spin_lock+0x21/0x30 [ 2680.147435] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2680.147755] process_one_work+0x2c8/0x700 [ 2680.147764] worker_thread+0x296/0x6e0 [ 2680.147769] ? rescuer_thread+0x570/0x570 [ 2680.147772] kthread+0x1d1/0x200 [ 2680.147777] ? set_kthread_struct+0x70/0x70 [ 2680.147782] ret_from_fork+0x1f/0x30 [ 2680.737389] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2680.742662] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2680.744400] ? watchdog+0x40/0x40 [ 2680.745809] ? hrtimer_interrupt+0x128/0x2c0 [ 2680.747612] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2680.749717] ? apic_timer_interrupt+0xf/0x20 [ 2680.751223] [ 2680.751968] ? native_safe_halt+0x2e/0x50 [ 2680.753730] kvm_wait+0xcd/0xe0 [ 2680.755016] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2680.757255] do_raw_spin_lock+0x177/0x190 [ 2680.758965] _raw_spin_lock+0x21/0x30 [ 2680.760493] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2680.762831] ? do_raw_spin_unlock+0x75/0x190 [ 2680.764637] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2680.766865] ? do_wait_intr+0xf0/0xf0 [ 2680.768301] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2680.770044] mdc_close+0x53b/0xd70 [mdc] [ 2680.771239] lmv_close+0x18d/0x490 [lmv] [ 2680.773221] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 2680.775947] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2680.778608] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2680.780791] ll_md_close+0x274/0x800 [lustre] [ 2680.792230] ll_file_release+0x128/0xc50 [lustre] [ 2680.796211] __fput+0x10d/0x440 [ 2680.798932] ____fput+0x12/0x20 [ 2680.800083] task_work_run+0xc6/0x110 [ 2680.803150] exit_to_usermode_loop+0x1dd/0x1f0 [ 2680.808671] do_syscall_64+0x3d6/0x440 [ 2680.812347] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2680.814617] RIP: 0033:0x7ff54af45cd8 [ 2680.817045] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2680.836953] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2680.842216] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 2680.844981] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 2680.851571] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 2680.860917] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 2680.863991] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2696.057005] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 2696.060324] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2696.090228] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2696.094501] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2696.098317] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2696.100582] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2696.109229] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2696.114461] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2696.118812] RDX: 0000000000006704 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 2696.123659] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 2696.128341] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 2696.131578] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2696.135661] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 2696.142234] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2696.143855] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 2696.148338] Call Trace: [ 2696.150221] [ 2696.150653] ? show_regs.cold.9+0x22/0x2f [ 2696.153885] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2696.158917] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2696.162054] ? watchdog+0x40/0x40 [ 2696.164234] ? hrtimer_interrupt+0x128/0x2c0 [ 2696.166539] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2696.169683] ? apic_timer_interrupt+0xf/0x20 [ 2696.173691] [ 2696.174222] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2696.180127] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2696.181825] do_raw_spin_lock+0x177/0x190 [ 2696.187550] _raw_spin_lock+0x21/0x30 [ 2696.192594] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2696.198843] obd_mod_free+0x46/0x140 [ptlrpc] [ 2696.206907] mdc_commit_open+0x91/0xb0 [mdc] [ 2696.211690] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2696.220906] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2696.229488] after_reply+0xe2c/0x1780 [ptlrpc] [ 2696.232498] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2696.235344] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2696.237967] ? do_wait_intr+0xf0/0xf0 [ 2696.240683] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2696.244699] kthread+0x1d1/0x200 [ 2696.247005] ? set_kthread_struct+0x70/0x70 [ 2696.248794] ret_from_fork+0x1f/0x30 [ 2700.255078] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 741s! [ 2700.257920] Showing busy workqueues and worker pools: [ 2700.260537] workqueue events: flags=0x0 [ 2700.262500] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2700.265775] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 2700.270641] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 2700.274866] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 2700.281022] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2700.284625] pending: vmstat_shepherd [ 2700.286567] workqueue events_long: flags=0x0 [ 2700.288747] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2700.291880] pending: lu_objects_flush [obdclass] [ 2700.294525] workqueue events_unbound: flags=0x2 [ 2700.296755] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 2700.301344] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2700.320613] workqueue events_power_efficient: flags=0x80 [ 2700.335738] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2700.348298] pending: neigh_periodic_work [ 2700.352740] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2700.361655] pending: do_cache_clean, check_lifetime [ 2700.371136] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2700.378585] pending: neigh_periodic_work [ 2700.380885] workqueue writeback: flags=0x4a [ 2700.385907] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2700.391994] pending: wb_workfn, wb_workfn [ 2700.396894] workqueue xprtiod: flags=0xa [ 2700.399810] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 2700.407604] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 2700.421888] workqueue ipv6_addrconf: flags=0x40008 [ 2700.426600] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2700.433651] pending: addrconf_verify_work [ 2700.435973] workqueue knbd0-recv: flags=0x1a [ 2700.438886] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2700.442589] in-flight: 657:recv_work [ 2700.446455] workqueue ptlrpc_pinger: flags=0x2 [ 2700.448784] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2700.450679] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 2700.457839] pool 8: cpus=0-3 flags=0x4 nice=0 hung=741s workers=4 idle: 20831 12642 28859 [ 2700.463578] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2690s workers=2 idle: 659 [ 2708.060301] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 2708.062287] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [ 2708.063896] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2708.063969] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2708.063977] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2708.063984] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2708.064638] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2708.064650] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2708.064654] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 2708.064658] RAX: 0000000000005d8f RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2708.064661] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 2708.064663] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 2708.064665] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 2708.064667] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 2708.064670] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 2708.064673] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2708.064676] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 2708.064684] Call Trace: [ 2708.064690] [ 2708.064696] ? show_regs.cold.9+0x22/0x2f [ 2708.064705] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2708.064712] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2708.064717] ? watchdog+0x40/0x40 [ 2708.064721] ? hrtimer_interrupt+0x128/0x2c0 [ 2708.064725] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2708.064735] ? apic_timer_interrupt+0xf/0x20 [ 2708.064740] [ 2708.064743] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2708.064747] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2708.064751] do_raw_spin_lock+0x177/0x190 [ 2708.064756] _raw_spin_lock+0x21/0x30 [ 2708.064760] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2708.064966] process_one_work+0x2c8/0x700 [ 2708.064972] worker_thread+0x296/0x6e0 [ 2708.064976] ? rescuer_thread+0x570/0x570 [ 2708.064979] kthread+0x1d1/0x200 [ 2708.064983] ? set_kthread_struct+0x70/0x70 [ 2708.064986] ret_from_fork+0x1f/0x30 [ 2708.065882] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [ 2708.065890] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2708.066011] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2708.066017] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2708.066020] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2708.066031] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2708.066035] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2708.066039] RAX: 0000000000007f0d RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2708.066041] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 2708.066043] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2708.066045] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 2708.066047] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 2708.066050] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 2708.066052] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2708.066054] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 2708.066062] Call Trace: [ 2708.066065] [ 2708.066069] ? show_regs.cold.9+0x22/0x2f [ 2708.066077] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2708.066082] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2708.066086] ? watchdog+0x40/0x40 [ 2708.066090] ? hrtimer_interrupt+0x128/0x2c0 [ 2708.066094] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2708.066100] ? apic_timer_interrupt+0xf/0x20 [ 2708.066105] [ 2708.066107] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2708.066113] do_raw_spin_lock+0x177/0x190 [ 2708.066118] _raw_spin_lock+0x21/0x30 [ 2708.066121] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2708.067143] ? do_raw_spin_unlock+0x75/0x190 [ 2708.067152] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2708.076566] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2708.076642] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2708.076650] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2708.076652] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2708.076668] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2708.076672] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2708.076677] RAX: 0000000000007b85 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2708.076679] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 2708.076681] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2708.076683] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 2708.076685] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 2708.076689] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 2708.076691] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2708.076693] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 2708.076702] Call Trace: [ 2708.076706] [ 2708.076712] ? show_regs.cold.9+0x22/0x2f [ 2708.076720] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2708.076726] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2708.076730] ? watchdog+0x40/0x40 [ 2708.076734] ? hrtimer_interrupt+0x128/0x2c0 [ 2708.076738] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2708.076747] ? apic_timer_interrupt+0xf/0x20 [ 2708.076751] [ 2708.076754] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2708.076761] do_raw_spin_lock+0x177/0x190 [ 2708.076767] _raw_spin_lock+0x21/0x30 [ 2708.076771] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2708.077086] ? do_raw_spin_unlock+0x75/0x190 [ 2708.077092] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2708.132518] ? do_wait_intr+0xf0/0xf0 [ 2708.132542] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2708.132846] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2708.133074] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2708.133302] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2708.133435] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2708.133479] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2708.133509] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2708.133539] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2708.133627] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2708.133856] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2708.133886] lmv_intent_open+0x338/0x1150 [lmv] [ 2708.133922] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2708.134008] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2708.134089] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2708.134114] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2708.134214] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2708.134299] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2708.134386] ? do_raw_spin_unlock+0x75/0x190 [ 2708.134393] ? _raw_spin_unlock+0x12/0x30 [ 2708.134400] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2708.134428] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2708.134447] ll_atomic_open+0x36a/0xd70 [lustre] [ 2708.134526] lookup_open+0x356/0xb40 [ 2708.134534] do_last+0x313/0xfc0 [ 2708.134539] ? nd_jump_root+0xe5/0x160 [ 2708.134543] ? path_init+0x437/0x520 [ 2708.134548] path_openat+0xf7/0x500 [ 2708.134552] ? do_raw_spin_unlock+0x75/0x190 [ 2708.134557] do_filp_open+0x99/0x140 [ 2708.134563] ? getname_flags+0x6e/0x330 [ 2708.134567] ? __check_object_size+0xff/0x256 [ 2708.134575] ? do_raw_spin_unlock+0x75/0x190 [ 2708.134579] ? _raw_spin_unlock+0x12/0x30 [ 2708.134583] do_sys_openat2+0x2b4/0x410 [ 2708.134587] do_sys_open+0x73/0xa0 [ 2708.134591] __x64_sys_openat+0x24/0x30 [ 2708.134594] do_syscall_64+0xc1/0x440 [ 2708.134601] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2708.134607] RIP: 0033:0x7ff54af45332 [ 2708.134616] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2708.134619] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2708.134625] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 2708.134628] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 2708.134630] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 2708.134632] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 2708.134634] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 2708.947783] ? do_wait_intr+0xf0/0xf0 [ 2708.949491] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2708.953152] mdc_close+0x53b/0xd70 [mdc] [ 2708.956747] lmv_close+0x18d/0x490 [lmv] [ 2708.962951] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 2708.968088] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2708.972333] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2708.976371] ll_md_close+0x274/0x800 [lustre] [ 2708.983222] ll_file_release+0x128/0xc50 [lustre] [ 2708.990366] __fput+0x10d/0x440 [ 2708.996393] ____fput+0x12/0x20 [ 2708.999424] task_work_run+0xc6/0x110 [ 2709.004239] exit_to_usermode_loop+0x1dd/0x1f0 [ 2709.006361] do_syscall_64+0x3d6/0x440 [ 2709.008294] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2709.010461] RIP: 0033:0x7ff54af45cd8 [ 2709.012427] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2709.037287] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2709.045426] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 2709.053631] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 2709.063426] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 2709.072943] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 2709.078858] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2728.056554] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 2728.061479] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2728.120063] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2728.126633] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2728.136242] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2728.138124] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2728.159363] RSP: 0000:ffffaf9fc1b97bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2728.162583] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2728.164859] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 2728.167884] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 2728.173257] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 2728.178155] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2728.183096] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 2728.191660] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2728.194698] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 2728.199645] Call Trace: [ 2728.201467] [ 2728.202400] ? show_regs.cold.9+0x22/0x2f [ 2728.204544] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2728.210122] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2728.212637] ? watchdog+0x40/0x40 [ 2728.213955] ? hrtimer_interrupt+0x128/0x2c0 [ 2728.217423] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2728.221730] ? apic_timer_interrupt+0xf/0x20 [ 2728.227199] [ 2728.227886] ? native_safe_halt+0x2e/0x50 [ 2728.233730] kvm_wait+0xcd/0xe0 [ 2728.235905] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2728.239743] do_raw_spin_lock+0x177/0x190 [ 2728.243715] _raw_spin_lock+0x21/0x30 [ 2728.249355] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2728.259808] obd_mod_free+0x46/0x140 [ptlrpc] [ 2728.261621] mdc_commit_open+0x91/0xb0 [mdc] [ 2728.265367] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2728.273357] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2728.277190] after_reply+0xe2c/0x1780 [ptlrpc] [ 2728.281336] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2728.284558] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2728.286692] ? do_wait_intr+0xf0/0xf0 [ 2728.290068] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2728.295252] kthread+0x1d1/0x200 [ 2728.296756] ? set_kthread_struct+0x70/0x70 [ 2728.299444] ret_from_fork+0x1f/0x30 [ 2730.975069] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 772s! [ 2730.979242] Showing busy workqueues and worker pools: [ 2730.981843] workqueue events: flags=0x0 [ 2730.983975] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2730.987376] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 2730.993019] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 2730.996564] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 2731.002516] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2731.008448] pending: vmstat_shepherd [ 2731.010053] workqueue events_long: flags=0x0 [ 2731.011754] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2731.016089] pending: lu_objects_flush [obdclass] [ 2731.019462] workqueue events_unbound: flags=0x2 [ 2731.022171] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 2731.025577] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2731.032715] workqueue events_power_efficient: flags=0x80 [ 2731.035670] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2731.039323] pending: neigh_periodic_work [ 2731.041609] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2731.045261] pending: do_cache_clean, check_lifetime [ 2731.047547] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2731.050508] pending: neigh_periodic_work [ 2731.052394] workqueue writeback: flags=0x4a [ 2731.054534] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2731.058219] pending: wb_workfn, wb_workfn [ 2731.060742] workqueue xprtiod: flags=0xa [ 2731.062937] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 2731.066198] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 2731.071245] workqueue ipv6_addrconf: flags=0x40008 [ 2731.074349] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2731.077602] pending: addrconf_verify_work [ 2731.080550] workqueue knbd0-recv: flags=0x1a [ 2731.083491] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2731.086664] in-flight: 657:recv_work [ 2731.088402] workqueue ptlrpc_pinger: flags=0x2 [ 2731.091313] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2731.094834] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 2731.098394] pool 8: cpus=0-3 flags=0x4 nice=0 hung=772s workers=4 idle: 20831 12642 28859 [ 2731.102865] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2721s workers=2 idle: 659 [ 2733.862391] rcu: INFO: rcu_sched self-detected stall on CPU [ 2733.871394] rcu: 0-...!: (407940 ticks this GP) idle=98a/1/0x4000000000000002 softirq=125154/125154 fqs=959 [ 2733.881374] (t=780022 jiffies g=253585 q=2940) [ 2733.887038] rcu: rcu_sched kthread starved for 774982 jiffies! g253585 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [ 2733.896168] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 2733.901805] rcu: RCU grace-period kthread stack dump: [ 2733.911267] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 2733.919135] Call Trace: [ 2733.920331] __schedule+0x351/0xcb0 [ 2733.922239] schedule+0xc0/0x180 [ 2733.923860] schedule_timeout+0xb4/0x190 [ 2733.925819] ? do_raw_spin_unlock+0x75/0x190 [ 2733.927467] ? __next_timer_interrupt+0x160/0x160 [ 2733.930021] ? prepare_to_swait_event+0xf3/0x140 [ 2733.932230] rcu_gp_kthread+0x8dd/0x15b0 [ 2733.934236] ? force_qs_rnp+0x2c0/0x2c0 [ 2733.935371] kthread+0x1d1/0x200 [ 2733.936329] ? set_kthread_struct+0x70/0x70 [ 2733.937479] ret_from_fork+0x1f/0x30 [ 2733.939172] rcu: Stack dump where RCU GP kthread last ran: [ 2733.944504] Sending NMI from CPU 0 to CPUs 1: [ 2733.949215] NMI backtrace for cpu 1 skipped: idling at native_safe_halt+0x2e/0x50 [ 2733.950174] NMI backtrace for cpu 0 [ 2733.956366] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2733.966724] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2733.972252] Call Trace: [ 2733.976503] [ 2733.977447] ? dump_stack+0xbb/0x10e [ 2733.979482] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 2733.985085] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 2733.988244] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 2733.990687] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 2733.993962] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 2733.999726] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 2734.009882] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 2734.013159] ? update_process_times+0x6d/0xc0 [ 2734.020158] ? tick_sched_handle.isra.18+0x27/0x90 [ 2734.026165] ? tick_sched_timer+0x53/0xb0 [ 2734.032691] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2734.040123] ? tick_sched_handle.isra.18+0x90/0x90 [ 2734.044070] ? hrtimer_interrupt+0x128/0x2c0 [ 2734.047481] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2734.050062] ? apic_timer_interrupt+0xf/0x20 [ 2734.052137] [ 2734.053291] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2734.057923] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2734.063289] ? do_raw_spin_lock+0x177/0x190 [ 2734.066374] ? _raw_spin_lock+0x21/0x30 [ 2734.069148] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2734.073468] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 2734.076684] ? mdc_commit_open+0x91/0xb0 [mdc] [ 2734.079065] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2734.083390] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2734.087542] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 2734.092130] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2734.096892] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2734.101106] ? do_wait_intr+0xf0/0xf0 [ 2734.104277] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2734.114080] ? kthread+0x1d1/0x200 [ 2734.116645] ? set_kthread_struct+0x70/0x70 [ 2734.118083] ? ret_from_fork+0x1f/0x30 [ 2736.059269] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 2736.060234] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [ 2736.065596] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2736.065672] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2736.065691] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2736.065698] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2736.077225] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2736.077307] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2736.077315] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2736.077318] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2736.077333] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2736.077338] RSP: 0000:ffffaf9fc15234f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2736.077342] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2736.077345] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 2736.077347] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2736.077349] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 2736.077352] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 2736.077355] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 2736.077358] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2736.077360] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 2736.077370] Call Trace: [ 2736.077376] [ 2736.077382] ? show_regs.cold.9+0x22/0x2f [ 2736.077395] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2736.077402] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2736.077406] ? watchdog+0x40/0x40 [ 2736.077410] ? hrtimer_interrupt+0x128/0x2c0 [ 2736.077415] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2736.077423] ? apic_timer_interrupt+0xf/0x20 [ 2736.077428] [ 2736.077432] ? native_safe_halt+0x2e/0x50 [ 2736.104699] [ 2736.104710] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2736.104783] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2736.104788] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2736.104794] RAX: 000000000000526c RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2736.104796] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 2736.104798] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 2736.104800] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 2736.104802] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 2736.104805] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 2736.104808] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2736.104810] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 2736.104819] Call Trace: [ 2736.104825] [ 2736.104830] ? show_regs.cold.9+0x22/0x2f [ 2736.104840] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2736.104846] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2736.104851] ? watchdog+0x40/0x40 [ 2736.104854] ? hrtimer_interrupt+0x128/0x2c0 [ 2736.104857] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2736.104865] ? apic_timer_interrupt+0xf/0x20 [ 2736.104876] [ 2736.104878] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2736.104883] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2736.104890] do_raw_spin_lock+0x177/0x190 [ 2736.104907] _raw_spin_lock+0x21/0x30 [ 2736.104912] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2736.105306] process_one_work+0x2c8/0x700 [ 2736.105320] worker_thread+0x296/0x6e0 [ 2736.105328] ? rescuer_thread+0x570/0x570 [ 2736.105332] kthread+0x1d1/0x200 [ 2736.105338] ? set_kthread_struct+0x70/0x70 [ 2736.105342] ret_from_fork+0x1f/0x30 [ 2736.384162] kvm_wait+0xcd/0xe0 [ 2736.386296] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2736.389152] do_raw_spin_lock+0x177/0x190 [ 2736.392791] _raw_spin_lock+0x21/0x30 [ 2736.395133] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2736.398396] ? do_raw_spin_unlock+0x75/0x190 [ 2736.400858] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2736.403775] ? do_wait_intr+0xf0/0xf0 [ 2736.405802] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2736.408884] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2736.411957] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2736.415237] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2736.418051] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2736.420832] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2736.424443] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2736.426335] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2736.429057] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2736.431880] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2736.434163] lmv_intent_open+0x338/0x1150 [lmv] [ 2736.436844] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2736.442328] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2736.445182] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2736.447415] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2736.453045] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2736.461111] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2736.463265] ? do_raw_spin_unlock+0x75/0x190 [ 2736.466222] ? _raw_spin_unlock+0x12/0x30 [ 2736.467204] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2736.468740] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2736.470324] ll_atomic_open+0x36a/0xd70 [lustre] [ 2736.477867] lookup_open+0x356/0xb40 [ 2736.479248] do_last+0x313/0xfc0 [ 2736.481671] ? nd_jump_root+0xe5/0x160 [ 2736.485688] ? path_init+0x437/0x520 [ 2736.486719] path_openat+0xf7/0x500 [ 2736.487817] ? do_raw_spin_unlock+0x75/0x190 [ 2736.494007] do_filp_open+0x99/0x140 [ 2736.495280] ? getname_flags+0x6e/0x330 [ 2736.501288] ? __check_object_size+0xff/0x256 [ 2736.502600] ? do_raw_spin_unlock+0x75/0x190 [ 2736.510194] ? _raw_spin_unlock+0x12/0x30 [ 2736.515546] do_sys_openat2+0x2b4/0x410 [ 2736.519822] do_sys_open+0x73/0xa0 [ 2736.521783] __x64_sys_openat+0x24/0x30 [ 2736.523989] do_syscall_64+0xc1/0x440 [ 2736.525266] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2736.535037] RIP: 0033:0x7ff54af45332 [ 2736.537781] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2736.550310] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2736.559094] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 2736.564515] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 2736.569027] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 2736.573191] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 2736.580282] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 2760.055314] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 2760.057474] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2760.062378] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 2760.100550] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2760.100566] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2760.100569] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2760.100649] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2760.100653] RSP: 0000:ffffaf9fc1b97bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2760.100659] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2760.100671] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 2760.100673] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 2760.100675] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 2760.100677] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2760.100680] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 2760.100683] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2760.100685] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 2760.100692] Call Trace: [ 2760.100697] [ 2760.100703] ? show_regs.cold.9+0x22/0x2f [ 2760.100713] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2760.100720] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2760.100724] ? watchdog+0x40/0x40 [ 2760.100728] ? hrtimer_interrupt+0x128/0x2c0 [ 2760.100731] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2760.100739] ? apic_timer_interrupt+0xf/0x20 [ 2760.100744] [ 2760.100746] ? native_safe_halt+0x2e/0x50 [ 2760.100751] kvm_wait+0xcd/0xe0 [ 2760.100758] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2760.100766] do_raw_spin_lock+0x177/0x190 [ 2760.100770] _raw_spin_lock+0x21/0x30 [ 2760.100777] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2760.101331] obd_mod_free+0x46/0x140 [ptlrpc] [ 2760.101761] mdc_commit_open+0x91/0xb0 [mdc] [ 2760.101806] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2760.107505] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2760.107587] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2760.107595] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2760.107598] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2760.107613] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2760.107617] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2760.107622] RAX: 0000000000005891 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2760.107624] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 2760.107626] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2760.107628] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 2760.107631] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 2760.107634] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 2760.107636] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2760.107639] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 2760.107648] Call Trace: [ 2760.107653] [ 2760.107679] ? show_regs.cold.9+0x22/0x2f [ 2760.107692] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2760.107700] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2760.107704] ? watchdog+0x40/0x40 [ 2760.107709] ? hrtimer_interrupt+0x128/0x2c0 [ 2760.107714] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2760.107723] ? apic_timer_interrupt+0xf/0x20 [ 2760.107728] [ 2760.107732] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2760.107739] do_raw_spin_lock+0x177/0x190 [ 2760.107745] _raw_spin_lock+0x21/0x30 [ 2760.107750] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2760.109624] ? do_raw_spin_unlock+0x75/0x190 [ 2760.109634] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2760.109906] ? do_wait_intr+0xf0/0xf0 [ 2760.109914] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2760.110134] mdc_close+0x53b/0xd70 [mdc] [ 2760.110173] lmv_close+0x18d/0x490 [lmv] [ 2760.110207] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 2760.111908] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2760.112034] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2760.112130] ll_md_close+0x274/0x800 [lustre] [ 2760.112219] ll_file_release+0x128/0xc50 [lustre] [ 2760.112873] __fput+0x10d/0x440 [ 2760.112882] ____fput+0x12/0x20 [ 2760.112886] task_work_run+0xc6/0x110 [ 2760.112894] exit_to_usermode_loop+0x1dd/0x1f0 [ 2760.112901] do_syscall_64+0x3d6/0x440 [ 2760.112906] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2760.112914] RIP: 0033:0x7ff54af45cd8 [ 2760.112922] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2760.112926] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2760.112931] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 2760.112933] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 2760.112935] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 2760.112937] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 2760.112939] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2760.575092] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2760.584132] after_reply+0xe2c/0x1780 [ptlrpc] [ 2760.587440] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2760.595107] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2760.597226] ? do_wait_intr+0xf0/0xf0 [ 2760.599897] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2760.601886] kthread+0x1d1/0x200 [ 2760.604200] ? set_kthread_struct+0x70/0x70 [ 2760.607143] ret_from_fork+0x1f/0x30 [ 2761.183117] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 802s! [ 2761.186183] Showing busy workqueues and worker pools: [ 2761.189190] workqueue events: flags=0x0 [ 2761.191722] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2761.196105] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 2761.201646] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 2761.206190] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 2761.216254] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2761.219049] pending: vmstat_shepherd [ 2761.221149] workqueue events_long: flags=0x0 [ 2761.223237] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2761.227071] pending: lu_objects_flush [obdclass] [ 2761.231731] workqueue events_unbound: flags=0x2 [ 2761.242171] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 2761.249933] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2761.257714] workqueue events_power_efficient: flags=0x80 [ 2761.260922] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2761.265941] pending: neigh_periodic_work [ 2761.268881] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2761.273723] pending: do_cache_clean, check_lifetime [ 2761.277170] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2761.283140] pending: neigh_periodic_work [ 2761.286223] workqueue writeback: flags=0x4a [ 2761.288832] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2761.294525] pending: wb_workfn, wb_workfn [ 2761.296108] workqueue xprtiod: flags=0xa [ 2761.302241] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 2761.312979] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 2761.325530] workqueue ipv6_addrconf: flags=0x40008 [ 2761.339226] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2761.344142] pending: addrconf_verify_work [ 2761.346856] workqueue knbd0-recv: flags=0x1a [ 2761.349934] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2761.353724] in-flight: 657:recv_work [ 2761.361195] workqueue ptlrpc_pinger: flags=0x2 [ 2761.368966] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2761.376866] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 2761.384896] pool 8: cpus=0-3 flags=0x4 nice=0 hung=802s workers=4 idle: 20831 12642 28859 [ 2761.391180] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2751s workers=2 idle: 659 [ 2764.060293] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [ 2764.062874] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 2764.064573] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2764.064662] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2764.064669] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2764.064672] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2764.064687] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2764.064691] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2764.064696] RAX: 0000000000000823 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2764.064698] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 2764.064701] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2764.064703] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 2764.064705] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 2764.064708] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 2764.064711] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2764.064713] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 2764.064721] Call Trace: [ 2764.064726] [ 2764.064732] ? show_regs.cold.9+0x22/0x2f [ 2764.064742] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2764.064748] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2764.064752] ? watchdog+0x40/0x40 [ 2764.064757] ? hrtimer_interrupt+0x128/0x2c0 [ 2764.064760] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2764.064822] ? apic_timer_interrupt+0xf/0x20 [ 2764.064827] [ 2764.064830] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2764.064836] do_raw_spin_lock+0x177/0x190 [ 2764.064841] _raw_spin_lock+0x21/0x30 [ 2764.064846] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2764.065450] ? do_raw_spin_unlock+0x75/0x190 [ 2764.065456] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2764.065672] ? do_wait_intr+0xf0/0xf0 [ 2764.065680] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2764.065896] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2764.066390] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2764.066680] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2764.066794] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2764.066831] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2764.066859] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2764.066888] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2764.067520] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2764.067797] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2764.067832] lmv_intent_open+0x338/0x1150 [lmv] [ 2764.067864] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2764.068324] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2764.068414] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2764.068438] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2764.068529] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2764.068612] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2764.068704] ? do_raw_spin_unlock+0x75/0x190 [ 2764.068712] ? _raw_spin_unlock+0x12/0x30 [ 2764.068719] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2764.068746] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2764.078125] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2764.078200] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2764.078207] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2764.078214] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2764.080877] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2764.080894] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2764.080898] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2764.080903] RAX: 0000000000007113 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2764.080905] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 2764.080907] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 2764.080909] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 2764.080911] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 2764.080914] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 2764.080918] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2764.080920] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 2764.080928] Call Trace: [ 2764.080934] [ 2764.080940] ? show_regs.cold.9+0x22/0x2f [ 2764.080950] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2764.080957] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2764.080962] ? watchdog+0x40/0x40 [ 2764.080966] ? hrtimer_interrupt+0x128/0x2c0 [ 2764.080969] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2764.080978] ? apic_timer_interrupt+0xf/0x20 [ 2764.080982] [ 2764.080984] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2764.080988] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2764.080994] do_raw_spin_lock+0x177/0x190 [ 2764.080999] _raw_spin_lock+0x21/0x30 [ 2764.081003] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2764.130165] ll_atomic_open+0x36a/0xd70 [lustre] [ 2764.130303] lookup_open+0x356/0xb40 [ 2764.130314] do_last+0x313/0xfc0 [ 2764.130319] ? nd_jump_root+0xe5/0x160 [ 2764.130324] ? path_init+0x437/0x520 [ 2764.130329] path_openat+0xf7/0x500 [ 2764.130334] ? do_raw_spin_unlock+0x75/0x190 [ 2764.130341] do_filp_open+0x99/0x140 [ 2764.130348] ? getname_flags+0x6e/0x330 [ 2764.130352] ? __check_object_size+0xff/0x256 [ 2764.130359] ? do_raw_spin_unlock+0x75/0x190 [ 2764.130363] ? _raw_spin_unlock+0x12/0x30 [ 2764.130368] do_sys_openat2+0x2b4/0x410 [ 2764.130373] do_sys_open+0x73/0xa0 [ 2764.130377] __x64_sys_openat+0x24/0x30 [ 2764.130380] do_syscall_64+0xc1/0x440 [ 2764.130386] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2764.130391] RIP: 0033:0x7ff54af45332 [ 2764.130398] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2764.130401] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2764.130407] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 2764.130409] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 2764.130412] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 2764.130414] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 2764.130416] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 2764.801764] process_one_work+0x2c8/0x700 [ 2764.804552] worker_thread+0x296/0x6e0 [ 2764.807570] ? rescuer_thread+0x570/0x570 [ 2764.809874] kthread+0x1d1/0x200 [ 2764.811945] ? set_kthread_struct+0x70/0x70 [ 2764.813388] ret_from_fork+0x1f/0x30 [ 2788.055052] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 2788.058810] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 2788.059181] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2788.059559] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2788.059566] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2788.059569] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2788.059595] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2788.059600] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2788.059604] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2788.059606] RDX: 0000000000007518 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 2788.059608] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 2788.059610] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 2788.059612] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2788.059615] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 2788.059619] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2788.059621] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 2788.059630] Call Trace: [ 2788.059639] [ 2788.059647] ? show_regs.cold.9+0x22/0x2f [ 2788.059657] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2788.059663] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2788.059668] ? watchdog+0x40/0x40 [ 2788.059673] ? hrtimer_interrupt+0x128/0x2c0 [ 2788.059676] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2788.059688] ? apic_timer_interrupt+0xf/0x20 [ 2788.059692] [ 2788.059695] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2788.059699] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2788.059704] do_raw_spin_lock+0x177/0x190 [ 2788.059708] _raw_spin_lock+0x21/0x30 [ 2788.059713] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2788.060071] obd_mod_free+0x46/0x140 [ptlrpc] [ 2788.060283] mdc_commit_open+0x91/0xb0 [mdc] [ 2788.060321] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2788.060524] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2788.060718] after_reply+0xe2c/0x1780 [ptlrpc] [ 2788.060912] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2788.061532] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2788.061803] ? do_wait_intr+0xf0/0xf0 [ 2788.061815] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2788.070441] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) [ 2788.094116] kthread+0x1d1/0x200 [ 2788.094141] ? set_kthread_struct+0x70/0x70 [ 2788.094147] ret_from_fork+0x1f/0x30 [ 2788.317377] mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2788.345399] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2788.357653] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2788.365581] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2788.369647] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2788.383592] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2788.387584] RAX: 0000000000007d24 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2788.390764] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 2788.394330] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2788.398055] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 2788.401549] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 2788.405086] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 2788.409038] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2788.411360] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 2788.414469] Call Trace: [ 2788.415711] [ 2788.416764] ? show_regs.cold.9+0x22/0x2f [ 2788.418843] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2788.421280] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2788.423395] ? watchdog+0x40/0x40 [ 2788.424726] ? hrtimer_interrupt+0x128/0x2c0 [ 2788.429782] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2788.431876] ? apic_timer_interrupt+0xf/0x20 [ 2788.439024] [ 2788.440540] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2788.443672] do_raw_spin_lock+0x177/0x190 [ 2788.446887] _raw_spin_lock+0x21/0x30 [ 2788.449412] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2788.453147] ? do_raw_spin_unlock+0x75/0x190 [ 2788.456031] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2788.459356] ? do_wait_intr+0xf0/0xf0 [ 2788.461625] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2788.467037] mdc_close+0x53b/0xd70 [mdc] [ 2788.470367] lmv_close+0x18d/0x490 [lmv] [ 2788.472763] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 2788.478649] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2788.484650] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2788.487856] ll_md_close+0x274/0x800 [lustre] [ 2788.491676] ll_file_release+0x128/0xc50 [lustre] [ 2788.494763] __fput+0x10d/0x440 [ 2788.496787] ____fput+0x12/0x20 [ 2788.500301] task_work_run+0xc6/0x110 [ 2788.501722] exit_to_usermode_loop+0x1dd/0x1f0 [ 2788.509605] do_syscall_64+0x3d6/0x440 [ 2788.515877] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2788.519868] RIP: 0033:0x7ff54af45cd8 [ 2788.521144] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2788.537304] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2788.542072] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 2788.545788] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 2788.550445] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 2788.555516] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 2788.559704] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2791.903882] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 833s! [ 2791.912123] Showing busy workqueues and worker pools: [ 2791.917417] workqueue events: flags=0x0 [ 2791.921650] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2791.928720] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 2791.938585] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 2791.943729] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 2791.953274] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2791.957923] pending: vmstat_shepherd [ 2791.961038] workqueue events_long: flags=0x0 [ 2791.963238] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2791.966739] pending: lu_objects_flush [obdclass] [ 2791.973077] workqueue events_unbound: flags=0x2 [ 2791.976516] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 2791.980579] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2791.987905] workqueue events_power_efficient: flags=0x80 [ 2791.991389] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2791.995313] pending: neigh_periodic_work [ 2791.998099] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2792.002309] pending: do_cache_clean, check_lifetime [ 2792.005716] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2792.010588] pending: neigh_periodic_work [ 2792.013891] workqueue writeback: flags=0x4a [ 2792.016700] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2792.020713] pending: wb_workfn, wb_workfn [ 2792.024330] workqueue xprtiod: flags=0xa [ 2792.026628] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 2792.031102] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 2792.036078] workqueue ipv6_addrconf: flags=0x40008 [ 2792.038711] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2792.042447] pending: addrconf_verify_work [ 2792.045580] workqueue knbd0-recv: flags=0x1a [ 2792.049084] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2792.053248] in-flight: 657:recv_work [ 2792.055664] workqueue ptlrpc_pinger: flags=0x2 [ 2792.058154] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2792.063288] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 2792.063905] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 2792.063959] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [ 2792.063975] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2792.064043] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2792.064049] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2792.064051] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2792.064063] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2792.064066] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2792.064070] RAX: 0000000000007d0b RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2792.064073] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 2792.064075] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2792.064076] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 2792.064079] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 2792.064083] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 2792.064085] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2792.064087] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 2792.064096] Call Trace: [ 2792.064103] [ 2792.064109] ? show_regs.cold.9+0x22/0x2f [ 2792.064120] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2792.064126] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2792.064131] ? watchdog+0x40/0x40 [ 2792.064134] ? hrtimer_interrupt+0x128/0x2c0 [ 2792.065191] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2792.065206] ? apic_timer_interrupt+0xf/0x20 [ 2792.065211] [ 2792.065214] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2792.065222] do_raw_spin_lock+0x177/0x190 [ 2792.065228] _raw_spin_lock+0x21/0x30 [ 2792.065233] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2792.065704] ? do_raw_spin_unlock+0x75/0x190 [ 2792.065711] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2792.065905] ? do_wait_intr+0xf0/0xf0 [ 2792.065914] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2792.066107] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2792.066694] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2792.066897] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2792.067033] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2792.067073] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2792.067098] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2792.067668] pool 8: cpus=0-3 flags=0x4 nice=0 hung=833s workers=4 idle: 20831 12642 28859 [ 2792.067689] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2782s workers=2 idle: 659 [ 2792.067121] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2792.072438] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2792.072630] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2792.072638] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2792.072647] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2792.073233] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2792.073246] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2792.073251] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 2792.073256] RAX: 0000000000005990 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2792.073259] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 2792.073261] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 2792.073264] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 2792.073266] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 2792.073269] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 2792.073272] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2792.073274] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 2792.073284] Call Trace: [ 2792.073292] [ 2792.073299] ? show_regs.cold.9+0x22/0x2f [ 2792.073310] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2792.073318] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2792.073324] ? watchdog+0x40/0x40 [ 2792.073336] ? hrtimer_interrupt+0x128/0x2c0 [ 2792.073341] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2792.073360] ? apic_timer_interrupt+0xf/0x20 [ 2792.073365] [ 2792.073369] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2792.073374] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2792.073380] do_raw_spin_lock+0x177/0x190 [ 2792.073384] _raw_spin_lock+0x21/0x30 [ 2792.073389] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2792.078729] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2792.079392] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2792.079437] lmv_intent_open+0x338/0x1150 [lmv] [ 2792.079525] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2792.079645] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2792.079730] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2792.079751] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2792.080112] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2792.080324] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2792.080426] ? do_raw_spin_unlock+0x75/0x190 [ 2792.080436] ? _raw_spin_unlock+0x12/0x30 [ 2792.080444] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2792.080481] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2792.080501] ll_atomic_open+0x36a/0xd70 [lustre] [ 2792.080587] lookup_open+0x356/0xb40 [ 2792.080597] do_last+0x313/0xfc0 [ 2792.080603] ? nd_jump_root+0xe5/0x160 [ 2792.080608] ? path_init+0x437/0x520 [ 2792.080612] path_openat+0xf7/0x500 [ 2792.080618] ? do_raw_spin_unlock+0x75/0x190 [ 2792.080623] do_filp_open+0x99/0x140 [ 2792.080629] ? getname_flags+0x6e/0x330 [ 2792.080634] ? __check_object_size+0xff/0x256 [ 2792.080641] ? do_raw_spin_unlock+0x75/0x190 [ 2792.080645] ? _raw_spin_unlock+0x12/0x30 [ 2792.080649] do_sys_openat2+0x2b4/0x410 [ 2792.080653] do_sys_open+0x73/0xa0 [ 2792.080657] __x64_sys_openat+0x24/0x30 [ 2792.080661] do_syscall_64+0xc1/0x440 [ 2792.080668] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2792.080675] RIP: 0033:0x7ff54af45332 [ 2792.080683] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2792.080686] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2792.080691] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 2792.080694] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 2792.080696] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 2792.080698] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 2792.080701] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 2792.529964] process_one_work+0x2c8/0x700 [ 2792.531897] worker_thread+0x296/0x6e0 [ 2792.534241] ? rescuer_thread+0x570/0x570 [ 2792.536328] kthread+0x1d1/0x200 [ 2792.537891] ? set_kthread_struct+0x70/0x70 [ 2792.540303] ret_from_fork+0x1f/0x30 [ 2816.062283] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 2816.069890] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2816.121458] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2816.133486] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2816.142599] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2816.146853] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2816.165568] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2816.169738] RAX: 0000000000001706 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2816.173799] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 2816.179344] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2816.184262] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 2816.188558] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 2816.192466] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 2816.198376] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2816.203937] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 2816.211671] Call Trace: [ 2816.213422] [ 2816.216455] ? show_regs.cold.9+0x22/0x2f [ 2816.219620] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2816.225955] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2816.229151] ? watchdog+0x40/0x40 [ 2816.231076] ? hrtimer_interrupt+0x128/0x2c0 [ 2816.233240] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2816.240369] ? apic_timer_interrupt+0xf/0x20 [ 2816.242667] [ 2816.246630] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2816.251492] do_raw_spin_lock+0x177/0x190 [ 2816.260442] _raw_spin_lock+0x21/0x30 [ 2816.263897] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2816.269733] ? do_raw_spin_unlock+0x75/0x190 [ 2816.278421] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2816.286333] ? do_wait_intr+0xf0/0xf0 [ 2816.294553] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2816.300909] mdc_close+0x53b/0xd70 [mdc] [ 2816.302286] lmv_close+0x18d/0x490 [lmv] [ 2816.303608] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 2816.312591] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2816.320348] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2816.328733] ll_md_close+0x274/0x800 [lustre] [ 2816.331411] ll_file_release+0x128/0xc50 [lustre] [ 2816.335300] __fput+0x10d/0x440 [ 2816.336794] ____fput+0x12/0x20 [ 2816.339325] task_work_run+0xc6/0x110 [ 2816.341297] exit_to_usermode_loop+0x1dd/0x1f0 [ 2816.343937] do_syscall_64+0x3d6/0x440 [ 2816.345526] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2816.348496] RIP: 0033:0x7ff54af45cd8 [ 2816.350531] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2816.367832] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2816.374478] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 2816.382510] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 2816.388445] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 2816.394567] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 2816.402661] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2820.055399] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 2820.059178] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 2820.060091] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 2820.060101] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2820.060178] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2820.060183] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2820.060187] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2820.060203] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2820.060207] RSP: 0000:ffffaf9fc15234f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2820.060212] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2820.060214] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 2820.060216] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2820.060219] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 2820.060221] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 2820.060225] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 2820.060227] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2820.060229] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 2820.060239] Call Trace: [ 2820.060247] [ 2820.060253] ? show_regs.cold.9+0x22/0x2f [ 2820.060266] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2820.060274] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2820.060279] ? watchdog+0x40/0x40 [ 2820.060282] ? hrtimer_interrupt+0x128/0x2c0 [ 2820.060286] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2820.060293] ? apic_timer_interrupt+0xf/0x20 [ 2820.060297] [ 2820.060300] ? native_safe_halt+0x2e/0x50 [ 2820.060304] kvm_wait+0xcd/0xe0 [ 2820.060311] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2820.060320] do_raw_spin_lock+0x177/0x190 [ 2820.060325] _raw_spin_lock+0x21/0x30 [ 2820.060330] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2820.060660] ? do_raw_spin_unlock+0x75/0x190 [ 2820.060665] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2820.060876] ? do_wait_intr+0xf0/0xf0 [ 2820.060889] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2820.061250] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2820.061467] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2820.061672] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2820.061881] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2820.061924] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2820.061954] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2820.061979] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2820.062130] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2820.062358] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2820.062389] lmv_intent_open+0x338/0x1150 [lmv] [ 2820.062424] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2820.062509] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2820.062593] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2820.062615] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2820.062698] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2820.062925] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2820.063071] ? do_raw_spin_unlock+0x75/0x190 [ 2820.063088] ? _raw_spin_unlock+0x12/0x30 [ 2820.063096] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2820.063122] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2820.063141] ll_atomic_open+0x36a/0xd70 [lustre] [ 2820.063222] lookup_open+0x356/0xb40 [ 2820.063231] do_last+0x313/0xfc0 [ 2820.063236] ? nd_jump_root+0xe5/0x160 [ 2820.063240] ? path_init+0x437/0x520 [ 2820.063245] path_openat+0xf7/0x500 [ 2820.063250] ? do_raw_spin_unlock+0x75/0x190 [ 2820.063255] do_filp_open+0x99/0x140 [ 2820.063261] ? getname_flags+0x6e/0x330 [ 2820.063266] ? __check_object_size+0xff/0x256 [ 2820.063273] ? do_raw_spin_unlock+0x75/0x190 [ 2820.063278] ? _raw_spin_unlock+0x12/0x30 [ 2820.063282] do_sys_openat2+0x2b4/0x410 [ 2820.063287] do_sys_open+0x73/0xa0 [ 2820.063290] __x64_sys_openat+0x24/0x30 [ 2820.063294] do_syscall_64+0xc1/0x440 [ 2820.063300] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2820.063306] RIP: 0033:0x7ff54af45332 [ 2820.063313] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2820.063318] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2820.063323] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 2820.063326] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 2820.063327] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 2820.063329] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 2820.063331] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 2820.065533] Modules linked in: [ 2820.078966] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2820.079052] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2820.079059] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2820.079066] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2820.082326] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2820.082411] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2820.082419] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2820.082423] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2820.082441] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2820.082445] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2820.082449] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2820.082452] RDX: 0000000000006068 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 2820.082454] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 2820.082456] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 2820.082458] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2820.082463] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 2820.082465] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2820.082468] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 2820.082478] Call Trace: [ 2820.082487] [ 2820.082497] ? show_regs.cold.9+0x22/0x2f [ 2820.082512] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2820.082520] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2820.082524] ? watchdog+0x40/0x40 [ 2820.082528] ? hrtimer_interrupt+0x128/0x2c0 [ 2820.082532] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2820.082542] ? apic_timer_interrupt+0xf/0x20 [ 2820.082546] [ 2820.082549] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2820.082554] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2820.082558] do_raw_spin_lock+0x177/0x190 [ 2820.082563] _raw_spin_lock+0x21/0x30 [ 2820.082569] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2820.082957] obd_mod_free+0x46/0x140 [ptlrpc] [ 2820.083509] mdc_commit_open+0x91/0xb0 [mdc] [ 2820.124139] [ 2820.124154] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2820.124180] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2820.124184] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2820.124189] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2820.124191] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 2820.124194] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 2820.124196] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 2820.124199] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 2820.124203] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 2820.124206] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2820.124208] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 2820.124218] Call Trace: [ 2820.124225] [ 2820.124231] ? show_regs.cold.9+0x22/0x2f [ 2820.124246] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2820.124254] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2820.124260] ? watchdog+0x40/0x40 [ 2820.124264] ? hrtimer_interrupt+0x128/0x2c0 [ 2820.124268] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2820.124276] ? apic_timer_interrupt+0xf/0x20 [ 2820.124281] [ 2820.124284] ? native_safe_halt+0x2e/0x50 [ 2820.124288] kvm_wait+0xcd/0xe0 [ 2820.124296] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2820.124308] do_raw_spin_lock+0x177/0x190 [ 2820.124312] _raw_spin_lock+0x21/0x30 [ 2820.124317] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2820.124672] process_one_work+0x2c8/0x700 [ 2820.124682] worker_thread+0x296/0x6e0 [ 2820.124686] ? rescuer_thread+0x570/0x570 [ 2820.124690] kthread+0x1d1/0x200 [ 2820.124696] ? set_kthread_struct+0x70/0x70 [ 2820.124700] ret_from_fork+0x1f/0x30 [ 2820.733162] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2820.736374] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2820.739686] after_reply+0xe2c/0x1780 [ptlrpc] [ 2820.742680] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2820.746005] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2820.748409] ? do_wait_intr+0xf0/0xf0 [ 2820.749778] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2820.752496] kthread+0x1d1/0x200 [ 2820.754346] ? set_kthread_struct+0x70/0x70 [ 2820.756871] ret_from_fork+0x1f/0x30 [ 2822.111286] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 863s! [ 2822.117825] Showing busy workqueues and worker pools: [ 2822.121642] workqueue events: flags=0x0 [ 2822.124443] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2822.132474] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 2822.143621] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 2822.151720] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 2822.163575] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2822.172956] pending: vmstat_shepherd [ 2822.181347] workqueue events_long: flags=0x0 [ 2822.183474] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2822.195873] pending: lu_objects_flush [obdclass] [ 2822.206302] workqueue events_unbound: flags=0x2 [ 2822.207765] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 2822.214644] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2822.225878] workqueue events_power_efficient: flags=0x80 [ 2822.235704] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2822.244692] pending: neigh_periodic_work [ 2822.246111] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2822.257445] pending: do_cache_clean, check_lifetime [ 2822.263946] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2822.278755] pending: neigh_periodic_work [ 2822.282698] workqueue writeback: flags=0x4a [ 2822.286776] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2822.293804] pending: wb_workfn, wb_workfn [ 2822.298529] workqueue xprtiod: flags=0xa [ 2822.302113] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 2822.304210] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 2822.314247] workqueue ipv6_addrconf: flags=0x40008 [ 2822.320452] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2822.323454] pending: addrconf_verify_work [ 2822.325361] workqueue knbd0-recv: flags=0x1a [ 2822.327435] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2822.330431] in-flight: 657:recv_work [ 2822.332386] workqueue ptlrpc_pinger: flags=0x2 [ 2822.334396] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2822.336577] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 2822.339344] pool 8: cpus=0-3 flags=0x4 nice=0 hung=863s workers=4 idle: 20831 12642 28859 [ 2822.343593] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2812s workers=2 idle: 659 [ 2844.060059] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 2844.074210] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2844.114291] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2844.124463] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2844.129761] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2844.135518] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2844.153810] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2844.164145] RAX: 0000000000006f8b RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2844.168542] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 2844.177516] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2844.186048] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 2844.193451] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 2844.197979] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 2844.205448] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2844.209807] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 2844.217047] Call Trace: [ 2844.217803] [ 2844.218475] ? show_regs.cold.9+0x22/0x2f [ 2844.225279] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2844.230076] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2844.233516] ? watchdog+0x40/0x40 [ 2844.235368] ? hrtimer_interrupt+0x128/0x2c0 [ 2844.243135] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2844.248391] ? apic_timer_interrupt+0xf/0x20 [ 2844.253063] [ 2844.256110] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2844.257919] do_raw_spin_lock+0x177/0x190 [ 2844.266635] _raw_spin_lock+0x21/0x30 [ 2844.268299] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2844.272748] ? do_raw_spin_unlock+0x75/0x190 [ 2844.276259] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2844.281547] ? do_wait_intr+0xf0/0xf0 [ 2844.285492] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2844.292079] mdc_close+0x53b/0xd70 [mdc] [ 2844.298452] lmv_close+0x18d/0x490 [lmv] [ 2844.300352] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 2844.306859] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2844.313366] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2844.317361] ll_md_close+0x274/0x800 [lustre] [ 2844.321460] ll_file_release+0x128/0xc50 [lustre] [ 2844.323630] __fput+0x10d/0x440 [ 2844.324774] ____fput+0x12/0x20 [ 2844.331292] task_work_run+0xc6/0x110 [ 2844.333205] exit_to_usermode_loop+0x1dd/0x1f0 [ 2844.337779] do_syscall_64+0x3d6/0x440 [ 2844.339538] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2844.346470] RIP: 0033:0x7ff54af45cd8 [ 2844.352036] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2844.373210] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2844.379579] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 2844.382514] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 2844.388522] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 2844.406607] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 2844.418674] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2848.055062] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 2848.058382] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2848.060031] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 2848.060165] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 2848.060172] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2848.060248] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2848.060254] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2848.060261] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2848.060578] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2848.060591] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2848.060596] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2848.060601] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2848.060603] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 2848.060605] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 2848.060608] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 2848.060611] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 2848.060615] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 2848.060617] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2848.060620] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 2848.060630] Call Trace: [ 2848.060636] [ 2848.060641] ? show_regs.cold.9+0x22/0x2f [ 2848.060652] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2848.060675] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2848.060680] ? watchdog+0x40/0x40 [ 2848.060684] ? hrtimer_interrupt+0x128/0x2c0 [ 2848.060689] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2848.060698] ? apic_timer_interrupt+0xf/0x20 [ 2848.060703] [ 2848.060706] ? native_safe_halt+0x2e/0x50 [ 2848.060711] kvm_wait+0xcd/0xe0 [ 2848.060720] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2848.060729] do_raw_spin_lock+0x177/0x190 [ 2848.060733] _raw_spin_lock+0x21/0x30 [ 2848.060737] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2848.060964] process_one_work+0x2c8/0x700 [ 2848.060971] worker_thread+0x296/0x6e0 [ 2848.060976] ? rescuer_thread+0x570/0x570 [ 2848.060980] kthread+0x1d1/0x200 [ 2848.060984] ? set_kthread_struct+0x70/0x70 [ 2848.060988] ret_from_fork+0x1f/0x30 [ 2848.092353] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2848.092369] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2848.092372] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2848.092388] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2848.092393] RSP: 0000:ffffaf9fc1b97bb8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 2848.092400] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2848.092402] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 2848.092404] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 2848.092405] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 2848.092407] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2848.092410] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 2848.092413] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2848.092490] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 2848.092498] Call Trace: [ 2848.092504] [ 2848.092510] ? show_regs.cold.9+0x22/0x2f [ 2848.092522] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2848.092528] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2848.092532] ? watchdog+0x40/0x40 [ 2848.092535] ? hrtimer_interrupt+0x128/0x2c0 [ 2848.092539] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2848.092546] ? apic_timer_interrupt+0xf/0x20 [ 2848.092550] [ 2848.092553] ? native_safe_halt+0x2e/0x50 [ 2848.092557] kvm_wait+0xcd/0xe0 [ 2848.092565] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2848.092573] do_raw_spin_lock+0x177/0x190 [ 2848.092578] _raw_spin_lock+0x21/0x30 [ 2848.092584] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2848.096544] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2848.096617] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2848.096623] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2848.096625] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2848.096701] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2848.096705] RSP: 0000:ffffaf9fc15234f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2848.096710] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2848.096712] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 2848.096714] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2848.096716] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 2848.096718] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 2848.096721] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 2848.096724] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2848.096726] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 2848.096734] Call Trace: [ 2848.096739] [ 2848.096744] ? show_regs.cold.9+0x22/0x2f [ 2848.096754] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2848.096760] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2848.096764] ? watchdog+0x40/0x40 [ 2848.096769] ? hrtimer_interrupt+0x128/0x2c0 [ 2848.096772] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2848.096778] ? apic_timer_interrupt+0xf/0x20 [ 2848.096784] [ 2848.096787] ? native_safe_halt+0x2e/0x50 [ 2848.096791] kvm_wait+0xcd/0xe0 [ 2848.096799] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2848.096919] do_raw_spin_lock+0x177/0x190 [ 2848.096928] _raw_spin_lock+0x21/0x30 [ 2848.096932] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2848.097202] ? do_raw_spin_unlock+0x75/0x190 [ 2848.097209] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2848.097408] ? do_wait_intr+0xf0/0xf0 [ 2848.097427] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2848.097628] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2848.097856] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2848.098066] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2848.098193] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2848.098233] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2848.098262] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2848.098290] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2848.098374] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2848.098586] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2848.098614] lmv_intent_open+0x338/0x1150 [lmv] [ 2848.098646] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2848.098738] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2848.099110] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2848.099134] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2848.099220] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2848.099301] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2848.099381] ? do_raw_spin_unlock+0x75/0x190 [ 2848.099390] ? _raw_spin_unlock+0x12/0x30 [ 2848.099396] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2848.099488] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2848.099508] ll_atomic_open+0x36a/0xd70 [lustre] [ 2848.099591] lookup_open+0x356/0xb40 [ 2848.099600] do_last+0x313/0xfc0 [ 2848.099605] ? nd_jump_root+0xe5/0x160 [ 2848.099609] ? path_init+0x437/0x520 [ 2848.099613] path_openat+0xf7/0x500 [ 2848.099619] ? do_raw_spin_unlock+0x75/0x190 [ 2848.099623] do_filp_open+0x99/0x140 [ 2848.099629] ? getname_flags+0x6e/0x330 [ 2848.103464] obd_mod_free+0x46/0x140 [ptlrpc] [ 2848.130144] ? __check_object_size+0xff/0x256 [ 2848.130168] ? do_raw_spin_unlock+0x75/0x190 [ 2848.130175] ? _raw_spin_unlock+0x12/0x30 [ 2848.130183] do_sys_openat2+0x2b4/0x410 [ 2848.130190] do_sys_open+0x73/0xa0 [ 2848.130195] __x64_sys_openat+0x24/0x30 [ 2848.130198] do_syscall_64+0xc1/0x440 [ 2848.130205] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2848.130211] RIP: 0033:0x7ff54af45332 [ 2848.130220] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2848.130224] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2848.130230] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 2848.130232] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 2848.130235] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 2848.130237] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 2848.130239] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 2848.648063] mdc_commit_open+0x91/0xb0 [mdc] [ 2848.649691] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2848.652773] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2848.655110] after_reply+0xe2c/0x1780 [ptlrpc] [ 2848.657876] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2848.661089] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2848.663175] ? do_wait_intr+0xf0/0xf0 [ 2848.664550] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2848.666815] kthread+0x1d1/0x200 [ 2848.667750] ? set_kthread_struct+0x70/0x70 [ 2848.669730] ret_from_fork+0x1f/0x30 [ 2852.832228] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 893s! [ 2852.836236] Showing busy workqueues and worker pools: [ 2852.844307] workqueue events: flags=0x0 [ 2852.851800] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2852.866336] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 2852.885989] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 2852.902016] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 2852.912178] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2852.923161] pending: vmstat_shepherd [ 2852.926651] workqueue events_long: flags=0x0 [ 2852.929015] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2852.937454] pending: lu_objects_flush [obdclass] [ 2852.939489] workqueue events_unbound: flags=0x2 [ 2852.941790] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 2852.948209] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2852.952168] workqueue events_power_efficient: flags=0x80 [ 2852.957067] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2852.966115] pending: neigh_periodic_work [ 2852.970644] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2852.974985] pending: do_cache_clean, check_lifetime [ 2852.976432] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2852.979088] pending: neigh_periodic_work [ 2852.984987] workqueue writeback: flags=0x4a [ 2852.986996] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2852.989627] pending: wb_workfn, wb_workfn [ 2852.990884] workqueue xprtiod: flags=0xa [ 2852.993784] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 2852.998957] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 2853.003131] workqueue ipv6_addrconf: flags=0x40008 [ 2853.005893] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2853.010864] pending: addrconf_verify_work [ 2853.016131] workqueue knbd0-recv: flags=0x1a [ 2853.022806] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2853.026875] in-flight: 657:recv_work [ 2853.028302] workqueue ptlrpc_pinger: flags=0x2 [ 2853.034922] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2853.038096] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 2853.046791] pool 8: cpus=0-3 flags=0x4 nice=0 hung=894s workers=4 idle: 20831 12642 28859 [ 2853.055705] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2843s workers=2 idle: 659 [ 2872.059724] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 2872.070522] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2872.193799] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2872.214524] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2872.229761] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2872.241126] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2872.304455] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2872.326740] RAX: 0000000000007882 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2872.355610] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 2872.370894] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2872.380644] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 2872.398582] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 2872.408702] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 2872.418746] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2872.427782] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 2872.440582] Call Trace: [ 2872.443729] [ 2872.444734] ? show_regs.cold.9+0x22/0x2f [ 2872.449176] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2872.456552] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2872.461536] ? watchdog+0x40/0x40 [ 2872.465758] ? hrtimer_interrupt+0x128/0x2c0 [ 2872.473626] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2872.475815] ? apic_timer_interrupt+0xf/0x20 [ 2872.482814] [ 2872.485893] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2872.490933] do_raw_spin_lock+0x177/0x190 [ 2872.494936] _raw_spin_lock+0x21/0x30 [ 2872.498599] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2872.507719] ? do_raw_spin_unlock+0x75/0x190 [ 2872.517274] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2872.534550] ? do_wait_intr+0xf0/0xf0 [ 2872.537204] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2872.547916] mdc_close+0x53b/0xd70 [mdc] [ 2872.555323] lmv_close+0x18d/0x490 [lmv] [ 2872.569805] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 2872.586860] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2872.588260] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2872.599530] ll_md_close+0x274/0x800 [lustre] [ 2872.618322] ll_file_release+0x128/0xc50 [lustre] [ 2872.632788] __fput+0x10d/0x440 [ 2872.638682] ____fput+0x12/0x20 [ 2872.639857] task_work_run+0xc6/0x110 [ 2872.648652] exit_to_usermode_loop+0x1dd/0x1f0 [ 2872.656670] do_syscall_64+0x3d6/0x440 [ 2872.669660] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2872.679260] RIP: 0033:0x7ff54af45cd8 [ 2872.680635] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2872.714923] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2872.727265] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 2872.745850] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 2872.757723] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 2872.779636] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 2872.796500] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2876.060104] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 2876.060689] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 2876.087122] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2876.087278] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2876.087286] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2876.087293] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2876.087624] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2876.087637] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2876.087641] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2876.087646] RAX: 0000000000007e83 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2876.087648] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 2876.087650] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 2876.087653] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 2876.087655] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 2876.087658] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 2876.087661] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2876.087663] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 2876.087671] Call Trace: [ 2876.087678] [ 2876.087687] ? show_regs.cold.9+0x22/0x2f [ 2876.087699] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2876.087706] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2876.087711] ? watchdog+0x40/0x40 [ 2876.087715] ? hrtimer_interrupt+0x128/0x2c0 [ 2876.087719] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2876.087728] ? apic_timer_interrupt+0xf/0x20 [ 2876.087732] [ 2876.087734] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2876.087739] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2876.087744] do_raw_spin_lock+0x177/0x190 [ 2876.087748] _raw_spin_lock+0x21/0x30 [ 2876.087753] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2876.087971] process_one_work+0x2c8/0x700 [ 2876.087978] worker_thread+0x296/0x6e0 [ 2876.087982] ? rescuer_thread+0x570/0x570 [ 2876.087986] kthread+0x1d1/0x200 [ 2876.087990] ? set_kthread_struct+0x70/0x70 [ 2876.087994] ret_from_fork+0x1f/0x30 [ 2876.491575] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2876.593344] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2876.607620] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2876.615264] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2876.622825] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2876.644618] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2876.655930] RAX: 0000000000007f0e RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2876.670810] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 2876.681929] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2876.696208] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 2876.699397] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 2876.714939] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 2876.741216] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2876.747320] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 2876.766886] Call Trace: [ 2876.768752] [ 2876.777376] ? show_regs.cold.9+0x22/0x2f [ 2876.781222] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2876.790342] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2876.796484] ? watchdog+0x40/0x40 [ 2876.803614] ? hrtimer_interrupt+0x128/0x2c0 [ 2876.809137] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2876.814035] ? apic_timer_interrupt+0xf/0x20 [ 2876.815809] [ 2876.820549] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2876.828875] do_raw_spin_lock+0x177/0x190 [ 2876.831688] _raw_spin_lock+0x21/0x30 [ 2876.838619] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2876.845932] ? do_raw_spin_unlock+0x75/0x190 [ 2876.850943] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2876.854223] ? do_wait_intr+0xf0/0xf0 [ 2876.855877] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2876.859041] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2876.866173] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2876.869186] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2876.871888] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2876.875083] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2876.877853] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2876.880653] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2876.883164] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2876.887863] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2876.891226] lmv_intent_open+0x338/0x1150 [lmv] [ 2876.893305] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2876.897269] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2876.902668] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2876.909158] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2876.913172] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2876.916461] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2876.918467] ? do_raw_spin_unlock+0x75/0x190 [ 2876.922164] ? _raw_spin_unlock+0x12/0x30 [ 2876.928645] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2876.934321] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2876.940826] ll_atomic_open+0x36a/0xd70 [lustre] [ 2876.958070] lookup_open+0x356/0xb40 [ 2876.963630] do_last+0x313/0xfc0 [ 2876.970998] ? nd_jump_root+0xe5/0x160 [ 2876.980937] ? path_init+0x437/0x520 [ 2876.987107] path_openat+0xf7/0x500 [ 2876.988573] ? do_raw_spin_unlock+0x75/0x190 [ 2877.008800] do_filp_open+0x99/0x140 [ 2877.016260] ? getname_flags+0x6e/0x330 [ 2877.026674] ? __check_object_size+0xff/0x256 [ 2877.032681] ? do_raw_spin_unlock+0x75/0x190 [ 2877.042827] ? _raw_spin_unlock+0x12/0x30 [ 2877.051394] do_sys_openat2+0x2b4/0x410 [ 2877.058290] do_sys_open+0x73/0xa0 [ 2877.060719] __x64_sys_openat+0x24/0x30 [ 2877.076167] do_syscall_64+0xc1/0x440 [ 2877.077772] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2877.086506] RIP: 0033:0x7ff54af45332 [ 2877.095508] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2877.124263] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2877.129034] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 2877.134119] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 2877.141172] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 2877.144856] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 2877.154246] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 2880.059456] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [ 2880.062774] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2880.120286] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2880.138834] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2880.150012] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2880.160773] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2880.181461] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2880.189347] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2880.201989] RDX: 0000000000002472 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 2880.214087] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 2880.222670] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 2880.230110] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2880.236555] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 2880.245911] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2880.254048] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 2880.266295] Call Trace: [ 2880.268065] [ 2880.272687] ? show_regs.cold.9+0x22/0x2f [ 2880.284859] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2880.298173] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2880.307277] ? watchdog+0x40/0x40 [ 2880.312589] ? hrtimer_interrupt+0x128/0x2c0 [ 2880.319824] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2880.327319] ? apic_timer_interrupt+0xf/0x20 [ 2880.333817] [ 2880.339409] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2880.344446] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2880.346193] do_raw_spin_lock+0x177/0x190 [ 2880.347488] _raw_spin_lock+0x21/0x30 [ 2880.353312] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2880.372730] obd_mod_free+0x46/0x140 [ptlrpc] [ 2880.379760] mdc_commit_open+0x91/0xb0 [mdc] [ 2880.391780] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2880.403248] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2880.410408] after_reply+0xe2c/0x1780 [ptlrpc] [ 2880.417123] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2880.427253] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2880.429867] ? do_wait_intr+0xf0/0xf0 [ 2880.435721] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2880.443608] kthread+0x1d1/0x200 [ 2880.447589] ? set_kthread_struct+0x70/0x70 [ 2880.450053] ret_from_fork+0x1f/0x30 [ 2883.551654] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 924s! [ 2883.558394] Showing busy workqueues and worker pools: [ 2883.562759] workqueue events: flags=0x0 [ 2883.570023] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2883.582234] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 2883.593355] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 2883.599564] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 2883.609945] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2883.619918] pending: vmstat_shepherd [ 2883.623832] workqueue events_long: flags=0x0 [ 2883.629043] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2883.635253] pending: lu_objects_flush [obdclass] [ 2883.641148] workqueue events_unbound: flags=0x2 [ 2883.646054] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 2883.657365] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2883.668389] workqueue events_power_efficient: flags=0x80 [ 2883.676445] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2883.687644] pending: neigh_periodic_work [ 2883.693897] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2883.707013] pending: do_cache_clean, check_lifetime [ 2883.715262] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2883.731253] pending: neigh_periodic_work [ 2883.745627] workqueue writeback: flags=0x4a [ 2883.754070] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2883.770805] pending: wb_workfn, wb_workfn [ 2883.782435] workqueue xprtiod: flags=0xa [ 2883.792699] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 2883.798384] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 2883.817162] workqueue ipv6_addrconf: flags=0x40008 [ 2883.827297] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2883.858418] pending: addrconf_verify_work [ 2883.867414] workqueue knbd0-recv: flags=0x1a [ 2883.872695] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2883.886143] in-flight: 657:recv_work [ 2883.895593] workqueue ptlrpc_pinger: flags=0x2 [ 2883.904237] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2883.916387] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 2883.928023] pool 8: cpus=0-3 flags=0x4 nice=0 hung=925s workers=4 idle: 20831 12642 28859 [ 2883.942206] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2874s workers=2 idle: 659 [ 2900.058021] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 2900.063318] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2900.117733] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2900.123042] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2900.127020] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2900.130010] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2900.143014] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2900.149103] RAX: 0000000000004110 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2900.153879] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 2900.158466] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2900.164562] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 2900.172248] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 2900.173956] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 2900.188788] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2900.196717] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 2900.200687] Call Trace: [ 2900.206725] [ 2900.215962] ? show_regs.cold.9+0x22/0x2f [ 2900.218072] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2900.224447] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2900.229331] ? watchdog+0x40/0x40 [ 2900.231521] ? hrtimer_interrupt+0x128/0x2c0 [ 2900.234988] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2900.239958] ? apic_timer_interrupt+0xf/0x20 [ 2900.242599] [ 2900.243035] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2900.246987] do_raw_spin_lock+0x177/0x190 [ 2900.251862] _raw_spin_lock+0x21/0x30 [ 2900.254545] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2900.260271] ? do_raw_spin_unlock+0x75/0x190 [ 2900.261938] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2900.271060] ? do_wait_intr+0xf0/0xf0 [ 2900.273221] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2900.278082] mdc_close+0x53b/0xd70 [mdc] [ 2900.279900] lmv_close+0x18d/0x490 [lmv] [ 2900.283580] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 2900.288089] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2900.296917] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2900.301032] ll_md_close+0x274/0x800 [lustre] [ 2900.307894] ll_file_release+0x128/0xc50 [lustre] [ 2900.318754] __fput+0x10d/0x440 [ 2900.320468] ____fput+0x12/0x20 [ 2900.322240] task_work_run+0xc6/0x110 [ 2900.324070] exit_to_usermode_loop+0x1dd/0x1f0 [ 2900.326521] do_syscall_64+0x3d6/0x440 [ 2900.327855] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2900.331139] RIP: 0033:0x7ff54af45cd8 [ 2900.332536] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2900.346884] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2900.354489] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 2900.358764] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 2900.363102] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 2900.367651] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 2900.372864] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2904.059078] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 2904.060756] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 2904.068813] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2904.068905] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2904.068913] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2904.068921] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2904.081050] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2904.081168] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2904.081182] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2904.081185] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2904.081205] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2904.081210] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2904.081214] RAX: 0000000000001451 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2904.081217] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 2904.081219] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2904.081222] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 2904.081224] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 2904.081227] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 2904.081230] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2904.081232] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 2904.081239] Call Trace: [ 2904.081245] [ 2904.081252] ? show_regs.cold.9+0x22/0x2f [ 2904.081262] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2904.081280] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2904.081289] ? watchdog+0x40/0x40 [ 2904.081293] ? hrtimer_interrupt+0x128/0x2c0 [ 2904.081297] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2904.081307] ? apic_timer_interrupt+0xf/0x20 [ 2904.081311] [ 2904.081314] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2904.081321] do_raw_spin_lock+0x177/0x190 [ 2904.081326] _raw_spin_lock+0x21/0x30 [ 2904.081336] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2904.081820] ? do_raw_spin_unlock+0x75/0x190 [ 2904.081826] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2904.153295] [ 2904.153305] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2904.159777] ? do_wait_intr+0xf0/0xf0 [ 2904.174963] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2904.179347] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2904.179648] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2904.179930] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2904.180232] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2904.180478] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2904.180538] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2904.180570] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2904.180599] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2904.180771] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2904.229285] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 [ 2904.235995] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2904.236063] lmv_intent_open+0x338/0x1150 [lmv] [ 2904.236109] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2904.244382] ORIG_RAX: ffffffffffffff13 [ 2904.244394] RAX: 0000000000005a0a RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2904.244397] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 2904.244400] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 2904.244402] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 2904.244405] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 2904.244408] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 2904.244412] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2904.247989] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2904.259624] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 2904.259641] Call Trace: [ 2904.259649] [ 2904.259656] ? show_regs.cold.9+0x22/0x2f [ 2904.264636] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2904.269349] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2904.274108] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2904.277960] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2904.282492] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2904.282623] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2904.282709] ? do_raw_spin_unlock+0x75/0x190 [ 2904.282719] ? _raw_spin_unlock+0x12/0x30 [ 2904.282728] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2904.282758] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2904.282779] ll_atomic_open+0x36a/0xd70 [lustre] [ 2904.282863] lookup_open+0x356/0xb40 [ 2904.282873] do_last+0x313/0xfc0 [ 2904.282877] ? nd_jump_root+0xe5/0x160 [ 2904.282881] ? path_init+0x437/0x520 [ 2904.282885] path_openat+0xf7/0x500 [ 2904.282890] ? do_raw_spin_unlock+0x75/0x190 [ 2904.282894] do_filp_open+0x99/0x140 [ 2904.282900] ? getname_flags+0x6e/0x330 [ 2904.282904] ? __check_object_size+0xff/0x256 [ 2904.282912] ? do_raw_spin_unlock+0x75/0x190 [ 2904.282916] ? _raw_spin_unlock+0x12/0x30 [ 2904.282920] do_sys_openat2+0x2b4/0x410 [ 2904.282924] do_sys_open+0x73/0xa0 [ 2904.282928] __x64_sys_openat+0x24/0x30 [ 2904.282931] do_syscall_64+0xc1/0x440 [ 2904.282938] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2904.282945] RIP: 0033:0x7ff54af45332 [ 2904.282955] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2904.282960] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2904.282966] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 2904.282968] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 2904.282971] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 2904.282973] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 2904.282975] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 2904.635780] ? watchdog+0x40/0x40 [ 2904.638240] ? hrtimer_interrupt+0x128/0x2c0 [ 2904.641130] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2904.643400] ? apic_timer_interrupt+0xf/0x20 [ 2904.646347] [ 2904.647410] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2904.650304] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2904.656740] do_raw_spin_lock+0x177/0x190 [ 2904.659523] _raw_spin_lock+0x21/0x30 [ 2904.664174] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2904.666949] process_one_work+0x2c8/0x700 [ 2904.670050] worker_thread+0x296/0x6e0 [ 2904.673067] ? rescuer_thread+0x570/0x570 [ 2904.677324] kthread+0x1d1/0x200 [ 2904.679063] ? set_kthread_struct+0x70/0x70 [ 2904.683076] ret_from_fork+0x1f/0x30 [ 2908.059818] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [ 2908.066004] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2908.106696] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2908.121790] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2908.137847] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2908.140036] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2908.163525] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2908.167409] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2908.174968] RDX: 000000000000611a RSI: 0000000000000003 RDI: ffff96848811cae0 [ 2908.178811] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 2908.182050] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 2908.185755] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2908.189432] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 2908.192873] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2908.194958] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 2908.198169] Call Trace: [ 2908.199040] [ 2908.199710] ? show_regs.cold.9+0x22/0x2f [ 2908.201881] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2908.203951] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2908.209772] ? watchdog+0x40/0x40 [ 2908.214429] ? hrtimer_interrupt+0x128/0x2c0 [ 2908.217699] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2908.222810] ? apic_timer_interrupt+0xf/0x20 [ 2908.225640] [ 2908.226545] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2908.228793] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2908.231631] do_raw_spin_lock+0x177/0x190 [ 2908.233682] _raw_spin_lock+0x21/0x30 [ 2908.235959] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2908.241010] obd_mod_free+0x46/0x140 [ptlrpc] [ 2908.244827] mdc_commit_open+0x91/0xb0 [mdc] [ 2908.246989] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2908.255686] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2908.261796] after_reply+0xe2c/0x1780 [ptlrpc] [ 2908.263621] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2908.269526] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2908.276281] ? do_wait_intr+0xf0/0xf0 [ 2908.277799] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2908.282733] kthread+0x1d1/0x200 [ 2908.284595] ? set_kthread_struct+0x70/0x70 [ 2908.286609] ret_from_fork+0x1f/0x30 [ 2913.866942] rcu: INFO: rcu_sched self-detected stall on CPU [ 2913.872452] rcu: 0-...!: (502287 ticks this GP) idle=98a/1/0x4000000000000002 softirq=125154/125154 fqs=959 [ 2913.880458] (t=960026 jiffies g=253585 q=2988) [ 2913.884973] rcu: rcu_sched kthread starved for 954986 jiffies! g253585 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [ 2913.902874] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 2913.914925] rcu: RCU grace-period kthread stack dump: [ 2913.918424] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 2913.923493] Call Trace: [ 2913.924712] __schedule+0x351/0xcb0 [ 2913.926119] schedule+0xc0/0x180 [ 2913.927245] schedule_timeout+0xb4/0x190 [ 2913.931842] ? do_raw_spin_unlock+0x75/0x190 [ 2913.933812] ? __next_timer_interrupt+0x160/0x160 [ 2913.936336] ? prepare_to_swait_event+0xf3/0x140 [ 2913.939675] rcu_gp_kthread+0x8dd/0x15b0 [ 2913.941470] ? force_qs_rnp+0x2c0/0x2c0 [ 2913.943695] kthread+0x1d1/0x200 [ 2913.945445] ? set_kthread_struct+0x70/0x70 [ 2913.947253] ret_from_fork+0x1f/0x30 [ 2913.949249] rcu: Stack dump where RCU GP kthread last ran: [ 2913.951158] NMI backtrace for cpu 0 [ 2913.952621] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2913.956700] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2913.962287] Call Trace: [ 2913.963420] [ 2913.964455] ? dump_stack+0xbb/0x10e [ 2913.967610] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 2913.970409] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 2913.971966] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 2913.976639] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 2913.980296] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 2913.983816] ? rcu_check_gp_kthread_starvation+0x16f/0x188 [ 2913.986726] ? rcu_sched_clock_irq.cold.51+0xce/0x307 [ 2913.990414] ? update_process_times+0x6d/0xc0 [ 2913.992192] ? tick_sched_handle.isra.18+0x27/0x90 [ 2913.994094] ? tick_sched_timer+0x53/0xb0 [ 2913.995647] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2913.997449] ? tick_sched_handle.isra.18+0x90/0x90 [ 2913.998918] ? hrtimer_interrupt+0x128/0x2c0 [ 2914.000468] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2914.004196] ? apic_timer_interrupt+0xf/0x20 [ 2914.008220] [ 2914.009637] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2914.014302] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2914.016972] ? do_raw_spin_lock+0x177/0x190 [ 2914.023660] ? _raw_spin_lock+0x21/0x30 [ 2914.024721] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2914.026291] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 2914.033950] ? mdc_commit_open+0x91/0xb0 [mdc] [ 2914.036518] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2914.039086] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2914.042190] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 2914.044803] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2914.047910] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2914.050424] ? do_wait_intr+0xf0/0xf0 [ 2914.052456] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2914.055423] ? kthread+0x1d1/0x200 [ 2914.057164] ? set_kthread_struct+0x70/0x70 [ 2914.058807] ? ret_from_fork+0x1f/0x30 [ 2914.062328] NMI backtrace for cpu 0 [ 2914.064181] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2914.078255] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2914.091243] Call Trace: [ 2914.092435] [ 2914.092997] ? dump_stack+0xbb/0x10e [ 2914.093908] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 2914.100655] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 2914.104895] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 2914.107956] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 2914.110706] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 2914.112602] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 2914.120791] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 2914.129721] ? update_process_times+0x6d/0xc0 [ 2914.131796] ? tick_sched_handle.isra.18+0x27/0x90 [ 2914.140695] ? tick_sched_timer+0x53/0xb0 [ 2914.144214] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2914.146761] ? tick_sched_handle.isra.18+0x90/0x90 [ 2914.149177] ? hrtimer_interrupt+0x128/0x2c0 [ 2914.151045] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2914.153556] ? apic_timer_interrupt+0xf/0x20 [ 2914.155422] [ 2914.156330] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2914.163664] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2914.167947] ? do_raw_spin_lock+0x177/0x190 [ 2914.170072] ? _raw_spin_lock+0x21/0x30 [ 2914.176175] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2914.185250] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 2914.188465] ? mdc_commit_open+0x91/0xb0 [mdc] [ 2914.191056] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2914.195014] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2914.199419] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 2914.203817] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2914.208499] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2914.214602] ? do_wait_intr+0xf0/0xf0 [ 2914.219378] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2914.230679] ? kthread+0x1d1/0x200 [ 2914.232431] ? set_kthread_struct+0x70/0x70 [ 2914.234945] ? ret_from_fork+0x1f/0x30 [ 2914.277729] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 955s! [ 2914.281611] Showing busy workqueues and worker pools: [ 2914.288349] workqueue events: flags=0x0 [ 2914.296890] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2914.305529] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 2914.337679] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 2914.344008] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 2914.359582] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2914.362027] pending: vmstat_shepherd [ 2914.364955] workqueue events_long: flags=0x0 [ 2914.371507] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2914.374950] pending: lu_objects_flush [obdclass] [ 2914.378071] workqueue events_unbound: flags=0x2 [ 2914.380070] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 2914.386376] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2914.393320] workqueue events_power_efficient: flags=0x80 [ 2914.398539] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2914.406052] pending: neigh_periodic_work [ 2914.411871] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2914.421445] pending: do_cache_clean, check_lifetime [ 2914.425971] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2914.431578] pending: neigh_periodic_work [ 2914.433356] workqueue writeback: flags=0x4a [ 2914.437785] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2914.440572] pending: wb_workfn, wb_workfn [ 2914.442339] workqueue xprtiod: flags=0xa [ 2914.444495] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 2914.448097] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 2914.456487] workqueue ipv6_addrconf: flags=0x40008 [ 2914.461900] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2914.467351] pending: addrconf_verify_work [ 2914.472400] workqueue knbd0-recv: flags=0x1a [ 2914.479464] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2914.483519] in-flight: 657:recv_work [ 2914.488507] workqueue ptlrpc_pinger: flags=0x2 [ 2914.491225] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2914.499101] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 2914.507796] pool 8: cpus=0-3 flags=0x4 nice=0 hung=955s workers=4 idle: 20831 12642 28859 [ 2914.519865] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2904s workers=2 idle: 659 [ 2928.058026] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 2928.062784] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2928.112595] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2928.121693] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2928.127333] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2928.130886] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2928.147670] RSP: 0018:ffffaf9fc1543ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2928.156344] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2928.160775] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 2928.165768] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2928.170366] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 2928.173105] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 2928.175908] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 2928.178815] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2928.180749] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 2928.183648] Call Trace: [ 2928.184311] [ 2928.184800] ? show_regs.cold.9+0x22/0x2f [ 2928.185986] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2928.187317] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2928.188722] ? watchdog+0x40/0x40 [ 2928.189656] ? hrtimer_interrupt+0x128/0x2c0 [ 2928.191180] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2928.193784] ? apic_timer_interrupt+0xf/0x20 [ 2928.196019] [ 2928.197771] ? native_safe_halt+0x2e/0x50 [ 2928.199583] kvm_wait+0xcd/0xe0 [ 2928.201003] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2928.203440] do_raw_spin_lock+0x177/0x190 [ 2928.206736] _raw_spin_lock+0x21/0x30 [ 2928.208153] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2928.209860] ? do_raw_spin_unlock+0x75/0x190 [ 2928.220636] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2928.225608] ? do_wait_intr+0xf0/0xf0 [ 2928.227660] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2928.233122] mdc_close+0x53b/0xd70 [mdc] [ 2928.236114] lmv_close+0x18d/0x490 [lmv] [ 2928.239649] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 2928.245162] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2928.249760] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2928.252827] ll_md_close+0x274/0x800 [lustre] [ 2928.257131] ll_file_release+0x128/0xc50 [lustre] [ 2928.261704] __fput+0x10d/0x440 [ 2928.263577] ____fput+0x12/0x20 [ 2928.264745] task_work_run+0xc6/0x110 [ 2928.266775] exit_to_usermode_loop+0x1dd/0x1f0 [ 2928.275102] do_syscall_64+0x3d6/0x440 [ 2928.277183] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2928.280792] RIP: 0033:0x7ff54af45cd8 [ 2928.282265] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2928.304658] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2928.312612] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 2928.316400] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 2928.321427] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 2928.328480] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 2928.333776] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2932.059034] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 2932.062626] Modules linked in: lustre(O) [ 2932.062638] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 2932.062642] osp(O) ofd(O) lod(O) [ 2932.074446] Modules linked in: lustre(O) osp(O) ofd(O) [ 2932.077798] mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2932.077882] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2932.077889] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2932.077895] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2932.078673] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2932.078687] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2932.078691] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2932.078695] RAX: 0000000000002381 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2932.078697] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 2932.078699] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 2932.078701] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 2932.078703] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 2932.078706] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 2932.078709] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2932.078711] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 2932.078719] Call Trace: [ 2932.078725] [ 2932.078731] ? show_regs.cold.9+0x22/0x2f [ 2932.078741] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2932.078747] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2932.078752] ? watchdog+0x40/0x40 [ 2932.078757] ? hrtimer_interrupt+0x128/0x2c0 [ 2932.078761] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2932.078770] ? apic_timer_interrupt+0xf/0x20 [ 2932.078775] [ 2932.078777] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2932.078782] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2932.078787] do_raw_spin_lock+0x177/0x190 [ 2932.078791] _raw_spin_lock+0x21/0x30 [ 2932.078797] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2932.079081] process_one_work+0x2c8/0x700 [ 2932.079090] worker_thread+0x296/0x6e0 [ 2932.079094] ? rescuer_thread+0x570/0x570 [ 2932.079097] kthread+0x1d1/0x200 [ 2932.079101] ? set_kthread_struct+0x70/0x70 [ 2932.079104] ret_from_fork+0x1f/0x30 [ 2932.333289] lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2932.376378] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2932.392508] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2932.406514] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2932.415675] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2932.435881] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2932.440758] RAX: 0000000000006712 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2932.448150] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 2932.455335] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2932.462755] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 2932.470175] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 2932.474940] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 2932.483736] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2932.487532] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 2932.494036] Call Trace: [ 2932.497502] [ 2932.501684] ? show_regs.cold.9+0x22/0x2f [ 2932.510931] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2932.513727] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2932.517594] ? watchdog+0x40/0x40 [ 2932.520422] ? hrtimer_interrupt+0x128/0x2c0 [ 2932.525278] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2932.531274] ? apic_timer_interrupt+0xf/0x20 [ 2932.533294] [ 2932.535520] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2932.542802] do_raw_spin_lock+0x177/0x190 [ 2932.548907] _raw_spin_lock+0x21/0x30 [ 2932.551021] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2932.557464] ? do_raw_spin_unlock+0x75/0x190 [ 2932.559951] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2932.564870] ? do_wait_intr+0xf0/0xf0 [ 2932.571218] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2932.577533] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2932.591559] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2932.604295] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2932.606344] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2932.628937] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2932.634679] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2932.637144] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2932.642880] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2932.647523] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2932.651238] lmv_intent_open+0x338/0x1150 [lmv] [ 2932.657463] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2932.660700] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2932.669818] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2932.680601] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2932.683674] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2932.688294] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2932.700144] ? do_raw_spin_unlock+0x75/0x190 [ 2932.708331] ? _raw_spin_unlock+0x12/0x30 [ 2932.715672] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2932.721296] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2932.724919] ll_atomic_open+0x36a/0xd70 [lustre] [ 2932.735466] lookup_open+0x356/0xb40 [ 2932.737680] do_last+0x313/0xfc0 [ 2932.747574] ? nd_jump_root+0xe5/0x160 [ 2932.750602] ? path_init+0x437/0x520 [ 2932.756594] path_openat+0xf7/0x500 [ 2932.760889] ? do_raw_spin_unlock+0x75/0x190 [ 2932.769368] do_filp_open+0x99/0x140 [ 2932.775573] ? getname_flags+0x6e/0x330 [ 2932.780406] ? __check_object_size+0xff/0x256 [ 2932.786421] ? do_raw_spin_unlock+0x75/0x190 [ 2932.789569] ? _raw_spin_unlock+0x12/0x30 [ 2932.793766] do_sys_openat2+0x2b4/0x410 [ 2932.799045] do_sys_open+0x73/0xa0 [ 2932.804806] __x64_sys_openat+0x24/0x30 [ 2932.808253] do_syscall_64+0xc1/0x440 [ 2932.811119] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2932.815370] RIP: 0033:0x7ff54af45332 [ 2932.819241] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2932.838802] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2932.846584] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 2932.850807] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 2932.858662] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 2932.866264] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 2932.878292] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 2940.061030] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [ 2940.071699] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2940.122584] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2940.133040] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2940.148329] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2940.154288] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2940.167730] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2940.180706] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2940.190420] RDX: 000000000000587f RSI: 0000000000000003 RDI: ffff96848811cae0 [ 2940.197211] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 2940.205702] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 2940.216864] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2940.225426] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 2940.236058] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2940.242402] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 2940.254426] Call Trace: [ 2940.257471] [ 2940.259653] ? show_regs.cold.9+0x22/0x2f [ 2940.265026] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2940.270720] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2940.275973] ? watchdog+0x40/0x40 [ 2940.278337] ? hrtimer_interrupt+0x128/0x2c0 [ 2940.282068] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2940.285620] ? apic_timer_interrupt+0xf/0x20 [ 2940.289955] [ 2940.291411] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2940.298075] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2940.305111] do_raw_spin_lock+0x177/0x190 [ 2940.310143] _raw_spin_lock+0x21/0x30 [ 2940.315038] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2940.319043] obd_mod_free+0x46/0x140 [ptlrpc] [ 2940.327110] mdc_commit_open+0x91/0xb0 [mdc] [ 2940.334931] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2940.339110] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2940.344593] after_reply+0xe2c/0x1780 [ptlrpc] [ 2940.348880] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2940.356505] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2940.362411] ? do_wait_intr+0xf0/0xf0 [ 2940.365643] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2940.373985] kthread+0x1d1/0x200 [ 2940.376059] ? set_kthread_struct+0x70/0x70 [ 2940.381090] ret_from_fork+0x1f/0x30 [ 2944.994342] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 986s! [ 2945.000635] Showing busy workqueues and worker pools: [ 2945.002929] workqueue events: flags=0x0 [ 2945.004639] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2945.015067] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 2945.023337] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 2945.029397] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 2945.036952] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2945.044366] pending: vmstat_shepherd [ 2945.048418] workqueue events_long: flags=0x0 [ 2945.050499] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2945.058514] pending: lu_objects_flush [obdclass] [ 2945.072839] workqueue events_unbound: flags=0x2 [ 2945.084151] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 2945.091341] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2945.100588] workqueue events_power_efficient: flags=0x80 [ 2945.104250] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2945.110412] pending: neigh_periodic_work [ 2945.113077] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2945.119112] pending: do_cache_clean, check_lifetime [ 2945.125662] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2945.132005] pending: neigh_periodic_work [ 2945.134234] workqueue writeback: flags=0x4a [ 2945.136193] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2945.144099] pending: wb_workfn, wb_workfn [ 2945.147614] workqueue xprtiod: flags=0xa [ 2945.151551] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 2945.154397] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 2945.161249] workqueue ipv6_addrconf: flags=0x40008 [ 2945.165302] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2945.168571] pending: addrconf_verify_work [ 2945.169689] workqueue knbd0-recv: flags=0x1a [ 2945.171649] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2945.173649] in-flight: 657:recv_work [ 2945.175322] workqueue ptlrpc_pinger: flags=0x2 [ 2945.176663] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2945.188649] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 2945.190667] pool 8: cpus=0-3 flags=0x4 nice=0 hung=986s workers=4 idle: 20831 12642 28859 [ 2945.194528] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2935s workers=2 idle: 659 [ 2956.062539] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [ 2956.069535] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2956.128283] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2956.146907] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2956.160927] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2956.167915] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2956.194732] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2956.203148] RAX: 0000000000007cb5 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2956.205819] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 2956.223942] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2956.235935] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 2956.248617] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 2956.257524] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 2956.274908] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2956.279673] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 2956.283857] Call Trace: [ 2956.285440] [ 2956.286486] ? show_regs.cold.9+0x22/0x2f [ 2956.288650] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2956.291020] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2956.295020] ? watchdog+0x40/0x40 [ 2956.296951] ? hrtimer_interrupt+0x128/0x2c0 [ 2956.299466] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2956.302460] ? apic_timer_interrupt+0xf/0x20 [ 2956.311905] [ 2956.316433] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2956.323862] do_raw_spin_lock+0x177/0x190 [ 2956.326420] _raw_spin_lock+0x21/0x30 [ 2956.329348] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2956.333463] ? do_raw_spin_unlock+0x75/0x190 [ 2956.335551] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2956.338853] ? do_wait_intr+0xf0/0xf0 [ 2956.341111] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2956.344379] mdc_close+0x53b/0xd70 [mdc] [ 2956.347021] lmv_close+0x18d/0x490 [lmv] [ 2956.349484] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 2956.352767] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2956.356118] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2956.362036] ll_md_close+0x274/0x800 [lustre] [ 2956.374184] ll_file_release+0x128/0xc50 [lustre] [ 2956.375931] __fput+0x10d/0x440 [ 2956.387908] ____fput+0x12/0x20 [ 2956.393030] task_work_run+0xc6/0x110 [ 2956.401374] exit_to_usermode_loop+0x1dd/0x1f0 [ 2956.402932] do_syscall_64+0x3d6/0x440 [ 2956.414417] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2956.424098] RIP: 0033:0x7ff54af45cd8 [ 2956.433794] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2956.470915] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2956.482333] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 2956.490689] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 2956.497995] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 2956.504269] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 2956.508395] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2960.060082] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 2960.060163] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 2960.072418] Modules linked in: [ 2960.076681] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2960.076827] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2960.076835] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2960.076841] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2960.077490] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2960.077503] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2960.077507] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2960.077511] RAX: 0000000000007f14 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2960.077513] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 2960.077516] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 2960.077518] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 2960.077520] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 2960.077523] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 2960.077526] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2960.077528] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 2960.077537] Call Trace: [ 2960.077542] [ 2960.077618] ? show_regs.cold.9+0x22/0x2f [ 2960.077630] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2960.077636] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2960.077640] ? watchdog+0x40/0x40 [ 2960.077645] ? hrtimer_interrupt+0x128/0x2c0 [ 2960.077649] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2960.077657] ? apic_timer_interrupt+0xf/0x20 [ 2960.077661] [ 2960.077664] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2960.077669] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2960.077674] do_raw_spin_lock+0x177/0x190 [ 2960.077679] _raw_spin_lock+0x21/0x30 [ 2960.077683] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2960.077980] process_one_work+0x2c8/0x700 [ 2960.077987] worker_thread+0x296/0x6e0 [ 2960.077991] ? rescuer_thread+0x570/0x570 [ 2960.077994] kthread+0x1d1/0x200 [ 2960.078280] ? set_kthread_struct+0x70/0x70 [ 2960.078287] ret_from_fork+0x1f/0x30 [ 2960.335024] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2960.355352] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2960.359828] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2960.363288] RIP: 0010:__do_softirq+0xa2/0x4e8 [ 2960.365352] Code: f6 01 01 c7 44 24 10 0a 00 00 00 48 c7 c0 40 2a 03 00 65 66 c7 00 00 00 48 c7 c0 30 bb f3 a1 48 83 00 01 fb 66 0f 1f 44 00 00 ff ff ff ff 48 83 05 b9 1f f6 01 01 48 c7 c3 00 61 c0 a0 41 0f [ 2960.373109] RSP: 0000:ffffaf9fc0d8cf78 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2960.375951] RAX: ffffffffa1f3bb30 RBX: ffff9684c219df40 RCX: 0000000000000001 [ 2960.378819] RDX: 000000000000024e RSI: 0000000000000016 RDI: ffffffffa2e010c0 [ 2960.381526] RBP: 0000000000000000 R08: 000002b13143e4c0 R09: 000002b13143e4c0 [ 2960.387696] R10: 00000000000f4240 R11: 000002b13143e4c0 R12: 0000000000000000 [ 2960.394613] R13: 0000000000000000 R14: 0000000000000080 R15: 0000000000000000 [ 2960.398804] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 2960.402834] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2960.412496] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 2960.422884] Call Trace: [ 2960.424370] [ 2960.426588] ? show_regs.cold.9+0x22/0x2f [ 2960.428675] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2960.431447] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2960.433774] ? watchdog+0x40/0x40 [ 2960.435529] ? hrtimer_interrupt+0x128/0x2c0 [ 2960.437827] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2960.440355] ? apic_timer_interrupt+0xf/0x20 [ 2960.445226] ? apic_timer_interrupt+0xa/0x20 [ 2960.450330] ? __do_softirq+0xa2/0x4e8 [ 2960.453686] ? __do_softirq+0x73/0x4e8 [ 2960.459182] ? kvm_sched_clock_read+0x2c/0x50 [ 2960.465258] irq_exit_rcu+0x1c6/0x1e0 [ 2960.469631] irq_exit+0x12/0x30 [ 2960.479431] smp_apic_timer_interrupt+0xe0/0x270 [ 2960.485077] apic_timer_interrupt+0xf/0x20 [ 2960.487429] [ 2960.491422] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2960.500384] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2960.528678] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2960.544675] RAX: 0000000000007f0b RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2960.549643] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 2960.557208] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2960.562529] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 2960.566915] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 2960.574526] do_raw_spin_lock+0x177/0x190 [ 2960.577426] _raw_spin_lock+0x21/0x30 [ 2960.582432] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2960.587934] ? do_raw_spin_unlock+0x75/0x190 [ 2960.592972] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2960.596343] ? do_wait_intr+0xf0/0xf0 [ 2960.597630] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2960.602363] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2960.612889] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2960.619925] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2960.625290] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2960.629174] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2960.632935] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2960.643448] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2960.647284] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2960.653440] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2960.657325] lmv_intent_open+0x338/0x1150 [lmv] [ 2960.660513] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2960.665357] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2960.673883] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2960.675428] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2960.678670] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2960.685685] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2960.689307] ? do_raw_spin_unlock+0x75/0x190 [ 2960.692243] ? _raw_spin_unlock+0x12/0x30 [ 2960.695488] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2960.701295] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2960.709373] ll_atomic_open+0x36a/0xd70 [lustre] [ 2960.716322] lookup_open+0x356/0xb40 [ 2960.719558] do_last+0x313/0xfc0 [ 2960.721854] ? nd_jump_root+0xe5/0x160 [ 2960.729735] ? path_init+0x437/0x520 [ 2960.733807] path_openat+0xf7/0x500 [ 2960.735399] ? do_raw_spin_unlock+0x75/0x190 [ 2960.741850] do_filp_open+0x99/0x140 [ 2960.748477] ? getname_flags+0x6e/0x330 [ 2960.758816] ? __check_object_size+0xff/0x256 [ 2960.768961] ? do_raw_spin_unlock+0x75/0x190 [ 2960.778255] ? _raw_spin_unlock+0x12/0x30 [ 2960.782220] do_sys_openat2+0x2b4/0x410 [ 2960.783458] do_sys_open+0x73/0xa0 [ 2960.784402] __x64_sys_openat+0x24/0x30 [ 2960.787530] do_syscall_64+0xc1/0x440 [ 2960.791244] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2960.793374] RIP: 0033:0x7ff54af45332 [ 2960.794416] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2960.803225] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2960.806933] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 2960.810966] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 2960.814100] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 2960.818417] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 2960.828236] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 2972.057730] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 2972.065099] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2972.090403] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2972.096747] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2972.111863] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2972.114829] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2972.121259] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2972.124701] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2972.127413] RDX: 000000000000235b RSI: 0000000000000003 RDI: ffff96848811cae0 [ 2972.130334] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 2972.133281] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 2972.135911] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2972.137846] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 2972.141238] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2972.143658] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 2972.146538] Call Trace: [ 2972.147429] [ 2972.148090] ? show_regs.cold.9+0x22/0x2f [ 2972.149736] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2972.151642] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2972.153572] ? watchdog+0x40/0x40 [ 2972.155034] ? hrtimer_interrupt+0x128/0x2c0 [ 2972.156622] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2972.158622] ? apic_timer_interrupt+0xf/0x20 [ 2972.160233] [ 2972.161186] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2972.163747] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2972.165865] do_raw_spin_lock+0x177/0x190 [ 2972.167667] _raw_spin_lock+0x21/0x30 [ 2972.168967] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2972.171137] obd_mod_free+0x46/0x140 [ptlrpc] [ 2972.173343] mdc_commit_open+0x91/0xb0 [mdc] [ 2972.175378] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2972.177777] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2972.180207] after_reply+0xe2c/0x1780 [ptlrpc] [ 2972.181933] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2972.183358] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2972.185207] ? do_wait_intr+0xf0/0xf0 [ 2972.186559] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2972.188830] kthread+0x1d1/0x200 [ 2972.189666] ? set_kthread_struct+0x70/0x70 [ 2972.191299] ret_from_fork+0x1f/0x30 [ 2975.714085] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1016s! [ 2975.722241] Showing busy workqueues and worker pools: [ 2975.727052] workqueue events: flags=0x0 [ 2975.730431] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2975.732928] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 2975.738163] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 2975.742713] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 2975.747741] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2975.750187] pending: vmstat_shepherd [ 2975.752056] workqueue events_long: flags=0x0 [ 2975.753880] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2975.756661] pending: lu_objects_flush [obdclass] [ 2975.758521] workqueue events_unbound: flags=0x2 [ 2975.759537] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 2975.761220] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2975.765290] workqueue events_power_efficient: flags=0x80 [ 2975.767505] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2975.770082] pending: neigh_periodic_work [ 2975.771776] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2975.774750] pending: do_cache_clean, check_lifetime [ 2975.776634] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2975.779174] pending: neigh_periodic_work [ 2975.781053] workqueue writeback: flags=0x4a [ 2975.782621] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2975.784927] pending: wb_workfn, wb_workfn [ 2975.786713] workqueue xprtiod: flags=0xa [ 2975.788165] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 2975.790734] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 2975.807687] workqueue ipv6_addrconf: flags=0x40008 [ 2975.809423] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2975.810773] pending: addrconf_verify_work [ 2975.812043] workqueue knbd0-recv: flags=0x1a [ 2975.813830] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2975.815911] in-flight: 657:recv_work [ 2975.817020] workqueue ptlrpc_pinger: flags=0x2 [ 2975.818204] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2975.819804] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 2975.822510] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1016s workers=4 idle: 20831 12642 28859 [ 2975.825036] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2966s workers=2 idle: 659 [ 2984.062759] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [ 2984.072359] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2984.130261] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2984.153161] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2984.160097] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2984.174099] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2984.202780] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2984.211722] RAX: 0000000000006c88 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2984.219727] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 2984.224760] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2984.229218] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 2984.242088] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 2984.247087] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 2984.250457] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2984.255020] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 2984.260246] Call Trace: [ 2984.261178] [ 2984.261894] ? show_regs.cold.9+0x22/0x2f [ 2984.266905] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2984.276150] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2984.284795] ? watchdog+0x40/0x40 [ 2984.288689] ? hrtimer_interrupt+0x128/0x2c0 [ 2984.306117] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2984.310647] ? apic_timer_interrupt+0xf/0x20 [ 2984.318251] [ 2984.319209] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2984.330096] do_raw_spin_lock+0x177/0x190 [ 2984.336942] _raw_spin_lock+0x21/0x30 [ 2984.342196] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2984.353337] ? do_raw_spin_unlock+0x75/0x190 [ 2984.367156] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2984.381342] ? do_wait_intr+0xf0/0xf0 [ 2984.395829] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2984.410179] mdc_close+0x53b/0xd70 [mdc] [ 2984.417504] lmv_close+0x18d/0x490 [lmv] [ 2984.420696] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 2984.431086] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2984.442213] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2984.448603] ll_md_close+0x274/0x800 [lustre] [ 2984.463722] ll_file_release+0x128/0xc50 [lustre] [ 2984.473796] __fput+0x10d/0x440 [ 2984.478162] ____fput+0x12/0x20 [ 2984.486148] task_work_run+0xc6/0x110 [ 2984.491613] exit_to_usermode_loop+0x1dd/0x1f0 [ 2984.500758] do_syscall_64+0x3d6/0x440 [ 2984.505559] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2984.512308] RIP: 0033:0x7ff54af45cd8 [ 2984.522883] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2984.552729] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2984.562931] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 2984.567090] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 2984.577249] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 2984.583255] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 2984.589225] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2988.059105] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 2988.060021] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 2988.062116] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2988.062192] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2988.062201] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2988.062207] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2988.066808] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2988.066883] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2988.066890] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2988.066893] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2988.066975] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2988.066979] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 2988.066984] RAX: 00000000000004ab RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2988.066986] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 2988.066989] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2988.066991] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 2988.066993] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 2988.066997] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 2988.066999] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2988.067002] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 2988.067071] Call Trace: [ 2988.067077] [ 2988.067082] ? show_regs.cold.9+0x22/0x2f [ 2988.067093] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2988.067099] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2988.067103] ? watchdog+0x40/0x40 [ 2988.067107] ? hrtimer_interrupt+0x128/0x2c0 [ 2988.067111] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2988.067119] ? apic_timer_interrupt+0xf/0x20 [ 2988.067123] [ 2988.067127] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2988.067133] do_raw_spin_lock+0x177/0x190 [ 2988.067138] _raw_spin_lock+0x21/0x30 [ 2988.067143] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2988.068429] ? do_raw_spin_unlock+0x75/0x190 [ 2988.068438] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2988.068750] ? do_wait_intr+0xf0/0xf0 [ 2988.068758] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2988.069125] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2988.069569] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2988.106181] [ 2988.106189] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2988.106205] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2988.106210] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2988.106215] RAX: 0000000000006b84 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 2988.106218] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 2988.106223] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 2988.106225] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 2988.106228] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 2988.106231] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 2988.106234] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2988.106236] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 2988.106243] Call Trace: [ 2988.106250] [ 2988.106256] ? show_regs.cold.9+0x22/0x2f [ 2988.106264] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2988.106271] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2988.106330] ? watchdog+0x40/0x40 [ 2988.106335] ? hrtimer_interrupt+0x128/0x2c0 [ 2988.106339] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2988.106349] ? apic_timer_interrupt+0xf/0x20 [ 2988.106354] [ 2988.106357] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2988.106361] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2988.106370] do_raw_spin_lock+0x177/0x190 [ 2988.106375] _raw_spin_lock+0x21/0x30 [ 2988.106380] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2988.106984] process_one_work+0x2c8/0x700 [ 2988.107213] worker_thread+0x296/0x6e0 [ 2988.107217] ? rescuer_thread+0x570/0x570 [ 2988.107221] kthread+0x1d1/0x200 [ 2988.107226] ? set_kthread_struct+0x70/0x70 [ 2988.107230] ret_from_fork+0x1f/0x30 [ 2988.688127] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2988.697889] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2988.702999] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2988.722345] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2988.724111] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2988.742683] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2988.744749] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2988.756997] lmv_intent_open+0x338/0x1150 [lmv] [ 2988.766905] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2988.769386] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2988.772463] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2988.774952] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2988.779950] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2988.793666] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2988.802866] ? do_raw_spin_unlock+0x75/0x190 [ 2988.812616] ? _raw_spin_unlock+0x12/0x30 [ 2988.819539] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2988.823580] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2988.826826] ll_atomic_open+0x36a/0xd70 [lustre] [ 2988.836734] lookup_open+0x356/0xb40 [ 2988.838094] do_last+0x313/0xfc0 [ 2988.839276] ? nd_jump_root+0xe5/0x160 [ 2988.843228] ? path_init+0x437/0x520 [ 2988.845244] path_openat+0xf7/0x500 [ 2988.849240] ? do_raw_spin_unlock+0x75/0x190 [ 2988.853308] do_filp_open+0x99/0x140 [ 2988.857721] ? getname_flags+0x6e/0x330 [ 2988.866080] ? __check_object_size+0xff/0x256 [ 2988.871087] ? do_raw_spin_unlock+0x75/0x190 [ 2988.874979] ? _raw_spin_unlock+0x12/0x30 [ 2988.877772] do_sys_openat2+0x2b4/0x410 [ 2988.879929] do_sys_open+0x73/0xa0 [ 2988.883039] __x64_sys_openat+0x24/0x30 [ 2988.884225] do_syscall_64+0xc1/0x440 [ 2988.886233] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2988.888100] RIP: 0033:0x7ff54af45332 [ 2988.889630] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2988.896728] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2988.902306] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 2988.904843] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 2988.911284] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 2988.916818] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 2988.922597] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 3000.055116] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [ 3000.059405] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3000.085882] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3000.092704] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3000.098073] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3000.100414] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3000.111817] RSP: 0000:ffffaf9fc1b97bb8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 3000.117176] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 3000.126561] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 3000.132062] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 3000.138681] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 3000.142094] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3000.144929] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 3000.150145] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3000.154344] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 3000.158931] Call Trace: [ 3000.159881] [ 3000.160901] ? show_regs.cold.9+0x22/0x2f [ 3000.163928] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3000.166853] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3000.168004] ? watchdog+0x40/0x40 [ 3000.168921] ? hrtimer_interrupt+0x128/0x2c0 [ 3000.172551] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3000.177040] ? apic_timer_interrupt+0xf/0x20 [ 3000.180816] [ 3000.181789] ? native_safe_halt+0x2e/0x50 [ 3000.183844] kvm_wait+0xcd/0xe0 [ 3000.186457] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3000.190939] do_raw_spin_lock+0x177/0x190 [ 3000.195286] _raw_spin_lock+0x21/0x30 [ 3000.197566] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3000.203719] obd_mod_free+0x46/0x140 [ptlrpc] [ 3000.208989] mdc_commit_open+0x91/0xb0 [mdc] [ 3000.216470] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3000.219396] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3000.223645] after_reply+0xe2c/0x1780 [ptlrpc] [ 3000.230756] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3000.234601] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3000.237771] ? do_wait_intr+0xf0/0xf0 [ 3000.241956] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3000.247883] kthread+0x1d1/0x200 [ 3000.249380] ? set_kthread_struct+0x70/0x70 [ 3000.253943] ret_from_fork+0x1f/0x30 [ 3005.920317] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1047s! [ 3005.930463] Showing busy workqueues and worker pools: [ 3005.932312] workqueue events: flags=0x0 [ 3005.933798] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3005.942689] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 3005.958621] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3005.969518] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 3005.978303] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3005.987640] pending: vmstat_shepherd [ 3005.992803] workqueue events_long: flags=0x0 [ 3005.996920] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3006.003836] pending: lu_objects_flush [obdclass] [ 3006.014519] workqueue events_unbound: flags=0x2 [ 3006.021730] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3006.030693] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3006.048754] workqueue events_power_efficient: flags=0x80 [ 3006.053910] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3006.058922] pending: neigh_periodic_work [ 3006.062408] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3006.065389] pending: do_cache_clean, check_lifetime [ 3006.069776] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3006.075387] pending: neigh_periodic_work [ 3006.079985] workqueue writeback: flags=0x4a [ 3006.087699] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3006.099032] pending: wb_workfn, wb_workfn [ 3006.102624] workqueue xprtiod: flags=0xa [ 3006.106756] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3006.115354] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 3006.122775] workqueue ipv6_addrconf: flags=0x40008 [ 3006.127919] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3006.135685] pending: addrconf_verify_work [ 3006.139903] workqueue knbd0-recv: flags=0x1a [ 3006.145429] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3006.151717] in-flight: 657:recv_work [ 3006.155394] workqueue ptlrpc_pinger: flags=0x2 [ 3006.158873] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3006.163414] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 3006.172482] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1047s workers=4 idle: 20831 12642 28859 [ 3006.179181] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2996s workers=2 idle: 659 [ 3012.058027] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [ 3012.061423] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3012.104586] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3012.111859] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3012.117777] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3012.120259] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3012.130754] RSP: 0018:ffffaf9fc1543ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3012.138124] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 3012.145286] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 3012.148680] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3012.151656] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 3012.157369] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 3012.162792] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 3012.171402] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3012.175977] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 3012.180760] Call Trace: [ 3012.182258] [ 3012.183034] ? show_regs.cold.9+0x22/0x2f [ 3012.185253] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3012.187513] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3012.189727] ? watchdog+0x40/0x40 [ 3012.192548] ? hrtimer_interrupt+0x128/0x2c0 [ 3012.195515] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3012.197645] ? apic_timer_interrupt+0xf/0x20 [ 3012.199489] [ 3012.200232] ? native_safe_halt+0x2e/0x50 [ 3012.201486] kvm_wait+0xcd/0xe0 [ 3012.202736] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3012.205325] do_raw_spin_lock+0x177/0x190 [ 3012.206763] _raw_spin_lock+0x21/0x30 [ 3012.209034] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3012.214362] ? do_raw_spin_unlock+0x75/0x190 [ 3012.217264] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3012.221526] ? do_wait_intr+0xf0/0xf0 [ 3012.222744] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3012.227851] mdc_close+0x53b/0xd70 [mdc] [ 3012.231432] lmv_close+0x18d/0x490 [lmv] [ 3012.234514] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 3012.239900] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3012.245656] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3012.250938] ll_md_close+0x274/0x800 [lustre] [ 3012.252881] ll_file_release+0x128/0xc50 [lustre] [ 3012.254727] __fput+0x10d/0x440 [ 3012.256385] ____fput+0x12/0x20 [ 3012.257699] task_work_run+0xc6/0x110 [ 3012.259661] exit_to_usermode_loop+0x1dd/0x1f0 [ 3012.262195] do_syscall_64+0x3d6/0x440 [ 3012.264436] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3012.270249] RIP: 0033:0x7ff54af45cd8 [ 3012.272494] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3012.303939] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3012.308585] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 3012.320350] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 3012.324540] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 3012.328720] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 3012.332563] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3016.059037] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 3016.060025] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 3016.061242] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3016.061319] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3016.061326] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3016.061332] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3016.061869] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3016.061879] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3016.061883] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3016.061887] RAX: 0000000000002b83 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 3016.061889] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 3016.061891] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 3016.061893] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 3016.061895] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 3016.061898] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 3016.061901] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3016.061904] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 3016.061912] Call Trace: [ 3016.061917] [ 3016.061922] ? show_regs.cold.9+0x22/0x2f [ 3016.061930] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3016.061940] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3016.061945] ? watchdog+0x40/0x40 [ 3016.061948] ? hrtimer_interrupt+0x128/0x2c0 [ 3016.061952] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3016.061958] ? apic_timer_interrupt+0xf/0x20 [ 3016.061962] [ 3016.061964] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3016.061968] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3016.061973] do_raw_spin_lock+0x177/0x190 [ 3016.061977] _raw_spin_lock+0x21/0x30 [ 3016.061981] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3016.062247] process_one_work+0x2c8/0x700 [ 3016.062253] worker_thread+0x296/0x6e0 [ 3016.062257] ? rescuer_thread+0x570/0x570 [ 3016.062260] kthread+0x1d1/0x200 [ 3016.062265] ? set_kthread_struct+0x70/0x70 [ 3016.062268] ret_from_fork+0x1f/0x30 [ 3016.206401] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3016.239360] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3016.243502] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3016.247135] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3016.249133] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3016.256674] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 3016.259425] RAX: 000000000000300d RBX: ffff96848811cae0 RCX: 0000000000000001 [ 3016.261929] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 3016.263991] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3016.266224] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 3016.268902] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 3016.271328] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 3016.274399] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3016.276497] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 3016.278967] Call Trace: [ 3016.279990] [ 3016.280764] ? show_regs.cold.9+0x22/0x2f [ 3016.282227] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3016.284198] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3016.285869] ? watchdog+0x40/0x40 [ 3016.286907] ? hrtimer_interrupt+0x128/0x2c0 [ 3016.288267] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3016.289525] ? apic_timer_interrupt+0xf/0x20 [ 3016.290589] [ 3016.291147] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3016.292786] do_raw_spin_lock+0x177/0x190 [ 3016.293958] _raw_spin_lock+0x21/0x30 [ 3016.299224] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3016.307020] ? do_raw_spin_unlock+0x75/0x190 [ 3016.310796] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3016.333961] ? do_wait_intr+0xf0/0xf0 [ 3016.345193] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3016.352033] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3016.362172] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3016.377136] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3016.379350] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3016.397084] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3016.406241] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3016.407797] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3016.417972] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3016.430938] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3016.444204] lmv_intent_open+0x338/0x1150 [lmv] [ 3016.450199] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3016.456696] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3016.462829] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3016.466237] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3016.468090] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3016.476267] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3016.483355] ? do_raw_spin_unlock+0x75/0x190 [ 3016.488223] ? _raw_spin_unlock+0x12/0x30 [ 3016.494049] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3016.506258] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3016.517400] ll_atomic_open+0x36a/0xd70 [lustre] [ 3016.526071] lookup_open+0x356/0xb40 [ 3016.529073] do_last+0x313/0xfc0 [ 3016.537192] ? nd_jump_root+0xe5/0x160 [ 3016.540394] ? path_init+0x437/0x520 [ 3016.549911] path_openat+0xf7/0x500 [ 3016.558967] ? do_raw_spin_unlock+0x75/0x190 [ 3016.564452] do_filp_open+0x99/0x140 [ 3016.565718] ? getname_flags+0x6e/0x330 [ 3016.574851] ? __check_object_size+0xff/0x256 [ 3016.578567] ? do_raw_spin_unlock+0x75/0x190 [ 3016.582893] ? _raw_spin_unlock+0x12/0x30 [ 3016.584755] do_sys_openat2+0x2b4/0x410 [ 3016.586152] do_sys_open+0x73/0xa0 [ 3016.589153] __x64_sys_openat+0x24/0x30 [ 3016.592795] do_syscall_64+0xc1/0x440 [ 3016.598845] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3016.604415] RIP: 0033:0x7ff54af45332 [ 3016.613347] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3016.628152] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3016.632982] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 3016.638151] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 3016.648641] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 3016.660004] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 3016.673863] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 3032.055042] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 3032.057220] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3032.090782] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3032.116067] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3032.130575] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3032.132371] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3032.146678] RSP: 0000:ffffaf9fc1b97bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3032.151609] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 3032.160221] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 3032.168323] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 3032.180351] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 3032.195639] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3032.206597] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 3032.223585] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3032.229757] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 3032.239541] Call Trace: [ 3032.243430] [ 3032.247229] ? show_regs.cold.9+0x22/0x2f [ 3032.251330] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3032.258649] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3032.267822] ? watchdog+0x40/0x40 [ 3032.272568] ? hrtimer_interrupt+0x128/0x2c0 [ 3032.278785] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3032.290832] ? apic_timer_interrupt+0xf/0x20 [ 3032.303574] [ 3032.309445] ? native_safe_halt+0x2e/0x50 [ 3032.316236] kvm_wait+0xcd/0xe0 [ 3032.321245] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3032.328227] do_raw_spin_lock+0x177/0x190 [ 3032.334779] _raw_spin_lock+0x21/0x30 [ 3032.338341] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3032.342152] obd_mod_free+0x46/0x140 [ptlrpc] [ 3032.350080] mdc_commit_open+0x91/0xb0 [mdc] [ 3032.355765] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3032.366331] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3032.374207] after_reply+0xe2c/0x1780 [ptlrpc] [ 3032.379732] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3032.387146] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3032.390752] ? do_wait_intr+0xf0/0xf0 [ 3032.394193] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3032.400436] kthread+0x1d1/0x200 [ 3032.402722] ? set_kthread_struct+0x70/0x70 [ 3032.408374] ret_from_fork+0x1f/0x30 [ 3036.645667] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1077s! [ 3036.659746] Showing busy workqueues and worker pools: [ 3036.663849] workqueue events: flags=0x0 [ 3036.669687] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3036.694968] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 3036.705106] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3036.712788] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 3036.716784] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3036.720106] pending: vmstat_shepherd [ 3036.722718] workqueue events_long: flags=0x0 [ 3036.726436] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3036.736734] pending: lu_objects_flush [obdclass] [ 3036.742425] workqueue events_unbound: flags=0x2 [ 3036.748710] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3036.752355] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3036.758617] workqueue events_power_efficient: flags=0x80 [ 3036.762726] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3036.774206] pending: neigh_periodic_work [ 3036.780061] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3036.784795] pending: do_cache_clean, check_lifetime [ 3036.787314] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3036.790327] pending: neigh_periodic_work [ 3036.792689] workqueue writeback: flags=0x4a [ 3036.795142] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3036.798277] pending: wb_workfn, wb_workfn [ 3036.800850] workqueue xprtiod: flags=0xa [ 3036.802453] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3036.805429] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 3036.808885] workqueue ipv6_addrconf: flags=0x40008 [ 3036.810974] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3036.814103] pending: addrconf_verify_work [ 3036.816286] workqueue knbd0-recv: flags=0x1a [ 3036.818448] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3036.821062] in-flight: 657:recv_work [ 3036.822536] workqueue ptlrpc_pinger: flags=0x2 [ 3036.824285] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3036.826452] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 3036.829234] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1077s workers=4 idle: 20831 12642 28859 [ 3036.832562] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3027s workers=2 idle: 659 [ 3040.062372] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [ 3040.075452] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3040.144572] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3040.154285] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3040.161780] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3040.168296] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3040.186616] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3040.191821] RAX: 0000000000007d14 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 3040.196761] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 3040.201800] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3040.210790] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 3040.215825] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 3040.222651] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 3040.234961] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3040.241549] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 3040.248977] Call Trace: [ 3040.251119] [ 3040.252545] ? show_regs.cold.9+0x22/0x2f [ 3040.255205] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3040.259890] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3040.264642] ? watchdog+0x40/0x40 [ 3040.266552] ? hrtimer_interrupt+0x128/0x2c0 [ 3040.268700] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3040.272184] ? apic_timer_interrupt+0xf/0x20 [ 3040.274029] [ 3040.274869] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3040.278112] do_raw_spin_lock+0x177/0x190 [ 3040.280659] _raw_spin_lock+0x21/0x30 [ 3040.282521] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3040.285416] ? do_raw_spin_unlock+0x75/0x190 [ 3040.287747] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3040.290843] ? do_wait_intr+0xf0/0xf0 [ 3040.292851] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3040.295553] mdc_close+0x53b/0xd70 [mdc] [ 3040.297482] lmv_close+0x18d/0x490 [lmv] [ 3040.299557] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 3040.302626] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3040.305721] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3040.308696] ll_md_close+0x274/0x800 [lustre] [ 3040.311706] ll_file_release+0x128/0xc50 [lustre] [ 3040.314748] __fput+0x10d/0x440 [ 3040.316567] ____fput+0x12/0x20 [ 3040.318484] task_work_run+0xc6/0x110 [ 3040.320629] exit_to_usermode_loop+0x1dd/0x1f0 [ 3040.323296] do_syscall_64+0x3d6/0x440 [ 3040.326205] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3040.328987] RIP: 0033:0x7ff54af45cd8 [ 3040.330928] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3040.341353] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3040.345098] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 3040.349005] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 3040.352875] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 3040.356835] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 3040.361327] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3044.059124] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 3044.062274] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3044.066523] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [ 3044.084057] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3044.084071] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3044.084077] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3044.084355] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3044.084366] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3044.084370] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3044.084374] RAX: 0000000000005d8c RBX: ffff96848811cae0 RCX: 0000000000000001 [ 3044.084377] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 3044.084379] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 3044.084381] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 3044.084384] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 3044.084387] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 3044.084390] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3044.084392] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 3044.084401] Call Trace: [ 3044.084406] [ 3044.084412] ? show_regs.cold.9+0x22/0x2f [ 3044.084421] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3044.084428] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3044.084432] ? watchdog+0x40/0x40 [ 3044.084436] ? hrtimer_interrupt+0x128/0x2c0 [ 3044.084441] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3044.084449] ? apic_timer_interrupt+0xf/0x20 [ 3044.084453] [ 3044.084456] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3044.084460] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3044.084465] do_raw_spin_lock+0x177/0x190 [ 3044.084469] _raw_spin_lock+0x21/0x30 [ 3044.084473] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3044.084689] process_one_work+0x2c8/0x700 [ 3044.084696] worker_thread+0x296/0x6e0 [ 3044.084701] ? rescuer_thread+0x570/0x570 [ 3044.084705] kthread+0x1d1/0x200 [ 3044.084710] ? set_kthread_struct+0x70/0x70 [ 3044.084714] ret_from_fork+0x1f/0x30 [ 3044.206585] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3044.221251] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3044.229616] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3044.233704] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3044.236368] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3044.251882] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3044.255281] RAX: 0000000000003086 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 3044.259009] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 3044.264717] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3044.272296] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 3044.280771] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 3044.283847] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 3044.287966] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3044.290844] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 3044.293929] Call Trace: [ 3044.294815] [ 3044.295427] ? show_regs.cold.9+0x22/0x2f [ 3044.297004] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3044.298553] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3044.300277] ? watchdog+0x40/0x40 [ 3044.301539] ? hrtimer_interrupt+0x128/0x2c0 [ 3044.303353] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3044.305122] ? apic_timer_interrupt+0xf/0x20 [ 3044.306444] [ 3044.307348] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3044.308811] do_raw_spin_lock+0x177/0x190 [ 3044.310130] _raw_spin_lock+0x21/0x30 [ 3044.311495] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3044.313538] ? do_raw_spin_unlock+0x75/0x190 [ 3044.315305] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3044.317248] ? do_wait_intr+0xf0/0xf0 [ 3044.318373] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3044.320342] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3044.322291] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3044.324050] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3044.326206] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3044.327752] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3044.329576] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3044.331239] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3044.342047] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3044.344038] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3044.357056] lmv_intent_open+0x338/0x1150 [lmv] [ 3044.362915] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3044.371914] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3044.377776] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3044.381588] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3044.386725] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3044.393089] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3044.396187] ? do_raw_spin_unlock+0x75/0x190 [ 3044.405410] ? _raw_spin_unlock+0x12/0x30 [ 3044.407082] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3044.415451] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3044.418236] ll_atomic_open+0x36a/0xd70 [lustre] [ 3044.423275] lookup_open+0x356/0xb40 [ 3044.429443] do_last+0x313/0xfc0 [ 3044.430920] ? nd_jump_root+0xe5/0x160 [ 3044.433064] ? path_init+0x437/0x520 [ 3044.438572] path_openat+0xf7/0x500 [ 3044.444549] ? do_raw_spin_unlock+0x75/0x190 [ 3044.446665] do_filp_open+0x99/0x140 [ 3044.448771] ? getname_flags+0x6e/0x330 [ 3044.454324] ? __check_object_size+0xff/0x256 [ 3044.456534] ? do_raw_spin_unlock+0x75/0x190 [ 3044.461703] ? _raw_spin_unlock+0x12/0x30 [ 3044.465480] do_sys_openat2+0x2b4/0x410 [ 3044.468555] do_sys_open+0x73/0xa0 [ 3044.470524] __x64_sys_openat+0x24/0x30 [ 3044.472381] do_syscall_64+0xc1/0x440 [ 3044.474408] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3044.482205] RIP: 0033:0x7ff54af45332 [ 3044.488796] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3044.506591] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3044.513892] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 3044.522355] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 3044.526820] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 3044.531388] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 3044.538643] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 3064.056350] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 3064.060304] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3064.088180] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3064.093954] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3064.098507] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3064.106461] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3064.122959] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3064.126873] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 3064.130011] RDX: 000000000000242b RSI: 0000000000000003 RDI: ffff96848811cae0 [ 3064.133019] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 3064.136033] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 3064.139037] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3064.142091] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 3064.145323] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3064.147776] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 3064.152384] Call Trace: [ 3064.153649] [ 3064.154986] ? show_regs.cold.9+0x22/0x2f [ 3064.156917] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3064.159187] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3064.161472] ? watchdog+0x40/0x40 [ 3064.163250] ? hrtimer_interrupt+0x128/0x2c0 [ 3064.165307] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3064.167779] ? apic_timer_interrupt+0xf/0x20 [ 3064.170103] [ 3064.171262] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3064.174090] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3064.177199] do_raw_spin_lock+0x177/0x190 [ 3064.179132] _raw_spin_lock+0x21/0x30 [ 3064.181271] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3064.183778] obd_mod_free+0x46/0x140 [ptlrpc] [ 3064.186172] mdc_commit_open+0x91/0xb0 [mdc] [ 3064.188482] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3064.191905] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3064.195291] after_reply+0xe2c/0x1780 [ptlrpc] [ 3064.197659] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3064.205080] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3064.207211] ? do_wait_intr+0xf0/0xf0 [ 3064.210323] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3064.213102] kthread+0x1d1/0x200 [ 3064.214502] ? set_kthread_struct+0x70/0x70 [ 3064.216500] ret_from_fork+0x1f/0x30 [ 3066.849650] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1107s! [ 3066.859485] Showing busy workqueues and worker pools: [ 3066.862695] workqueue events: flags=0x0 [ 3066.867073] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3066.879657] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 3066.890878] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3066.895856] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 3066.906016] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3066.927049] pending: vmstat_shepherd [ 3066.933695] workqueue events_long: flags=0x0 [ 3066.944838] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3066.954830] pending: lu_objects_flush [obdclass] [ 3066.967049] workqueue events_unbound: flags=0x2 [ 3066.976430] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3066.992868] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3067.003547] workqueue events_power_efficient: flags=0x80 [ 3067.016758] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3067.023439] pending: neigh_periodic_work [ 3067.031480] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3067.038796] pending: do_cache_clean, check_lifetime [ 3067.043918] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3067.053288] pending: neigh_periodic_work [ 3067.056417] workqueue writeback: flags=0x4a [ 3067.058971] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3067.063733] pending: wb_workfn, wb_workfn [ 3067.067627] workqueue xprtiod: flags=0xa [ 3067.070815] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3067.076952] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 3067.081273] workqueue ipv6_addrconf: flags=0x40008 [ 3067.084290] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3067.089563] pending: addrconf_verify_work [ 3067.091950] workqueue knbd0-recv: flags=0x1a [ 3067.097877] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3067.103691] in-flight: 657:recv_work [ 3067.107491] workqueue ptlrpc_pinger: flags=0x2 [ 3067.110775] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3067.115594] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 3067.118712] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1108s workers=4 idle: 20831 12642 28859 [ 3067.124293] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3057s workers=2 idle: 659 [ 3068.058608] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 3068.061935] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3068.082422] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3068.086278] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3068.089357] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3068.090849] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3068.096461] RSP: 0018:ffffaf9fc1543ae0 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 3068.099489] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 3068.101866] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 3068.104029] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3068.106279] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 3068.108413] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 3068.110786] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 3068.113691] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3068.116110] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 3068.118573] Call Trace: [ 3068.120128] [ 3068.120932] ? show_regs.cold.9+0x22/0x2f [ 3068.122408] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3068.123929] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3068.125198] ? watchdog+0x40/0x40 [ 3068.126307] ? hrtimer_interrupt+0x128/0x2c0 [ 3068.127562] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3068.129223] ? apic_timer_interrupt+0xf/0x20 [ 3068.131314] [ 3068.132084] ? native_safe_halt+0x2e/0x50 [ 3068.133683] kvm_wait+0xcd/0xe0 [ 3068.135042] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3068.137276] do_raw_spin_lock+0x177/0x190 [ 3068.138920] _raw_spin_lock+0x21/0x30 [ 3068.140104] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3068.143068] ? do_raw_spin_unlock+0x75/0x190 [ 3068.144711] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3068.152490] ? do_wait_intr+0xf0/0xf0 [ 3068.156218] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3068.161348] mdc_close+0x53b/0xd70 [mdc] [ 3068.165262] lmv_close+0x18d/0x490 [lmv] [ 3068.167918] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 3068.170613] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3068.173227] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3068.175244] ll_md_close+0x274/0x800 [lustre] [ 3068.176805] ll_file_release+0x128/0xc50 [lustre] [ 3068.178763] __fput+0x10d/0x440 [ 3068.180008] ____fput+0x12/0x20 [ 3068.181321] task_work_run+0xc6/0x110 [ 3068.183353] exit_to_usermode_loop+0x1dd/0x1f0 [ 3068.184854] do_syscall_64+0x3d6/0x440 [ 3068.186415] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3068.188487] RIP: 0033:0x7ff54af45cd8 [ 3068.191470] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3068.203208] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3068.208263] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 3068.210568] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 3068.213216] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 3068.216048] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 3068.218695] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3072.060720] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 3072.061107] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [ 3072.065028] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3072.065112] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3072.065120] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3072.065126] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3072.065430] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3072.065441] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3072.065445] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3072.065449] RAX: 000000000000568f RBX: ffff96848811cae0 RCX: 0000000000000001 [ 3072.065452] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 3072.065454] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 3072.065456] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 3072.065458] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 3072.066500] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 3072.066504] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3072.066506] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 3072.066515] Call Trace: [ 3072.066521] [ 3072.066526] ? show_regs.cold.9+0x22/0x2f [ 3072.066536] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3072.066544] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3072.066548] ? watchdog+0x40/0x40 [ 3072.066552] ? hrtimer_interrupt+0x128/0x2c0 [ 3072.066556] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3072.066564] ? apic_timer_interrupt+0xf/0x20 [ 3072.066569] [ 3072.066572] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3072.066578] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3072.066584] do_raw_spin_lock+0x177/0x190 [ 3072.066590] _raw_spin_lock+0x21/0x30 [ 3072.066728] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3072.066994] process_one_work+0x2c8/0x700 [ 3072.067001] worker_thread+0x296/0x6e0 [ 3072.067005] ? rescuer_thread+0x570/0x570 [ 3072.067009] kthread+0x1d1/0x200 [ 3072.067013] ? set_kthread_struct+0x70/0x70 [ 3072.067017] ret_from_fork+0x1f/0x30 [ 3072.548544] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3072.606346] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3072.620156] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3072.635306] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3072.637427] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3072.667697] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3072.678712] RAX: 0000000000007f0e RBX: ffff96848811cae0 RCX: 0000000000000001 [ 3072.696747] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 3072.703641] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3072.717725] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 3072.727354] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 3072.734374] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 3072.746343] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3072.758448] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 3072.764248] Call Trace: [ 3072.764860] [ 3072.765550] ? show_regs.cold.9+0x22/0x2f [ 3072.794739] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3072.804351] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3072.810271] ? watchdog+0x40/0x40 [ 3072.811460] ? hrtimer_interrupt+0x128/0x2c0 [ 3072.826069] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3072.830730] ? apic_timer_interrupt+0xf/0x20 [ 3072.841820] [ 3072.842775] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3072.844513] do_raw_spin_lock+0x177/0x190 [ 3072.849565] _raw_spin_lock+0x21/0x30 [ 3072.850844] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3072.859997] ? do_raw_spin_unlock+0x75/0x190 [ 3072.867496] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3072.872240] ? do_wait_intr+0xf0/0xf0 [ 3072.878879] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3072.881928] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3072.886166] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3072.894761] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3072.904897] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3072.912673] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3072.921284] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3072.925265] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3072.937116] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3072.949622] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3072.959907] lmv_intent_open+0x338/0x1150 [lmv] [ 3072.963392] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3072.978271] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3072.980931] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3072.992440] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3072.997761] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3073.009242] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3073.019016] ? do_raw_spin_unlock+0x75/0x190 [ 3073.021405] ? _raw_spin_unlock+0x12/0x30 [ 3073.026034] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3073.030067] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3073.034434] ll_atomic_open+0x36a/0xd70 [lustre] [ 3073.044472] lookup_open+0x356/0xb40 [ 3073.053441] do_last+0x313/0xfc0 [ 3073.054356] ? nd_jump_root+0xe5/0x160 [ 3073.060394] ? path_init+0x437/0x520 [ 3073.062883] path_openat+0xf7/0x500 [ 3073.072340] ? do_raw_spin_unlock+0x75/0x190 [ 3073.081396] do_filp_open+0x99/0x140 [ 3073.092201] ? getname_flags+0x6e/0x330 [ 3073.099190] ? __check_object_size+0xff/0x256 [ 3073.110045] ? do_raw_spin_unlock+0x75/0x190 [ 3073.118086] ? _raw_spin_unlock+0x12/0x30 [ 3073.124136] do_sys_openat2+0x2b4/0x410 [ 3073.130414] do_sys_open+0x73/0xa0 [ 3073.137739] __x64_sys_openat+0x24/0x30 [ 3073.147428] do_syscall_64+0xc1/0x440 [ 3073.151011] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3073.159875] RIP: 0033:0x7ff54af45332 [ 3073.164769] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3073.188893] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3073.207235] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 3073.211069] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 3073.226199] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 3073.235403] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 3073.247172] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 3092.055064] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 3092.059734] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3092.103610] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3092.138678] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3092.156154] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3092.159350] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3092.208455] RSP: 0000:ffffaf9fc1b97bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3092.211647] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 3092.214982] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 3092.218066] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 3092.221037] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 3092.224141] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3092.226738] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 3092.230096] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3092.232335] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 3092.235059] Call Trace: [ 3092.236087] [ 3092.236941] ? show_regs.cold.9+0x22/0x2f [ 3092.238572] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3092.240470] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3092.242238] ? watchdog+0x40/0x40 [ 3092.243571] ? hrtimer_interrupt+0x128/0x2c0 [ 3092.245072] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3092.246920] ? apic_timer_interrupt+0xf/0x20 [ 3092.248726] [ 3092.249854] ? native_safe_halt+0x2e/0x50 [ 3092.251533] kvm_wait+0xcd/0xe0 [ 3092.252677] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3092.254597] do_raw_spin_lock+0x177/0x190 [ 3092.260388] _raw_spin_lock+0x21/0x30 [ 3092.271100] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3092.273671] obd_mod_free+0x46/0x140 [ptlrpc] [ 3092.276385] mdc_commit_open+0x91/0xb0 [mdc] [ 3092.278353] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3092.286532] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3092.289231] after_reply+0xe2c/0x1780 [ptlrpc] [ 3092.296123] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3092.304140] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3092.306680] ? do_wait_intr+0xf0/0xf0 [ 3092.310194] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3092.314605] kthread+0x1d1/0x200 [ 3092.316666] ? set_kthread_struct+0x70/0x70 [ 3092.322312] ret_from_fork+0x1f/0x30 [ 3093.869458] rcu: INFO: rcu_sched self-detected stall on CPU [ 3093.872104] rcu: 0-...!: (591734 ticks this GP) idle=98a/1/0x4000000000000002 softirq=125154/125154 fqs=959 [ 3093.882677] (t=1140029 jiffies g=253585 q=3123) [ 3093.885045] rcu: rcu_sched kthread starved for 1134989 jiffies! g253585 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [ 3093.891650] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 3093.895901] rcu: RCU grace-period kthread stack dump: [ 3093.898463] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 3093.902871] Call Trace: [ 3093.903940] __schedule+0x351/0xcb0 [ 3093.905451] schedule+0xc0/0x180 [ 3093.907120] schedule_timeout+0xb4/0x190 [ 3093.908825] ? do_raw_spin_unlock+0x75/0x190 [ 3093.910620] ? __next_timer_interrupt+0x160/0x160 [ 3093.912573] ? prepare_to_swait_event+0xf3/0x140 [ 3093.914761] rcu_gp_kthread+0x8dd/0x15b0 [ 3093.916458] ? force_qs_rnp+0x2c0/0x2c0 [ 3093.918002] kthread+0x1d1/0x200 [ 3093.924501] ? set_kthread_struct+0x70/0x70 [ 3093.927998] ret_from_fork+0x1f/0x30 [ 3093.930284] rcu: Stack dump where RCU GP kthread last ran: [ 3093.933108] Sending NMI from CPU 0 to CPUs 2: [ 3093.936853] NMI backtrace for cpu 2 [ 3093.936859] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3093.936860] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3093.936862] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3093.936864] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3093.936865] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3093.936867] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 [ 3093.936869] RAX: 0000000000006b7c RBX: ffff96848811cae0 RCX: 0000000000000001 [ 3093.936871] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 3093.936872] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 3093.936873] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 3093.936875] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 3093.936876] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 3093.936877] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3093.936879] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 3093.936880] Call Trace: [ 3093.936881] [ 3093.936881] ? show_regs.cold.9+0x22/0x2f [ 3093.936882] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 3093.936884] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 3093.936885] ? nmi_handle+0xbd/0x210 [ 3093.936885] ? default_do_nmi+0x4b/0x1b0 [ 3093.936886] ? do_nmi+0x280/0x380 [ 3093.936887] ? end_repeat_nmi+0x16/0x1f [ 3093.936888] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3093.936889] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3093.936890] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3093.936891] [ 3093.936892] do_raw_spin_lock+0x177/0x190 [ 3093.936892] _raw_spin_lock+0x21/0x30 [ 3093.936893] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3093.936894] process_one_work+0x2c8/0x700 [ 3093.936895] worker_thread+0x296/0x6e0 [ 3093.936896] ? rescuer_thread+0x570/0x570 [ 3093.936897] kthread+0x1d1/0x200 [ 3093.936898] ? set_kthread_struct+0x70/0x70 [ 3093.936899] ret_from_fork+0x1f/0x30 [ 3093.937435] NMI backtrace for cpu 0 [ 3094.045004] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3094.049346] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3094.059836] Call Trace: [ 3094.060890] [ 3094.061999] ? dump_stack+0xbb/0x10e [ 3094.066416] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 3094.069358] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 3094.071611] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 3094.073853] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 3094.076019] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 3094.078476] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 3094.080591] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 3094.082922] ? update_process_times+0x6d/0xc0 [ 3094.084802] ? tick_sched_handle.isra.18+0x27/0x90 [ 3094.087310] ? tick_sched_timer+0x53/0xb0 [ 3094.088901] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3094.091380] ? tick_sched_handle.isra.18+0x90/0x90 [ 3094.093312] ? hrtimer_interrupt+0x128/0x2c0 [ 3094.095221] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3094.114910] ? apic_timer_interrupt+0xf/0x20 [ 3094.117576] [ 3094.118529] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3094.121003] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3094.131385] ? do_raw_spin_lock+0x177/0x190 [ 3094.133520] ? _raw_spin_lock+0x21/0x30 [ 3094.137937] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3094.140737] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 3094.143050] ? mdc_commit_open+0x91/0xb0 [mdc] [ 3094.155438] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3094.165222] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3094.167851] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 3094.171998] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3094.175564] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3094.177858] ? do_wait_intr+0xf0/0xf0 [ 3094.179824] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3094.182928] ? kthread+0x1d1/0x200 [ 3094.184632] ? set_kthread_struct+0x70/0x70 [ 3094.186814] ? ret_from_fork+0x1f/0x30 [ 3096.058022] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 3096.061086] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3096.081116] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3096.087750] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3096.091450] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3096.092572] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3096.098960] RSP: 0018:ffffaf9fc1543ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3096.101213] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 3096.103112] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 3096.105929] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3096.108166] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 3096.111765] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 3096.114095] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 3096.117260] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3096.119989] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 3096.122056] Call Trace: [ 3096.122631] [ 3096.123221] ? show_regs.cold.9+0x22/0x2f [ 3096.125389] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3096.126636] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3096.128038] ? watchdog+0x40/0x40 [ 3096.129112] ? hrtimer_interrupt+0x128/0x2c0 [ 3096.130159] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3096.131529] ? apic_timer_interrupt+0xf/0x20 [ 3096.133345] [ 3096.133909] ? native_safe_halt+0x2e/0x50 [ 3096.135801] kvm_wait+0xcd/0xe0 [ 3096.137206] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3096.140113] do_raw_spin_lock+0x177/0x190 [ 3096.143001] _raw_spin_lock+0x21/0x30 [ 3096.145516] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3096.147719] ? do_raw_spin_unlock+0x75/0x190 [ 3096.150028] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3096.152398] ? do_wait_intr+0xf0/0xf0 [ 3096.154051] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3096.156638] mdc_close+0x53b/0xd70 [mdc] [ 3096.158576] lmv_close+0x18d/0x490 [lmv] [ 3096.160097] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 3096.162544] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3096.164671] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3096.166627] ll_md_close+0x274/0x800 [lustre] [ 3096.168473] ll_file_release+0x128/0xc50 [lustre] [ 3096.170388] __fput+0x10d/0x440 [ 3096.171510] ____fput+0x12/0x20 [ 3096.173059] task_work_run+0xc6/0x110 [ 3096.174278] exit_to_usermode_loop+0x1dd/0x1f0 [ 3096.175929] do_syscall_64+0x3d6/0x440 [ 3096.177635] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3096.179790] RIP: 0033:0x7ff54af45cd8 [ 3096.181541] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3096.190763] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3096.195026] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 3096.198878] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 3096.202386] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 3096.206121] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 3096.209646] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3097.567077] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1138s! [ 3097.571273] Showing busy workqueues and worker pools: [ 3097.573497] workqueue events: flags=0x0 [ 3097.574785] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3097.577641] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 3097.582015] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3097.585123] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 3097.590181] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3097.593231] pending: vmstat_shepherd [ 3097.595000] workqueue events_long: flags=0x0 [ 3097.596715] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3097.599709] pending: lu_objects_flush [obdclass] [ 3097.605754] workqueue events_unbound: flags=0x2 [ 3097.609096] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3097.611621] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3097.617187] workqueue events_power_efficient: flags=0x80 [ 3097.621933] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3097.624923] pending: neigh_periodic_work [ 3097.627754] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3097.632900] pending: do_cache_clean, check_lifetime [ 3097.636608] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3097.641441] pending: neigh_periodic_work [ 3097.643180] workqueue writeback: flags=0x4a [ 3097.646307] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3097.649377] pending: wb_workfn, wb_workfn [ 3097.651048] workqueue xprtiod: flags=0xa [ 3097.652768] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3097.655467] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 3097.658200] workqueue ipv6_addrconf: flags=0x40008 [ 3097.660407] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3097.662977] pending: addrconf_verify_work [ 3097.665013] workqueue knbd0-recv: flags=0x1a [ 3097.666744] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3097.669621] in-flight: 657:recv_work [ 3097.671376] workqueue ptlrpc_pinger: flags=0x2 [ 3097.673088] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3097.676338] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 3097.678534] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1138s workers=4 idle: 20831 12642 28859 [ 3097.681132] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3087s workers=2 idle: 659 [ 3100.060069] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [ 3100.063792] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3100.096633] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3100.109263] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3100.115315] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3100.119532] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3100.130833] RSP: 0000:ffffaf9fc15234f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3100.134916] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 3100.137734] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 3100.140485] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3100.143530] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 3100.146315] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 3100.149652] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 3100.155909] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3100.159798] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 3100.163069] Call Trace: [ 3100.164021] [ 3100.165792] ? show_regs.cold.9+0x22/0x2f [ 3100.167403] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3100.169080] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3100.171504] ? watchdog+0x40/0x40 [ 3100.173021] ? hrtimer_interrupt+0x128/0x2c0 [ 3100.175298] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3100.177445] ? apic_timer_interrupt+0xf/0x20 [ 3100.179090] [ 3100.179838] ? native_safe_halt+0x2e/0x50 [ 3100.181100] kvm_wait+0xcd/0xe0 [ 3100.182686] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3100.185413] do_raw_spin_lock+0x177/0x190 [ 3100.186819] _raw_spin_lock+0x21/0x30 [ 3100.190162] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3100.192985] ? do_raw_spin_unlock+0x75/0x190 [ 3100.194731] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3100.197655] ? do_wait_intr+0xf0/0xf0 [ 3100.199483] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3100.202109] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3100.205510] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3100.210339] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3100.215545] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3100.221245] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3100.225551] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3100.230529] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3100.234673] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3100.246518] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3100.257886] lmv_intent_open+0x338/0x1150 [lmv] [ 3100.263222] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3100.270506] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3100.276504] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3100.282807] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3100.286778] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3100.291677] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3100.295502] ? do_raw_spin_unlock+0x75/0x190 [ 3100.299466] ? _raw_spin_unlock+0x12/0x30 [ 3100.303518] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3100.306892] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3100.311511] ll_atomic_open+0x36a/0xd70 [lustre] [ 3100.313859] lookup_open+0x356/0xb40 [ 3100.317915] do_last+0x313/0xfc0 [ 3100.319406] ? nd_jump_root+0xe5/0x160 [ 3100.321907] ? path_init+0x437/0x520 [ 3100.323993] path_openat+0xf7/0x500 [ 3100.325906] ? do_raw_spin_unlock+0x75/0x190 [ 3100.327650] do_filp_open+0x99/0x140 [ 3100.329422] ? getname_flags+0x6e/0x330 [ 3100.331244] ? __check_object_size+0xff/0x256 [ 3100.332887] ? do_raw_spin_unlock+0x75/0x190 [ 3100.334482] ? _raw_spin_unlock+0x12/0x30 [ 3100.336898] do_sys_openat2+0x2b4/0x410 [ 3100.338780] do_sys_open+0x73/0xa0 [ 3100.340435] __x64_sys_openat+0x24/0x30 [ 3100.343947] do_syscall_64+0xc1/0x440 [ 3100.347799] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3100.350555] RIP: 0033:0x7ff54af45332 [ 3100.351773] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3100.359893] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3100.365506] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 3100.369610] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 3100.375159] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 3100.378072] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 3100.384411] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 3120.059309] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 3120.064821] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3120.094120] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3120.100896] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3120.106831] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3120.111029] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3120.117256] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3120.134010] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3120.141859] RAX: 00000000000043fd RBX: ffff96848811cae0 RCX: 0000000000000001 [ 3120.146514] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 3120.151413] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 3120.155796] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 3120.158961] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 3120.163400] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 3120.168141] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3120.171083] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 3120.174877] Call Trace: [ 3120.176139] [ 3120.177474] ? show_regs.cold.9+0x22/0x2f [ 3120.179906] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3120.182259] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3120.185120] ? watchdog+0x40/0x40 [ 3120.186238] ? hrtimer_interrupt+0x128/0x2c0 [ 3120.188024] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3120.190109] ? apic_timer_interrupt+0xf/0x20 [ 3120.191962] [ 3120.193047] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3120.195843] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3120.198745] do_raw_spin_lock+0x177/0x190 [ 3120.200153] _raw_spin_lock+0x21/0x30 [ 3120.202191] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3120.206052] process_one_work+0x2c8/0x700 [ 3120.207846] worker_thread+0x296/0x6e0 [ 3120.209631] ? rescuer_thread+0x570/0x570 [ 3120.212066] kthread+0x1d1/0x200 [ 3120.213699] ? set_kthread_struct+0x70/0x70 [ 3120.215992] ret_from_fork+0x1f/0x30 [ 3124.055222] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 3124.058318] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 3124.071937] Modules linked in: lustre(O) [ 3124.075598] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3124.075681] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3124.075688] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3124.075691] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3124.075704] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3124.075708] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3124.075712] RAX: 0000000000005719 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 3124.075714] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 3124.075716] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3124.075718] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 3124.075720] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 3124.075724] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 3124.075726] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3124.075729] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 3124.075737] Call Trace: [ 3124.075743] [ 3124.075749] ? show_regs.cold.9+0x22/0x2f [ 3124.077817] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3124.077887] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3124.077894] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3124.077896] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3124.077910] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3124.077914] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3124.077919] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 3124.077922] RDX: 0000000000000ab1 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 3124.077925] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 3124.077927] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 3124.077930] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3124.077932] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 3124.077936] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3124.077938] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 3124.077946] Call Trace: [ 3124.077951] [ 3124.077957] ? show_regs.cold.9+0x22/0x2f [ 3124.077967] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3124.077974] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3124.077978] ? watchdog+0x40/0x40 [ 3124.077983] ? hrtimer_interrupt+0x128/0x2c0 [ 3124.077987] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3124.077995] ? apic_timer_interrupt+0xf/0x20 [ 3124.078000] [ 3124.078004] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3124.078021] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3124.078027] do_raw_spin_lock+0x177/0x190 [ 3124.078031] _raw_spin_lock+0x21/0x30 [ 3124.078037] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3124.078350] obd_mod_free+0x46/0x140 [ptlrpc] [ 3124.078755] mdc_commit_open+0x91/0xb0 [mdc] [ 3124.078808] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3124.079186] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3124.079387] after_reply+0xe2c/0x1780 [ptlrpc] [ 3124.079713] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3124.080105] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3124.080353] ? do_wait_intr+0xf0/0xf0 [ 3124.080363] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3124.080794] kthread+0x1d1/0x200 [ 3124.080801] ? set_kthread_struct+0x70/0x70 [ 3124.080805] ret_from_fork+0x1f/0x30 [ 3124.335168] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3124.337883] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3124.339968] ? watchdog+0x40/0x40 [ 3124.341609] ? hrtimer_interrupt+0x128/0x2c0 [ 3124.345710] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3124.349274] ? apic_timer_interrupt+0xf/0x20 [ 3124.351315] [ 3124.352374] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3124.354349] do_raw_spin_lock+0x177/0x190 [ 3124.355527] _raw_spin_lock+0x21/0x30 [ 3124.358370] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3124.361673] ? do_raw_spin_unlock+0x75/0x190 [ 3124.363848] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3124.367219] ? do_wait_intr+0xf0/0xf0 [ 3124.369296] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3124.372039] mdc_close+0x53b/0xd70 [mdc] [ 3124.374226] lmv_close+0x18d/0x490 [lmv] [ 3124.377022] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 3124.386521] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3124.395106] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3124.402438] ll_md_close+0x274/0x800 [lustre] [ 3124.410684] ll_file_release+0x128/0xc50 [lustre] [ 3124.418891] __fput+0x10d/0x440 [ 3124.429317] ____fput+0x12/0x20 [ 3124.434137] task_work_run+0xc6/0x110 [ 3124.436473] exit_to_usermode_loop+0x1dd/0x1f0 [ 3124.449831] do_syscall_64+0x3d6/0x440 [ 3124.452493] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3124.455474] RIP: 0033:0x7ff54af45cd8 [ 3124.457860] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3124.476541] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3124.491257] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 3124.501524] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 3124.512090] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 3124.520265] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 3124.529869] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3127.777298] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1168s! [ 3127.784822] Showing busy workqueues and worker pools: [ 3127.789808] workqueue events: flags=0x0 [ 3127.794291] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3127.802601] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 3127.811803] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3127.815945] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 3127.824305] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3127.830359] pending: vmstat_shepherd [ 3127.832961] workqueue events_long: flags=0x0 [ 3127.836882] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3127.843347] pending: lu_objects_flush [obdclass] [ 3127.845980] workqueue events_unbound: flags=0x2 [ 3127.851449] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3127.862859] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3127.875523] workqueue events_power_efficient: flags=0x80 [ 3127.887737] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3127.896481] pending: neigh_periodic_work [ 3127.899756] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3127.918785] pending: do_cache_clean, check_lifetime [ 3127.922825] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3127.934876] pending: neigh_periodic_work [ 3127.947047] workqueue writeback: flags=0x4a [ 3127.947990] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3127.949625] pending: wb_workfn, wb_workfn [ 3127.951535] workqueue xprtiod: flags=0xa [ 3127.953302] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3127.956232] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 3127.960158] workqueue ipv6_addrconf: flags=0x40008 [ 3127.962051] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3127.964503] pending: addrconf_verify_work [ 3127.966404] workqueue knbd0-recv: flags=0x1a [ 3127.968635] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3127.970445] in-flight: 657:recv_work [ 3127.971982] workqueue ptlrpc_pinger: flags=0x2 [ 3127.974259] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3127.977482] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 3127.980346] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1169s workers=4 idle: 20831 12642 28859 [ 3127.995195] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3118s workers=2 idle: 659 [ 3128.060497] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [ 3128.063459] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3128.088477] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3128.093792] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3128.096924] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3128.098439] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3128.104887] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3128.107133] RAX: 0000000000006e2b RBX: ffff96848811cae0 RCX: 0000000000000001 [ 3128.109202] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 3128.111897] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3128.115237] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 3128.118511] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 3128.121045] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 3128.124357] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3128.127117] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 3128.129706] Call Trace: [ 3128.130731] [ 3128.131498] ? show_regs.cold.9+0x22/0x2f [ 3128.132906] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3128.134824] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3128.136668] ? watchdog+0x40/0x40 [ 3128.137821] ? hrtimer_interrupt+0x128/0x2c0 [ 3128.139247] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3128.140848] ? apic_timer_interrupt+0xf/0x20 [ 3128.145792] [ 3128.146980] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3128.151802] do_raw_spin_lock+0x177/0x190 [ 3128.157530] _raw_spin_lock+0x21/0x30 [ 3128.161457] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3128.166866] ? do_raw_spin_unlock+0x75/0x190 [ 3128.168727] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3128.176397] ? do_wait_intr+0xf0/0xf0 [ 3128.177955] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3128.182840] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3128.188819] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3128.193802] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3128.197540] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3128.201506] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3128.204910] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3128.207671] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3128.211032] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3128.215232] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3128.218475] lmv_intent_open+0x338/0x1150 [lmv] [ 3128.220902] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3128.224372] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3128.227977] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3128.230779] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3128.235753] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3128.239633] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3128.244867] ? do_raw_spin_unlock+0x75/0x190 [ 3128.253166] ? _raw_spin_unlock+0x12/0x30 [ 3128.256850] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3128.260903] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3128.263910] ll_atomic_open+0x36a/0xd70 [lustre] [ 3128.265744] lookup_open+0x356/0xb40 [ 3128.268230] do_last+0x313/0xfc0 [ 3128.270367] ? nd_jump_root+0xe5/0x160 [ 3128.272352] ? path_init+0x437/0x520 [ 3128.274778] path_openat+0xf7/0x500 [ 3128.277191] ? do_raw_spin_unlock+0x75/0x190 [ 3128.280898] do_filp_open+0x99/0x140 [ 3128.283363] ? getname_flags+0x6e/0x330 [ 3128.285750] ? __check_object_size+0xff/0x256 [ 3128.288676] ? do_raw_spin_unlock+0x75/0x190 [ 3128.291476] ? _raw_spin_unlock+0x12/0x30 [ 3128.294211] do_sys_openat2+0x2b4/0x410 [ 3128.296711] do_sys_open+0x73/0xa0 [ 3128.298835] __x64_sys_openat+0x24/0x30 [ 3128.301397] do_syscall_64+0xc1/0x440 [ 3128.304640] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3128.307841] RIP: 0033:0x7ff54af45332 [ 3128.310246] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3128.321768] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3128.326681] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 3128.330770] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 3128.335680] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 3128.339895] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 3128.344006] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 3148.059244] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 3148.062746] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3148.134450] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3148.159871] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3148.174773] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3148.187321] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3148.192312] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3148.221295] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3148.230991] RAX: 0000000000003886 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 3148.236391] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 3148.249932] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 3148.258632] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 3148.270138] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 3148.279996] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 3148.291335] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3148.299325] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 3148.309225] Call Trace: [ 3148.313270] [ 3148.319492] ? show_regs.cold.9+0x22/0x2f [ 3148.325222] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3148.333164] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3148.342265] ? watchdog+0x40/0x40 [ 3148.348672] ? hrtimer_interrupt+0x128/0x2c0 [ 3148.357120] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3148.365603] ? apic_timer_interrupt+0xf/0x20 [ 3148.371155] [ 3148.373142] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3148.380658] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3148.386204] do_raw_spin_lock+0x177/0x190 [ 3148.388228] _raw_spin_lock+0x21/0x30 [ 3148.390353] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3148.393863] process_one_work+0x2c8/0x700 [ 3148.396153] worker_thread+0x296/0x6e0 [ 3148.398047] ? rescuer_thread+0x570/0x570 [ 3148.400084] kthread+0x1d1/0x200 [ 3148.401611] ? set_kthread_struct+0x70/0x70 [ 3148.403624] ret_from_fork+0x1f/0x30 [ 3152.055246] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 3152.058055] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 3152.060083] Modules linked in: lustre(O) [ 3152.064729] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3152.064805] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3152.064812] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3152.064815] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3152.064829] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3152.064833] RSP: 0018:ffffaf9fc1543ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3152.064837] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 3152.064839] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 3152.064841] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3152.064843] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 3152.064845] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 3152.064849] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 3152.064851] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3152.064854] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 3152.064866] Call Trace: [ 3152.064871] [ 3152.064876] ? show_regs.cold.9+0x22/0x2f [ 3152.064885] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3152.064891] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3152.064895] ? watchdog+0x40/0x40 [ 3152.064899] ? hrtimer_interrupt+0x128/0x2c0 [ 3152.064902] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3152.064925] ? apic_timer_interrupt+0xf/0x20 [ 3152.064930] [ 3152.064932] ? native_safe_halt+0x2e/0x50 [ 3152.064936] kvm_wait+0xcd/0xe0 [ 3152.064946] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3152.064955] do_raw_spin_lock+0x177/0x190 [ 3152.064959] _raw_spin_lock+0x21/0x30 [ 3152.064965] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3152.101330] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3152.101401] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3152.101409] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3152.101413] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3152.101427] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3152.101431] RSP: 0000:ffffaf9fc1b97bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3152.101435] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 3152.101438] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 3152.101440] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 3152.101443] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 3152.101445] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3152.101448] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 3152.101451] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3152.101453] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 3152.101461] Call Trace: [ 3152.101467] [ 3152.101474] ? show_regs.cold.9+0x22/0x2f [ 3152.101483] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3152.101490] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3152.101494] ? watchdog+0x40/0x40 [ 3152.101498] ? hrtimer_interrupt+0x128/0x2c0 [ 3152.101503] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3152.101509] ? apic_timer_interrupt+0xf/0x20 [ 3152.101514] [ 3152.101517] ? native_safe_halt+0x2e/0x50 [ 3152.101521] kvm_wait+0xcd/0xe0 [ 3152.101529] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3152.101537] do_raw_spin_lock+0x177/0x190 [ 3152.101541] _raw_spin_lock+0x21/0x30 [ 3152.101546] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3152.114846] ? do_raw_spin_unlock+0x75/0x190 [ 3152.114879] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3152.116561] ? do_wait_intr+0xf0/0xf0 [ 3152.116570] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3152.129322] obd_mod_free+0x46/0x140 [ptlrpc] [ 3152.129814] mdc_commit_open+0x91/0xb0 [mdc] [ 3152.129853] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3152.140485] mdc_close+0x53b/0xd70 [mdc] [ 3152.140542] lmv_close+0x18d/0x490 [lmv] [ 3152.140575] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 3152.140767] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3152.143238] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3152.144369] ll_md_close+0x274/0x800 [lustre] [ 3152.144453] ll_file_release+0x128/0xc50 [lustre] [ 3152.144535] __fput+0x10d/0x440 [ 3152.144544] ____fput+0x12/0x20 [ 3152.144549] task_work_run+0xc6/0x110 [ 3152.144556] exit_to_usermode_loop+0x1dd/0x1f0 [ 3152.144563] do_syscall_64+0x3d6/0x440 [ 3152.144568] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3152.144577] RIP: 0033:0x7ff54af45cd8 [ 3152.144584] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3152.144589] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3152.144594] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 3152.144596] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 3152.144599] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 3152.144601] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 3152.144603] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3152.694515] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3152.706314] after_reply+0xe2c/0x1780 [ptlrpc] [ 3152.710794] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3152.716263] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3152.724222] ? do_wait_intr+0xf0/0xf0 [ 3152.726772] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3152.732962] kthread+0x1d1/0x200 [ 3152.734505] ? set_kthread_struct+0x70/0x70 [ 3152.736039] ret_from_fork+0x1f/0x30 [ 3156.060076] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [ 3156.076807] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3156.119390] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3156.125948] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3156.135742] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3156.140053] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3156.146616] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3156.149865] RAX: 0000000000001e4c RBX: ffff96848811cae0 RCX: 0000000000000001 [ 3156.152575] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 3156.155604] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3156.159356] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 3156.163342] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 3156.168139] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 3156.171102] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3156.173703] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 3156.177411] Call Trace: [ 3156.178967] [ 3156.179801] ? show_regs.cold.9+0x22/0x2f [ 3156.181368] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3156.183863] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3156.185437] ? watchdog+0x40/0x40 [ 3156.186994] ? hrtimer_interrupt+0x128/0x2c0 [ 3156.189117] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3156.191179] ? apic_timer_interrupt+0xf/0x20 [ 3156.193110] [ 3156.194071] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3156.196177] do_raw_spin_lock+0x177/0x190 [ 3156.197980] _raw_spin_lock+0x21/0x30 [ 3156.199832] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3156.202838] ? do_raw_spin_unlock+0x75/0x190 [ 3156.204717] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3156.207078] ? do_wait_intr+0xf0/0xf0 [ 3156.208941] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3156.211743] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3156.214216] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3156.217034] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3156.219357] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3156.221894] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3156.223975] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3156.226122] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3156.229061] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3156.232385] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3156.236131] lmv_intent_open+0x338/0x1150 [lmv] [ 3156.240295] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3156.246337] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3156.252202] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3156.256164] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3156.261846] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3156.267601] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3156.270803] ? do_raw_spin_unlock+0x75/0x190 [ 3156.273084] ? _raw_spin_unlock+0x12/0x30 [ 3156.275084] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3156.280260] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3156.282934] ll_atomic_open+0x36a/0xd70 [lustre] [ 3156.285213] lookup_open+0x356/0xb40 [ 3156.288430] do_last+0x313/0xfc0 [ 3156.289691] ? nd_jump_root+0xe5/0x160 [ 3156.297292] ? path_init+0x437/0x520 [ 3156.301242] path_openat+0xf7/0x500 [ 3156.304316] ? do_raw_spin_unlock+0x75/0x190 [ 3156.306750] do_filp_open+0x99/0x140 [ 3156.308308] ? getname_flags+0x6e/0x330 [ 3156.311982] ? __check_object_size+0xff/0x256 [ 3156.314589] ? do_raw_spin_unlock+0x75/0x190 [ 3156.316977] ? _raw_spin_unlock+0x12/0x30 [ 3156.320300] do_sys_openat2+0x2b4/0x410 [ 3156.322359] do_sys_open+0x73/0xa0 [ 3156.323552] __x64_sys_openat+0x24/0x30 [ 3156.325875] do_syscall_64+0xc1/0x440 [ 3156.334556] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3156.338132] RIP: 0033:0x7ff54af45332 [ 3156.339365] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3156.359136] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3156.368084] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 3156.376173] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 3156.386706] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 3156.400152] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 3156.413682] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 3158.496849] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1199s! [ 3158.512416] Showing busy workqueues and worker pools: [ 3158.520907] workqueue events: flags=0x0 [ 3158.523744] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3158.528060] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 3158.535809] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3158.546572] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 3158.567633] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3158.576336] pending: vmstat_shepherd [ 3158.582570] workqueue events_long: flags=0x0 [ 3158.588939] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3158.593337] pending: lu_objects_flush [obdclass] [ 3158.607603] workqueue events_unbound: flags=0x2 [ 3158.612358] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3158.613871] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3158.627471] workqueue events_power_efficient: flags=0x80 [ 3158.633928] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3158.645837] pending: neigh_periodic_work [ 3158.655878] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3158.661487] pending: do_cache_clean, check_lifetime [ 3158.669033] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3158.681480] pending: neigh_periodic_work [ 3158.685738] workqueue writeback: flags=0x4a [ 3158.691816] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3158.696390] pending: wb_workfn, wb_workfn [ 3158.701104] workqueue xprtiod: flags=0xa [ 3158.702752] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3158.712956] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 3158.719495] workqueue ipv6_addrconf: flags=0x40008 [ 3158.721561] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3158.727400] pending: addrconf_verify_work [ 3158.733597] workqueue knbd0-recv: flags=0x1a [ 3158.738470] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3158.744683] in-flight: 657:recv_work [ 3158.752309] workqueue ptlrpc_pinger: flags=0x2 [ 3158.755884] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3158.764565] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 3158.773941] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1199s workers=4 idle: 20831 12642 28859 [ 3158.782829] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3149s workers=2 idle: 659 [ 3176.059530] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 3176.070572] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3176.103098] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3176.117425] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3176.128459] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3176.132584] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3176.141607] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3176.164739] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3176.173588] RAX: 0000000000000201 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 3176.182046] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 3176.191045] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 3176.194774] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 3176.203216] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 3176.209716] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 3176.221426] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3176.225712] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 3176.230421] Call Trace: [ 3176.235621] [ 3176.237387] ? show_regs.cold.9+0x22/0x2f [ 3176.241032] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3176.244407] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3176.246515] ? watchdog+0x40/0x40 [ 3176.248760] ? hrtimer_interrupt+0x128/0x2c0 [ 3176.254151] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3176.259420] ? apic_timer_interrupt+0xf/0x20 [ 3176.262469] [ 3176.263582] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3176.268501] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3176.270330] do_raw_spin_lock+0x177/0x190 [ 3176.275260] _raw_spin_lock+0x21/0x30 [ 3176.279688] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3176.282480] process_one_work+0x2c8/0x700 [ 3176.285802] worker_thread+0x296/0x6e0 [ 3176.288544] ? rescuer_thread+0x570/0x570 [ 3176.290611] kthread+0x1d1/0x200 [ 3176.292553] ? set_kthread_struct+0x70/0x70 [ 3176.298205] ret_from_fork+0x1f/0x30 [ 3180.058341] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 3180.062754] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3180.080967] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3180.085836] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3180.091355] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3180.094991] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3180.114913] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3180.123213] RAX: 0000000000003f13 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 3180.131173] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 3180.136437] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3180.145303] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 3180.152150] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 3180.157003] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 3180.162342] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3180.168777] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 3180.171347] Call Trace: [ 3180.178139] [ 3180.179057] ? show_regs.cold.9+0x22/0x2f [ 3180.181057] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3180.184123] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3180.185601] ? watchdog+0x40/0x40 [ 3180.187642] ? hrtimer_interrupt+0x128/0x2c0 [ 3180.189245] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3180.191300] ? apic_timer_interrupt+0xf/0x20 [ 3180.192892] [ 3180.193808] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3180.195792] do_raw_spin_lock+0x177/0x190 [ 3180.197313] _raw_spin_lock+0x21/0x30 [ 3180.198675] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3180.201230] ? do_raw_spin_unlock+0x75/0x190 [ 3180.202808] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3180.206290] ? do_wait_intr+0xf0/0xf0 [ 3180.207794] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3180.210315] mdc_close+0x53b/0xd70 [mdc] [ 3180.212157] lmv_close+0x18d/0x490 [lmv] [ 3180.213792] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 3180.216285] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3180.218611] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3180.220188] ll_md_close+0x274/0x800 [lustre] [ 3180.222154] ll_file_release+0x128/0xc50 [lustre] [ 3180.223673] __fput+0x10d/0x440 [ 3180.224758] ____fput+0x12/0x20 [ 3180.225918] task_work_run+0xc6/0x110 [ 3180.227083] exit_to_usermode_loop+0x1dd/0x1f0 [ 3180.228752] do_syscall_64+0x3d6/0x440 [ 3180.229935] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3180.231760] RIP: 0033:0x7ff54af45cd8 [ 3180.233300] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3180.241643] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3180.244530] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 3180.247139] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 3180.250017] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 3180.252759] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 3180.255749] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3184.057125] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 3184.060126] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 3184.063505] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) [ 3184.069722] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3184.069803] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3184.069813] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3184.069816] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3184.069834] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3184.069839] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3184.069844] RAX: 000000000000190c RBX: ffff96848811cae0 RCX: 0000000000000001 [ 3184.069846] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 3184.069849] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3184.069851] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 3184.069854] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 3184.069857] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 3184.069860] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3184.069862] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 3184.069872] Call Trace: [ 3184.069878] [ 3184.070855] ? show_regs.cold.9+0x22/0x2f [ 3184.081297] mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3184.081370] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3184.081525] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3184.081528] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3184.081543] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3184.081557] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3184.081561] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 3184.081563] RDX: 0000000000002b11 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 3184.081565] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 3184.081567] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 3184.081569] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3184.081572] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 3184.081575] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3184.081577] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 3184.081584] Call Trace: [ 3184.081589] [ 3184.081596] ? show_regs.cold.9+0x22/0x2f [ 3184.081605] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3184.081612] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3184.081616] ? watchdog+0x40/0x40 [ 3184.081620] ? hrtimer_interrupt+0x128/0x2c0 [ 3184.081624] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3184.081633] ? apic_timer_interrupt+0xf/0x20 [ 3184.081637] [ 3184.081640] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3184.081645] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3184.081650] do_raw_spin_lock+0x177/0x190 [ 3184.081654] _raw_spin_lock+0x21/0x30 [ 3184.081660] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3184.081965] obd_mod_free+0x46/0x140 [ptlrpc] [ 3184.082164] mdc_commit_open+0x91/0xb0 [mdc] [ 3184.082206] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3184.134409] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3184.134429] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3184.134433] ? watchdog+0x40/0x40 [ 3184.134438] ? hrtimer_interrupt+0x128/0x2c0 [ 3184.134442] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3184.134452] ? apic_timer_interrupt+0xf/0x20 [ 3184.134457] [ 3184.134462] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3184.134472] do_raw_spin_lock+0x177/0x190 [ 3184.134478] _raw_spin_lock+0x21/0x30 [ 3184.134483] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3184.134811] ? do_raw_spin_unlock+0x75/0x190 [ 3184.134818] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3184.135078] ? do_wait_intr+0xf0/0xf0 [ 3184.135087] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3184.135328] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3184.135535] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3184.135759] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3184.135874] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3184.135916] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3184.135945] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3184.135973] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3184.136053] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3184.136319] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3184.136350] lmv_intent_open+0x338/0x1150 [lmv] [ 3184.136384] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3184.136469] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3184.152076] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3184.154836] after_reply+0xe2c/0x1780 [ptlrpc] [ 3184.155096] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3184.155292] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3184.156959] ? do_wait_intr+0xf0/0xf0 [ 3184.156971] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3184.157192] kthread+0x1d1/0x200 [ 3184.157199] ? set_kthread_struct+0x70/0x70 [ 3184.157203] ret_from_fork+0x1f/0x30 [ 3184.529887] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3184.532244] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3184.534507] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3184.537177] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3184.540297] ? do_raw_spin_unlock+0x75/0x190 [ 3184.542285] ? _raw_spin_unlock+0x12/0x30 [ 3184.550145] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3184.553458] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3184.558795] ll_atomic_open+0x36a/0xd70 [lustre] [ 3184.566462] lookup_open+0x356/0xb40 [ 3184.568664] do_last+0x313/0xfc0 [ 3184.573049] ? nd_jump_root+0xe5/0x160 [ 3184.577199] ? path_init+0x437/0x520 [ 3184.581103] path_openat+0xf7/0x500 [ 3184.583982] ? do_raw_spin_unlock+0x75/0x190 [ 3184.587249] do_filp_open+0x99/0x140 [ 3184.592042] ? getname_flags+0x6e/0x330 [ 3184.599321] ? __check_object_size+0xff/0x256 [ 3184.607055] ? do_raw_spin_unlock+0x75/0x190 [ 3184.614717] ? _raw_spin_unlock+0x12/0x30 [ 3184.616168] do_sys_openat2+0x2b4/0x410 [ 3184.618313] do_sys_open+0x73/0xa0 [ 3184.619947] __x64_sys_openat+0x24/0x30 [ 3184.622964] do_syscall_64+0xc1/0x440 [ 3184.624573] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3184.626625] RIP: 0033:0x7ff54af45332 [ 3184.627812] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3184.635094] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3184.639215] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 3184.654976] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 3184.659526] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 3184.664277] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 3184.674496] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 3189.215196] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1230s! [ 3189.223192] Showing busy workqueues and worker pools: [ 3189.226268] workqueue events: flags=0x0 [ 3189.231729] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3189.238959] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 3189.248255] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3189.254839] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 3189.264764] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3189.271834] pending: vmstat_shepherd [ 3189.274118] workqueue events_long: flags=0x0 [ 3189.275770] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3189.281867] pending: lu_objects_flush [obdclass] [ 3189.289568] workqueue events_unbound: flags=0x2 [ 3189.291237] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3189.297106] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3189.307025] workqueue events_power_efficient: flags=0x80 [ 3189.308901] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3189.314300] pending: neigh_periodic_work [ 3189.317423] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3189.321206] pending: do_cache_clean, check_lifetime [ 3189.325779] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3189.333995] pending: neigh_periodic_work [ 3189.335430] workqueue writeback: flags=0x4a [ 3189.343819] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3189.352822] pending: wb_workfn, wb_workfn [ 3189.354860] workqueue xprtiod: flags=0xa [ 3189.356600] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3189.362463] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 3189.367207] workqueue ipv6_addrconf: flags=0x40008 [ 3189.369792] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3189.373144] pending: addrconf_verify_work [ 3189.374816] workqueue knbd0-recv: flags=0x1a [ 3189.377231] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3189.380316] in-flight: 657:recv_work [ 3189.382441] workqueue ptlrpc_pinger: flags=0x2 [ 3189.384216] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3189.386620] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 3189.389108] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1230s workers=4 idle: 20831 12642 28859 [ 3189.392205] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3179s workers=2 idle: 659 [ 3204.059085] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 3204.068317] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3204.100174] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3204.103873] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3204.107106] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3204.109912] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3204.113154] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3204.122221] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3204.124668] RAX: 0000000000004e15 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 3204.131091] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 3204.135013] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 3204.137111] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 3204.141907] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 3204.144630] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 3204.149585] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3204.153324] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 3204.157454] Call Trace: [ 3204.158451] [ 3204.160828] ? show_regs.cold.9+0x22/0x2f [ 3204.162493] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3204.166387] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3204.167877] ? watchdog+0x40/0x40 [ 3204.169549] ? hrtimer_interrupt+0x128/0x2c0 [ 3204.172006] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3204.173936] ? apic_timer_interrupt+0xf/0x20 [ 3204.175530] [ 3204.178010] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3204.180267] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3204.184268] do_raw_spin_lock+0x177/0x190 [ 3204.186916] _raw_spin_lock+0x21/0x30 [ 3204.188294] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3204.191648] process_one_work+0x2c8/0x700 [ 3204.193260] worker_thread+0x296/0x6e0 [ 3204.195718] ? rescuer_thread+0x570/0x570 [ 3204.197525] kthread+0x1d1/0x200 [ 3204.200065] ? set_kthread_struct+0x70/0x70 [ 3204.201481] ret_from_fork+0x1f/0x30 [ 3208.058434] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 3208.065342] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3208.115836] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3208.123507] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3208.127513] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3208.130586] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3208.146043] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3208.153329] RAX: 0000000000005583 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 3208.157915] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 3208.168804] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3208.183510] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 3208.197732] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 3208.201893] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 3208.208586] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3208.214414] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 3208.216879] Call Trace: [ 3208.219727] [ 3208.222815] ? show_regs.cold.9+0x22/0x2f [ 3208.225870] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3208.230723] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3208.234882] ? watchdog+0x40/0x40 [ 3208.236550] ? hrtimer_interrupt+0x128/0x2c0 [ 3208.240756] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3208.243558] ? apic_timer_interrupt+0xf/0x20 [ 3208.248076] [ 3208.249328] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3208.251855] do_raw_spin_lock+0x177/0x190 [ 3208.254309] _raw_spin_lock+0x21/0x30 [ 3208.257899] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3208.265401] ? do_raw_spin_unlock+0x75/0x190 [ 3208.268452] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3208.279009] ? do_wait_intr+0xf0/0xf0 [ 3208.285081] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3208.291004] mdc_close+0x53b/0xd70 [mdc] [ 3208.295097] lmv_close+0x18d/0x490 [lmv] [ 3208.299634] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 3208.303651] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3208.307525] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3208.310887] ll_md_close+0x274/0x800 [lustre] [ 3208.316627] ll_file_release+0x128/0xc50 [lustre] [ 3208.323584] __fput+0x10d/0x440 [ 3208.327015] ____fput+0x12/0x20 [ 3208.331099] task_work_run+0xc6/0x110 [ 3208.334986] exit_to_usermode_loop+0x1dd/0x1f0 [ 3208.340465] do_syscall_64+0x3d6/0x440 [ 3208.342516] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3208.350760] RIP: 0033:0x7ff54af45cd8 [ 3208.354088] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3208.375196] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3208.383839] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 3208.392091] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 3208.398911] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 3208.405605] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 3208.414593] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3212.062196] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 3212.080684] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3212.113344] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3212.119342] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3212.123650] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3212.127421] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3212.155398] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 3212.158792] RAX: 00000000000036cb RBX: ffff96848811cae0 RCX: 0000000000000001 [ 3212.165663] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 3212.172761] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3212.183092] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 3212.191362] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 3212.196589] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 3212.205338] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3212.211881] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 3212.218830] Call Trace: [ 3212.220479] [ 3212.221165] ? show_regs.cold.9+0x22/0x2f [ 3212.226593] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3212.229904] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3212.233321] ? watchdog+0x40/0x40 [ 3212.236349] ? hrtimer_interrupt+0x128/0x2c0 [ 3212.240487] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3212.243614] ? apic_timer_interrupt+0xf/0x20 [ 3212.246299] [ 3212.247115] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3212.249880] do_raw_spin_lock+0x177/0x190 [ 3212.253609] _raw_spin_lock+0x21/0x30 [ 3212.254826] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3212.261690] ? do_raw_spin_unlock+0x75/0x190 [ 3212.263408] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3212.264904] ? do_wait_intr+0xf0/0xf0 [ 3212.269677] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3212.276486] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3212.278707] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3212.281936] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3212.284646] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3212.286942] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3212.288950] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3212.290835] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3212.293272] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3212.296661] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3212.298691] lmv_intent_open+0x338/0x1150 [lmv] [ 3212.304768] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3212.309595] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3212.312363] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3212.314495] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3212.317203] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3212.320165] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3212.326840] ? do_raw_spin_unlock+0x75/0x190 [ 3212.328418] ? _raw_spin_unlock+0x12/0x30 [ 3212.334443] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3212.339598] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3212.342407] ll_atomic_open+0x36a/0xd70 [lustre] [ 3212.344730] lookup_open+0x356/0xb40 [ 3212.349790] do_last+0x313/0xfc0 [ 3212.353594] ? nd_jump_root+0xe5/0x160 [ 3212.357015] ? path_init+0x437/0x520 [ 3212.359788] path_openat+0xf7/0x500 [ 3212.362798] ? do_raw_spin_unlock+0x75/0x190 [ 3212.366065] do_filp_open+0x99/0x140 [ 3212.369041] ? getname_flags+0x6e/0x330 [ 3212.371584] ? __check_object_size+0xff/0x256 [ 3212.375251] ? do_raw_spin_unlock+0x75/0x190 [ 3212.377539] ? _raw_spin_unlock+0x12/0x30 [ 3212.380671] do_sys_openat2+0x2b4/0x410 [ 3212.383790] do_sys_open+0x73/0xa0 [ 3212.387184] __x64_sys_openat+0x24/0x30 [ 3212.390038] do_syscall_64+0xc1/0x440 [ 3212.393426] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3212.396897] RIP: 0033:0x7ff54af45332 [ 3212.398607] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3212.422845] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3212.438743] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 3212.443038] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 3212.450710] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 3212.456349] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 3212.463045] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 3216.060589] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [ 3216.072716] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3216.133575] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3216.139700] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3216.144750] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3216.147453] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3216.154688] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3216.159037] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 3216.162117] RDX: 0000000000007ef7 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 3216.165784] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 3216.175502] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 3216.179672] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3216.182491] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 3216.186012] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3216.191526] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 3216.195864] Call Trace: [ 3216.198719] [ 3216.199607] ? show_regs.cold.9+0x22/0x2f [ 3216.206342] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3216.210935] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3216.219140] ? watchdog+0x40/0x40 [ 3216.220993] ? hrtimer_interrupt+0x128/0x2c0 [ 3216.224981] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3216.232672] ? apic_timer_interrupt+0xf/0x20 [ 3216.236004] [ 3216.246982] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3216.254447] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3216.261997] do_raw_spin_lock+0x177/0x190 [ 3216.263400] _raw_spin_lock+0x21/0x30 [ 3216.265535] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3216.267939] obd_mod_free+0x46/0x140 [ptlrpc] [ 3216.277952] mdc_commit_open+0x91/0xb0 [mdc] [ 3216.283523] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3216.286973] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3216.303055] after_reply+0xe2c/0x1780 [ptlrpc] [ 3216.305954] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3216.317423] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3216.324442] ? do_wait_intr+0xf0/0xf0 [ 3216.328828] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3216.334462] kthread+0x1d1/0x200 [ 3216.336880] ? set_kthread_struct+0x70/0x70 [ 3216.347055] ret_from_fork+0x1f/0x30 [ 3219.423274] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1260s! [ 3219.428524] Showing busy workqueues and worker pools: [ 3219.434397] workqueue events: flags=0x0 [ 3219.441110] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3219.444397] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 3219.449893] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3219.452252] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 3219.457348] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3219.460435] pending: vmstat_shepherd [ 3219.462953] workqueue events_long: flags=0x0 [ 3219.466857] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3219.474010] pending: lu_objects_flush [obdclass] [ 3219.477780] workqueue events_unbound: flags=0x2 [ 3219.479445] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3219.490633] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3219.503048] workqueue events_power_efficient: flags=0x80 [ 3219.506736] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3219.521091] pending: neigh_periodic_work [ 3219.522273] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3219.534781] pending: do_cache_clean, check_lifetime [ 3219.541802] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3219.546721] pending: neigh_periodic_work [ 3219.549472] workqueue writeback: flags=0x4a [ 3219.551452] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3219.557194] pending: wb_workfn, wb_workfn [ 3219.558663] workqueue xprtiod: flags=0xa [ 3219.563930] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3219.566300] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 3219.571986] workqueue ipv6_addrconf: flags=0x40008 [ 3219.575901] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3219.578964] pending: addrconf_verify_work [ 3219.580952] workqueue knbd0-recv: flags=0x1a [ 3219.582794] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3219.585568] in-flight: 657:recv_work [ 3219.591618] workqueue ptlrpc_pinger: flags=0x2 [ 3219.593661] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3219.598594] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 3219.602990] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1260s workers=4 idle: 20831 12642 28859 [ 3219.611242] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3209s workers=2 idle: 659 [ 3232.059011] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 3232.061916] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3232.131022] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3232.149113] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3232.156697] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3232.169592] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3232.173242] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3232.202260] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3232.204989] RAX: 0000000000004d78 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 3232.229388] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 3232.238526] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 3232.243485] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 3232.247983] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 3232.254559] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 3232.263446] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3232.267739] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 3232.276229] Call Trace: [ 3232.278837] [ 3232.281270] ? show_regs.cold.9+0x22/0x2f [ 3232.287799] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3232.292167] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3232.294752] ? watchdog+0x40/0x40 [ 3232.297855] ? hrtimer_interrupt+0x128/0x2c0 [ 3232.304472] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3232.306300] ? apic_timer_interrupt+0xf/0x20 [ 3232.313831] [ 3232.315684] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3232.323769] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3232.333196] do_raw_spin_lock+0x177/0x190 [ 3232.341170] _raw_spin_lock+0x21/0x30 [ 3232.348198] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3232.357521] process_one_work+0x2c8/0x700 [ 3232.359429] worker_thread+0x296/0x6e0 [ 3232.362579] ? rescuer_thread+0x570/0x570 [ 3232.364847] kthread+0x1d1/0x200 [ 3232.367714] ? set_kthread_struct+0x70/0x70 [ 3232.369747] ret_from_fork+0x1f/0x30 [ 3236.065330] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 3236.073229] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3236.155787] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3236.166450] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3236.172797] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3236.177292] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3236.203615] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3236.223169] RAX: 0000000000002395 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 3236.240158] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 3236.245136] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3236.255053] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 3236.263310] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 3236.269231] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 3236.278164] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3236.284232] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 3236.309997] Call Trace: [ 3236.310786] [ 3236.321089] ? show_regs.cold.9+0x22/0x2f [ 3236.327461] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3236.340865] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3236.342347] ? watchdog+0x40/0x40 [ 3236.352297] ? hrtimer_interrupt+0x128/0x2c0 [ 3236.359583] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3236.366744] ? apic_timer_interrupt+0xf/0x20 [ 3236.367909] [ 3236.368562] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3236.391803] do_raw_spin_lock+0x177/0x190 [ 3236.406586] _raw_spin_lock+0x21/0x30 [ 3236.417433] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3236.434416] ? do_raw_spin_unlock+0x75/0x190 [ 3236.447104] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3236.458455] ? do_wait_intr+0xf0/0xf0 [ 3236.463591] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3236.481309] mdc_close+0x53b/0xd70 [mdc] [ 3236.487435] lmv_close+0x18d/0x490 [lmv] [ 3236.491970] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 3236.509449] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3236.523162] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3236.536115] ll_md_close+0x274/0x800 [lustre] [ 3236.541108] ll_file_release+0x128/0xc50 [lustre] [ 3236.546390] __fput+0x10d/0x440 [ 3236.557369] ____fput+0x12/0x20 [ 3236.563347] task_work_run+0xc6/0x110 [ 3236.566251] exit_to_usermode_loop+0x1dd/0x1f0 [ 3236.570391] do_syscall_64+0x3d6/0x440 [ 3236.577478] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3236.583425] RIP: 0033:0x7ff54af45cd8 [ 3236.588150] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3236.637211] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3236.652239] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 3236.673851] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 3236.689004] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 3236.713222] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 3236.727671] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3240.068355] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 3240.080088] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3240.140027] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3240.148997] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3240.158573] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3240.163100] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3240.188795] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3240.196570] RAX: 0000000000003b0e RBX: ffff96848811cae0 RCX: 0000000000000001 [ 3240.206984] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 3240.218428] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3240.229093] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 3240.237096] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 3240.248373] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 3240.262720] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3240.272567] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 3240.282290] Call Trace: [ 3240.284020] [ 3240.285747] ? show_regs.cold.9+0x22/0x2f [ 3240.288969] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3240.293669] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3240.297136] ? watchdog+0x40/0x40 [ 3240.302981] ? hrtimer_interrupt+0x128/0x2c0 [ 3240.307415] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3240.315612] ? apic_timer_interrupt+0xf/0x20 [ 3240.317177] [ 3240.317854] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3240.328049] do_raw_spin_lock+0x177/0x190 [ 3240.331511] _raw_spin_lock+0x21/0x30 [ 3240.334065] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3240.338862] ? do_raw_spin_unlock+0x75/0x190 [ 3240.345137] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3240.350619] ? do_wait_intr+0xf0/0xf0 [ 3240.353058] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3240.361418] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3240.367818] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3240.374012] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3240.386178] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3240.394043] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3240.403389] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3240.411822] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3240.417132] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3240.424758] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3240.429496] lmv_intent_open+0x338/0x1150 [lmv] [ 3240.434556] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3240.440926] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3240.447038] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3240.455906] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3240.464143] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3240.473397] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3240.479345] ? do_raw_spin_unlock+0x75/0x190 [ 3240.484679] ? _raw_spin_unlock+0x12/0x30 [ 3240.486920] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3240.495075] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3240.502851] ll_atomic_open+0x36a/0xd70 [lustre] [ 3240.506670] lookup_open+0x356/0xb40 [ 3240.508808] do_last+0x313/0xfc0 [ 3240.511544] ? nd_jump_root+0xe5/0x160 [ 3240.516821] ? path_init+0x437/0x520 [ 3240.518752] path_openat+0xf7/0x500 [ 3240.522673] ? do_raw_spin_unlock+0x75/0x190 [ 3240.528051] do_filp_open+0x99/0x140 [ 3240.534085] ? getname_flags+0x6e/0x330 [ 3240.540819] ? __check_object_size+0xff/0x256 [ 3240.544777] ? do_raw_spin_unlock+0x75/0x190 [ 3240.546251] ? _raw_spin_unlock+0x12/0x30 [ 3240.551571] do_sys_openat2+0x2b4/0x410 [ 3240.554407] do_sys_open+0x73/0xa0 [ 3240.557141] __x64_sys_openat+0x24/0x30 [ 3240.559879] do_syscall_64+0xc1/0x440 [ 3240.562486] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3240.565107] RIP: 0033:0x7ff54af45332 [ 3240.570804] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3240.581603] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3240.586407] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 3240.590512] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 3240.593917] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 3240.597451] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 3240.600487] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 3244.064445] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [ 3244.077001] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3244.121234] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3244.136200] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3244.151579] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3244.161441] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3244.192452] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3244.203343] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 3244.212467] RDX: 00000000000007d3 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 3244.229198] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 3244.238109] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 3244.244107] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3244.248462] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 3244.258071] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3244.264616] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 3244.272405] Call Trace: [ 3244.274434] [ 3244.276617] ? show_regs.cold.9+0x22/0x2f [ 3244.280332] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3244.286266] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3244.293675] ? watchdog+0x40/0x40 [ 3244.299194] ? hrtimer_interrupt+0x128/0x2c0 [ 3244.309460] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3244.313832] ? apic_timer_interrupt+0xf/0x20 [ 3244.325645] [ 3244.326419] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3244.343518] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3244.350678] do_raw_spin_lock+0x177/0x190 [ 3244.364820] _raw_spin_lock+0x21/0x30 [ 3244.370684] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3244.381685] obd_mod_free+0x46/0x140 [ptlrpc] [ 3244.394347] mdc_commit_open+0x91/0xb0 [mdc] [ 3244.400692] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3244.408824] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3244.423137] after_reply+0xe2c/0x1780 [ptlrpc] [ 3244.431556] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3244.446551] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3244.453878] ? do_wait_intr+0xf0/0xf0 [ 3244.455175] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3244.473295] kthread+0x1d1/0x200 [ 3244.482345] ? set_kthread_struct+0x70/0x70 [ 3244.484745] ret_from_fork+0x1f/0x30 [ 3249.631049] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1290s! [ 3249.635403] Showing busy workqueues and worker pools: [ 3249.645976] workqueue events: flags=0x0 [ 3249.649436] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3249.661314] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 3249.664482] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3249.667402] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 3249.679443] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3249.686358] pending: vmstat_shepherd [ 3249.688102] workqueue events_long: flags=0x0 [ 3249.691039] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3249.693274] pending: lu_objects_flush [obdclass] [ 3249.694874] workqueue events_unbound: flags=0x2 [ 3249.697172] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3249.706537] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3249.716142] workqueue events_power_efficient: flags=0x80 [ 3249.718291] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3249.731529] pending: neigh_periodic_work [ 3249.733134] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3249.743115] pending: do_cache_clean, check_lifetime [ 3249.759276] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3249.774505] pending: neigh_periodic_work [ 3249.779249] workqueue writeback: flags=0x4a [ 3249.782700] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3249.788494] pending: wb_workfn, wb_workfn [ 3249.792930] workqueue xprtiod: flags=0xa [ 3249.795257] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3249.798309] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 3249.807291] workqueue ipv6_addrconf: flags=0x40008 [ 3249.809461] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3249.816108] pending: addrconf_verify_work [ 3249.819111] workqueue knbd0-recv: flags=0x1a [ 3249.822535] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3249.825681] in-flight: 657:recv_work [ 3249.827521] workqueue ptlrpc_pinger: flags=0x2 [ 3249.830064] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3249.832314] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 3249.835244] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1290s workers=4 idle: 20831 12642 28859 [ 3249.838666] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3240s workers=2 idle: 659 [ 3260.059027] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 3260.061760] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3260.107712] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3260.117864] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3260.123391] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3260.127343] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3260.129889] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3260.139529] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3260.143483] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 3260.150699] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 3260.158357] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 3260.166712] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 3260.185998] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 3260.188291] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 3260.202129] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3260.208053] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 3260.213334] Call Trace: [ 3260.215153] [ 3260.216506] ? show_regs.cold.9+0x22/0x2f [ 3260.218382] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3260.222383] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3260.224360] ? watchdog+0x40/0x40 [ 3260.226090] ? hrtimer_interrupt+0x128/0x2c0 [ 3260.228980] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3260.231459] ? apic_timer_interrupt+0xf/0x20 [ 3260.234158] [ 3260.235374] ? native_safe_halt+0x2e/0x50 [ 3260.236762] kvm_wait+0xcd/0xe0 [ 3260.238807] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3260.242459] do_raw_spin_lock+0x177/0x190 [ 3260.245505] _raw_spin_lock+0x21/0x30 [ 3260.252844] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3260.259310] process_one_work+0x2c8/0x700 [ 3260.262408] worker_thread+0x296/0x6e0 [ 3260.266461] ? rescuer_thread+0x570/0x570 [ 3260.270771] kthread+0x1d1/0x200 [ 3260.272479] ? set_kthread_struct+0x70/0x70 [ 3260.281053] ret_from_fork+0x1f/0x30 [ 3264.059940] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 3264.073042] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3264.193653] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3264.206181] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3264.218018] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3264.230765] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3264.280233] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3264.290023] RAX: 000000000000744a RBX: ffff96848811cae0 RCX: 0000000000000001 [ 3264.296525] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 3264.304348] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3264.319025] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 3264.336303] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 3264.341589] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 3264.365849] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3264.369559] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 3264.377017] Call Trace: [ 3264.386644] [ 3264.388711] ? show_regs.cold.9+0x22/0x2f [ 3264.400560] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3264.413052] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3264.421840] ? watchdog+0x40/0x40 [ 3264.428487] ? hrtimer_interrupt+0x128/0x2c0 [ 3264.436435] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3264.450469] ? apic_timer_interrupt+0xf/0x20 [ 3264.453111] [ 3264.459036] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3264.462700] do_raw_spin_lock+0x177/0x190 [ 3264.471956] _raw_spin_lock+0x21/0x30 [ 3264.473361] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3264.482693] ? do_raw_spin_unlock+0x75/0x190 [ 3264.487481] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3264.499228] ? do_wait_intr+0xf0/0xf0 [ 3264.502997] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3264.512572] mdc_close+0x53b/0xd70 [mdc] [ 3264.520155] lmv_close+0x18d/0x490 [lmv] [ 3264.525573] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 3264.530744] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3264.536424] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3264.544587] ll_md_close+0x274/0x800 [lustre] [ 3264.549716] ll_file_release+0x128/0xc50 [lustre] [ 3264.554080] __fput+0x10d/0x440 [ 3264.562283] ____fput+0x12/0x20 [ 3264.564755] task_work_run+0xc6/0x110 [ 3264.571064] exit_to_usermode_loop+0x1dd/0x1f0 [ 3264.572661] do_syscall_64+0x3d6/0x440 [ 3264.581192] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3264.586479] RIP: 0033:0x7ff54af45cd8 [ 3264.592588] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3264.612901] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3264.619958] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 3264.630511] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 3264.642498] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 3264.644997] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 3264.658930] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3268.060848] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 3268.065680] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3268.104620] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3268.114824] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3268.123968] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3268.130715] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3268.143554] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 3268.147060] RAX: 0000000000007f8a RBX: ffff96848811cae0 RCX: 0000000000000001 [ 3268.150986] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 3268.154437] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3268.157698] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 3268.161961] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 3268.165907] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 3268.171046] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3268.174864] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 3268.179020] Call Trace: [ 3268.180444] [ 3268.181551] ? show_regs.cold.9+0x22/0x2f [ 3268.183274] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3268.185487] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3268.189702] ? watchdog+0x40/0x40 [ 3268.192489] ? hrtimer_interrupt+0x128/0x2c0 [ 3268.196511] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3268.200309] ? apic_timer_interrupt+0xf/0x20 [ 3268.201789] [ 3268.202441] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3268.205723] do_raw_spin_lock+0x177/0x190 [ 3268.210696] _raw_spin_lock+0x21/0x30 [ 3268.215061] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3268.220613] ? do_raw_spin_unlock+0x75/0x190 [ 3268.226715] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3268.229857] ? do_wait_intr+0xf0/0xf0 [ 3268.238769] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3268.241711] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3268.246461] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3268.251970] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3268.256475] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3268.259375] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3268.260935] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3268.265244] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3268.271418] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3268.281894] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3268.291234] lmv_intent_open+0x338/0x1150 [lmv] [ 3268.299879] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3268.304195] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3268.308990] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3268.311495] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3268.314017] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3268.316865] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3268.319299] ? do_raw_spin_unlock+0x75/0x190 [ 3268.321618] ? _raw_spin_unlock+0x12/0x30 [ 3268.323506] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3268.326017] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3268.330788] ll_atomic_open+0x36a/0xd70 [lustre] [ 3268.335524] lookup_open+0x356/0xb40 [ 3268.338941] do_last+0x313/0xfc0 [ 3268.347923] ? nd_jump_root+0xe5/0x160 [ 3268.351184] ? path_init+0x437/0x520 [ 3268.358825] path_openat+0xf7/0x500 [ 3268.361949] ? do_raw_spin_unlock+0x75/0x190 [ 3268.367291] do_filp_open+0x99/0x140 [ 3268.369268] ? getname_flags+0x6e/0x330 [ 3268.377929] ? __check_object_size+0xff/0x256 [ 3268.381656] ? do_raw_spin_unlock+0x75/0x190 [ 3268.385907] ? _raw_spin_unlock+0x12/0x30 [ 3268.390317] do_sys_openat2+0x2b4/0x410 [ 3268.393464] do_sys_open+0x73/0xa0 [ 3268.395515] __x64_sys_openat+0x24/0x30 [ 3268.398909] do_syscall_64+0xc1/0x440 [ 3268.401758] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3268.407288] RIP: 0033:0x7ff54af45332 [ 3268.410858] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3268.424755] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3268.428490] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 3268.431706] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 3268.435080] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 3268.438817] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 3268.442043] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 3273.877725] rcu: INFO: rcu_sched self-detected stall on CPU [ 3273.887371] rcu: 0-...!: (684352 ticks this GP) idle=98a/1/0x4000000000000002 softirq=125154/125154 fqs=959 [ 3273.901979] (t=1320037 jiffies g=253585 q=3184) [ 3273.907354] rcu: rcu_sched kthread starved for 1314997 jiffies! g253585 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [ 3273.917136] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 3273.924968] rcu: RCU grace-period kthread stack dump: [ 3273.927278] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 3273.934686] Call Trace: [ 3273.936494] __schedule+0x351/0xcb0 [ 3273.940551] schedule+0xc0/0x180 [ 3273.941431] schedule_timeout+0xb4/0x190 [ 3273.942324] ? do_raw_spin_unlock+0x75/0x190 [ 3273.947148] ? __next_timer_interrupt+0x160/0x160 [ 3273.948951] ? prepare_to_swait_event+0xf3/0x140 [ 3273.950930] rcu_gp_kthread+0x8dd/0x15b0 [ 3273.953015] ? force_qs_rnp+0x2c0/0x2c0 [ 3273.957495] kthread+0x1d1/0x200 [ 3273.959377] ? set_kthread_struct+0x70/0x70 [ 3273.962833] ret_from_fork+0x1f/0x30 [ 3273.964514] rcu: Stack dump where RCU GP kthread last ran: [ 3273.967794] Sending NMI from CPU 0 to CPUs 2: [ 3273.978568] NMI backtrace for cpu 2 [ 3273.978575] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3273.978577] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3273.978579] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3273.978582] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3273.978584] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3273.978585] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 [ 3273.978587] RAX: 0000000000007a8f RBX: ffff96848811cae0 RCX: 0000000000000001 [ 3273.978587] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 3273.978589] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 3273.978589] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 3273.978590] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 3273.978591] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 3273.978592] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3273.978593] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 3273.978593] Call Trace: [ 3273.978594] [ 3273.978594] ? show_regs.cold.9+0x22/0x2f [ 3273.978595] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 3273.978595] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 3273.978596] ? nmi_handle+0xbd/0x210 [ 3273.978596] ? default_do_nmi+0x4b/0x1b0 [ 3273.978597] ? do_nmi+0x280/0x380 [ 3273.978597] ? end_repeat_nmi+0x16/0x1f [ 3273.978598] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3273.978598] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3273.978599] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3273.978599] [ 3273.978600] do_raw_spin_lock+0x177/0x190 [ 3273.978600] _raw_spin_lock+0x21/0x30 [ 3273.978601] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3273.978601] process_one_work+0x2c8/0x700 [ 3273.978602] worker_thread+0x296/0x6e0 [ 3273.978603] ? rescuer_thread+0x570/0x570 [ 3273.978608] kthread+0x1d1/0x200 [ 3273.978608] ? set_kthread_struct+0x70/0x70 [ 3273.978609] ret_from_fork+0x1f/0x30 [ 3273.979664] NMI backtrace for cpu 0 [ 3274.144414] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3274.155939] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3274.161303] Call Trace: [ 3274.163834] [ 3274.164783] ? dump_stack+0xbb/0x10e [ 3274.167819] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 3274.173159] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 3274.175772] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 3274.179262] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 3274.181823] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 3274.186875] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 3274.188463] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 3274.191935] ? update_process_times+0x6d/0xc0 [ 3274.198139] ? tick_sched_handle.isra.18+0x27/0x90 [ 3274.202161] ? tick_sched_timer+0x53/0xb0 [ 3274.204296] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3274.209737] ? tick_sched_handle.isra.18+0x90/0x90 [ 3274.214929] ? hrtimer_interrupt+0x128/0x2c0 [ 3274.217935] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3274.223344] ? apic_timer_interrupt+0xf/0x20 [ 3274.225808] [ 3274.227187] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3274.233058] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3274.235828] ? do_raw_spin_lock+0x177/0x190 [ 3274.240785] ? _raw_spin_lock+0x21/0x30 [ 3274.243428] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3274.247215] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 3274.252235] ? mdc_commit_open+0x91/0xb0 [mdc] [ 3274.256327] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3274.260427] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3274.268927] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 3274.281142] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3274.286407] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3274.289967] ? do_wait_intr+0xf0/0xf0 [ 3274.291395] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3274.298210] ? kthread+0x1d1/0x200 [ 3274.300789] ? set_kthread_struct+0x70/0x70 [ 3274.304845] ? ret_from_fork+0x1f/0x30 [ 3280.352684] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1321s! [ 3280.365454] Showing busy workqueues and worker pools: [ 3280.368228] workqueue events: flags=0x0 [ 3280.372500] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3280.376933] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 3280.387399] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3280.391684] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 3280.403864] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3280.410197] pending: vmstat_shepherd [ 3280.412412] workqueue events_long: flags=0x0 [ 3280.416520] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3280.425057] pending: lu_objects_flush [obdclass] [ 3280.429152] workqueue events_unbound: flags=0x2 [ 3280.430586] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3280.433287] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3280.441736] workqueue events_power_efficient: flags=0x80 [ 3280.445015] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3280.454430] pending: neigh_periodic_work [ 3280.459938] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3280.462473] pending: do_cache_clean, check_lifetime [ 3280.468938] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3280.475975] pending: neigh_periodic_work [ 3280.481606] workqueue writeback: flags=0x4a [ 3280.485946] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3280.491359] pending: wb_workfn, wb_workfn [ 3280.493492] workqueue xprtiod: flags=0xa [ 3280.495351] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3280.498851] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 3280.504800] workqueue ipv6_addrconf: flags=0x40008 [ 3280.508585] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3280.513847] pending: addrconf_verify_work [ 3280.523895] workqueue knbd0-recv: flags=0x1a [ 3280.531760] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3280.538680] in-flight: 657:recv_work [ 3280.549944] workqueue ptlrpc_pinger: flags=0x2 [ 3280.554187] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3280.559278] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 3280.564322] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1321s workers=4 idle: 20831 12642 28859 [ 3280.571291] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3270s workers=2 idle: 659 [ 3292.059977] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 3292.064924] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3292.088900] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3292.094699] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3292.100993] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3292.104586] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3292.112958] RSP: 0018:ffffaf9fc1543ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3292.116463] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 3292.119633] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 3292.122928] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3292.131768] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 3292.140699] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 3292.148161] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 3292.158895] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3292.170949] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 3292.176161] Call Trace: [ 3292.179185] [ 3292.180356] ? show_regs.cold.9+0x22/0x2f [ 3292.182330] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3292.184623] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3292.186464] ? watchdog+0x40/0x40 [ 3292.187775] ? hrtimer_interrupt+0x128/0x2c0 [ 3292.189340] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3292.191689] ? apic_timer_interrupt+0xf/0x20 [ 3292.193786] [ 3292.195102] ? native_safe_halt+0x2e/0x50 [ 3292.197298] kvm_wait+0xcd/0xe0 [ 3292.198699] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3292.201262] do_raw_spin_lock+0x177/0x190 [ 3292.203164] _raw_spin_lock+0x21/0x30 [ 3292.204935] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3292.209344] ? do_raw_spin_unlock+0x75/0x190 [ 3292.211187] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3292.217798] ? do_wait_intr+0xf0/0xf0 [ 3292.221495] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3292.248345] mdc_close+0x53b/0xd70 [mdc] [ 3292.259361] lmv_close+0x18d/0x490 [lmv] [ 3292.272376] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 3292.288064] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3292.302076] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3292.318336] ll_md_close+0x274/0x800 [lustre] [ 3292.322214] ll_file_release+0x128/0xc50 [lustre] [ 3292.326741] __fput+0x10d/0x440 [ 3292.329622] ____fput+0x12/0x20 [ 3292.332569] task_work_run+0xc6/0x110 [ 3292.334992] exit_to_usermode_loop+0x1dd/0x1f0 [ 3292.338166] do_syscall_64+0x3d6/0x440 [ 3292.340773] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3292.343606] RIP: 0033:0x7ff54af45cd8 [ 3292.345605] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3292.367245] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3292.394208] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 3292.402154] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 3292.408098] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 3292.413495] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 3292.421158] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3296.060012] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 3296.066424] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3296.109704] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3296.129275] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3296.138364] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3296.147367] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3296.171985] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 3296.179749] RAX: 0000000000007e2e RBX: ffff96848811cae0 RCX: 0000000000000001 [ 3296.194374] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 3296.203784] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3296.213099] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 3296.222808] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 3296.233693] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 3296.244337] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3296.252382] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 3296.269002] Call Trace: [ 3296.270239] [ 3296.280134] ? show_regs.cold.9+0x22/0x2f [ 3296.283710] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3296.295013] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3296.301997] ? watchdog+0x40/0x40 [ 3296.310946] ? hrtimer_interrupt+0x128/0x2c0 [ 3296.314911] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3296.318777] ? apic_timer_interrupt+0xf/0x20 [ 3296.322105] [ 3296.328560] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3296.334598] do_raw_spin_lock+0x177/0x190 [ 3296.339583] _raw_spin_lock+0x21/0x30 [ 3296.342081] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3296.348037] ? do_raw_spin_unlock+0x75/0x190 [ 3296.350633] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3296.358571] ? do_wait_intr+0xf0/0xf0 [ 3296.361092] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3296.367745] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3296.373718] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3296.377416] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3296.387624] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3296.392677] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3296.394920] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3296.401685] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3296.406821] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3296.409940] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3296.412920] lmv_intent_open+0x338/0x1150 [lmv] [ 3296.416780] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3296.421386] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3296.431797] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3296.438821] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3296.447868] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3296.456859] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3296.459913] ? do_raw_spin_unlock+0x75/0x190 [ 3296.462455] ? _raw_spin_unlock+0x12/0x30 [ 3296.466769] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3296.469892] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3296.475701] ll_atomic_open+0x36a/0xd70 [lustre] [ 3296.482587] lookup_open+0x356/0xb40 [ 3296.484722] do_last+0x313/0xfc0 [ 3296.487047] ? nd_jump_root+0xe5/0x160 [ 3296.498271] ? path_init+0x437/0x520 [ 3296.499373] path_openat+0xf7/0x500 [ 3296.509677] ? do_raw_spin_unlock+0x75/0x190 [ 3296.513808] do_filp_open+0x99/0x140 [ 3296.525427] ? getname_flags+0x6e/0x330 [ 3296.529918] ? __check_object_size+0xff/0x256 [ 3296.533918] ? do_raw_spin_unlock+0x75/0x190 [ 3296.539912] ? _raw_spin_unlock+0x12/0x30 [ 3296.545258] do_sys_openat2+0x2b4/0x410 [ 3296.553585] do_sys_open+0x73/0xa0 [ 3296.556868] __x64_sys_openat+0x24/0x30 [ 3296.568770] do_syscall_64+0xc1/0x440 [ 3296.573914] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3296.580304] RIP: 0033:0x7ff54af45332 [ 3296.590752] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3296.617153] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3296.629447] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 3296.639773] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 3296.644359] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 3296.653576] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 3296.668440] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 3300.059711] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 3300.069869] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3300.140134] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3300.151101] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3300.165782] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3300.175177] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3300.176278] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3300.190923] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3300.195116] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 3300.199999] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 3300.203948] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 3300.207179] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 3300.211157] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 3300.214831] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 3300.218317] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3300.221120] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 3300.226203] Call Trace: [ 3300.226882] [ 3300.227621] ? show_regs.cold.9+0x22/0x2f [ 3300.229002] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3300.236117] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3300.239135] ? watchdog+0x40/0x40 [ 3300.242782] ? hrtimer_interrupt+0x128/0x2c0 [ 3300.244907] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3300.247888] ? apic_timer_interrupt+0xf/0x20 [ 3300.254661] [ 3300.255493] ? native_safe_halt+0x2e/0x50 [ 3300.256984] kvm_wait+0xcd/0xe0 [ 3300.263611] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3300.268231] do_raw_spin_lock+0x177/0x190 [ 3300.270331] _raw_spin_lock+0x21/0x30 [ 3300.272318] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3300.278498] process_one_work+0x2c8/0x700 [ 3300.280885] worker_thread+0x296/0x6e0 [ 3300.288584] ? rescuer_thread+0x570/0x570 [ 3300.293047] kthread+0x1d1/0x200 [ 3300.295236] ? set_kthread_struct+0x70/0x70 [ 3300.301132] ret_from_fork+0x1f/0x30 [ 3308.055078] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 3308.059702] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3308.088036] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3308.096872] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3308.102303] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3308.105829] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3308.116954] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3308.120931] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 3308.126842] RDX: 000000000000729b RSI: 0000000000000003 RDI: ffff96848811cae0 [ 3308.130888] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 3308.134472] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 3308.138231] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3308.142869] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 3308.149400] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3308.151835] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 3308.155244] Call Trace: [ 3308.156326] [ 3308.157353] ? show_regs.cold.9+0x22/0x2f [ 3308.159605] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3308.162485] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3308.166503] ? watchdog+0x40/0x40 [ 3308.169641] ? hrtimer_interrupt+0x128/0x2c0 [ 3308.173968] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3308.178333] ? apic_timer_interrupt+0xf/0x20 [ 3308.180674] [ 3308.181674] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3308.183884] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3308.189634] do_raw_spin_lock+0x177/0x190 [ 3308.191834] _raw_spin_lock+0x21/0x30 [ 3308.197646] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3308.204454] obd_mod_free+0x46/0x140 [ptlrpc] [ 3308.207541] mdc_commit_open+0x91/0xb0 [mdc] [ 3308.210834] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3308.213697] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3308.217204] after_reply+0xe2c/0x1780 [ptlrpc] [ 3308.220668] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3308.224575] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3308.227415] ? do_wait_intr+0xf0/0xf0 [ 3308.229727] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3308.232980] kthread+0x1d1/0x200 [ 3308.235399] ? set_kthread_struct+0x70/0x70 [ 3308.237119] ret_from_fork+0x1f/0x30 [ 3311.071262] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1352s! [ 3311.087990] Showing busy workqueues and worker pools: [ 3311.089962] workqueue events: flags=0x0 [ 3311.095754] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3311.100016] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 3311.112050] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3311.117579] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 3311.134620] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3311.142175] pending: vmstat_shepherd [ 3311.145064] workqueue events_long: flags=0x0 [ 3311.149990] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3311.157762] pending: lu_objects_flush [obdclass] [ 3311.163474] workqueue events_unbound: flags=0x2 [ 3311.166300] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3311.172933] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3311.184077] workqueue events_power_efficient: flags=0x80 [ 3311.189797] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3311.196907] pending: neigh_periodic_work [ 3311.200066] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3311.203250] pending: do_cache_clean, check_lifetime [ 3311.205125] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3311.208050] pending: neigh_periodic_work [ 3311.209640] workqueue writeback: flags=0x4a [ 3311.212050] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3311.213971] pending: wb_workfn, wb_workfn [ 3311.215707] workqueue xprtiod: flags=0xa [ 3311.217635] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3311.220256] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 3311.223404] workqueue ipv6_addrconf: flags=0x40008 [ 3311.225113] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3311.228085] pending: addrconf_verify_work [ 3311.232660] workqueue knbd0-recv: flags=0x1a [ 3311.237091] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3311.244826] in-flight: 657:recv_work [ 3311.253782] workqueue ptlrpc_pinger: flags=0x2 [ 3311.261954] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3311.268030] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 3311.276808] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1352s workers=4 idle: 20831 12642 28859 [ 3311.292689] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3301s workers=2 idle: 659 [ 3320.063079] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [ 3320.069219] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3320.114609] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3320.131327] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3320.137681] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3320.140505] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3320.166429] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3320.173015] RAX: 0000000000007688 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 3320.178146] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 3320.183249] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3320.189258] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 3320.196039] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 3320.203667] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 3320.211375] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3320.215612] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 3320.218759] Call Trace: [ 3320.220039] [ 3320.220769] ? show_regs.cold.9+0x22/0x2f [ 3320.224588] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3320.230217] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3320.235781] ? watchdog+0x40/0x40 [ 3320.239555] ? hrtimer_interrupt+0x128/0x2c0 [ 3320.243295] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3320.245388] ? apic_timer_interrupt+0xf/0x20 [ 3320.252090] [ 3320.253504] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3320.261826] do_raw_spin_lock+0x177/0x190 [ 3320.263454] _raw_spin_lock+0x21/0x30 [ 3320.269791] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3320.277451] ? do_raw_spin_unlock+0x75/0x190 [ 3320.279398] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3320.281888] ? do_wait_intr+0xf0/0xf0 [ 3320.283509] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3320.287322] mdc_close+0x53b/0xd70 [mdc] [ 3320.294417] lmv_close+0x18d/0x490 [lmv] [ 3320.295883] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 3320.304398] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3320.312297] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3320.315351] ll_md_close+0x274/0x800 [lustre] [ 3320.317252] ll_file_release+0x128/0xc50 [lustre] [ 3320.319485] __fput+0x10d/0x440 [ 3320.320822] ____fput+0x12/0x20 [ 3320.322522] task_work_run+0xc6/0x110 [ 3320.324594] exit_to_usermode_loop+0x1dd/0x1f0 [ 3320.329572] do_syscall_64+0x3d6/0x440 [ 3320.333001] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3320.338463] RIP: 0033:0x7ff54af45cd8 [ 3320.341649] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3320.353669] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3320.363722] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 3320.377920] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 3320.388915] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 3320.393270] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 3320.397012] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3324.064747] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 3324.070234] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3324.096310] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3324.102242] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3324.109580] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3324.112227] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3324.120484] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3324.126246] RAX: 0000000000007f85 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 3324.130383] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 3324.141161] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3324.149099] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 3324.155283] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 3324.162011] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 3324.167729] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3324.170421] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 3324.175525] Call Trace: [ 3324.176393] [ 3324.177052] ? show_regs.cold.9+0x22/0x2f [ 3324.184467] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3324.188791] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3324.192309] ? watchdog+0x40/0x40 [ 3324.194076] ? hrtimer_interrupt+0x128/0x2c0 [ 3324.196383] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3324.198428] ? apic_timer_interrupt+0xf/0x20 [ 3324.200581] [ 3324.201553] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3324.203611] do_raw_spin_lock+0x177/0x190 [ 3324.208114] _raw_spin_lock+0x21/0x30 [ 3324.210904] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3324.215395] ? do_raw_spin_unlock+0x75/0x190 [ 3324.218544] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3324.223611] ? do_wait_intr+0xf0/0xf0 [ 3324.229603] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3324.233280] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3324.238001] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3324.244468] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3324.250118] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3324.252083] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3324.262338] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3324.263801] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3324.269528] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3324.271713] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3324.275948] lmv_intent_open+0x338/0x1150 [lmv] [ 3324.280695] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3324.283289] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3324.284909] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3324.286733] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3324.288801] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3324.291433] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3324.293206] ? do_raw_spin_unlock+0x75/0x190 [ 3324.294683] ? _raw_spin_unlock+0x12/0x30 [ 3324.296230] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3324.298128] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3324.299709] ll_atomic_open+0x36a/0xd70 [lustre] [ 3324.301583] lookup_open+0x356/0xb40 [ 3324.302947] do_last+0x313/0xfc0 [ 3324.306434] ? nd_jump_root+0xe5/0x160 [ 3324.309444] ? path_init+0x437/0x520 [ 3324.310361] path_openat+0xf7/0x500 [ 3324.312493] ? do_raw_spin_unlock+0x75/0x190 [ 3324.314600] do_filp_open+0x99/0x140 [ 3324.315614] ? getname_flags+0x6e/0x330 [ 3324.324178] ? __check_object_size+0xff/0x256 [ 3324.325350] ? do_raw_spin_unlock+0x75/0x190 [ 3324.326633] ? _raw_spin_unlock+0x12/0x30 [ 3324.333146] do_sys_openat2+0x2b4/0x410 [ 3324.335483] do_sys_open+0x73/0xa0 [ 3324.336373] __x64_sys_openat+0x24/0x30 [ 3324.337391] do_syscall_64+0xc1/0x440 [ 3324.338734] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3324.341774] RIP: 0033:0x7ff54af45332 [ 3324.345484] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3324.361738] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3324.368473] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 3324.371485] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 3324.374691] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 3324.381991] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 3324.386967] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 3328.059055] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 3328.060510] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3328.096716] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3328.099326] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3328.101794] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3328.103598] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3328.104799] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3328.109667] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3328.111692] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 3328.113674] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 3328.115231] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 3328.117022] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 3328.118727] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 3328.120350] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 3328.122135] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3328.123508] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 3328.125122] Call Trace: [ 3328.125743] [ 3328.126208] ? show_regs.cold.9+0x22/0x2f [ 3328.127133] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3328.128297] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3328.129401] ? watchdog+0x40/0x40 [ 3328.130026] ? hrtimer_interrupt+0x128/0x2c0 [ 3328.131103] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3328.132800] ? apic_timer_interrupt+0xf/0x20 [ 3328.135712] [ 3328.136291] ? native_safe_halt+0x2e/0x50 [ 3328.137285] kvm_wait+0xcd/0xe0 [ 3328.137928] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3328.139068] do_raw_spin_lock+0x177/0x190 [ 3328.140111] _raw_spin_lock+0x21/0x30 [ 3328.141184] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3328.142800] process_one_work+0x2c8/0x700 [ 3328.143757] worker_thread+0x296/0x6e0 [ 3328.144626] ? rescuer_thread+0x570/0x570 [ 3328.145387] kthread+0x1d1/0x200 [ 3328.146264] ? set_kthread_struct+0x70/0x70 [ 3328.147489] ret_from_fork+0x1f/0x30 [ 3336.055049] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 3336.056981] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3336.073401] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3336.077418] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3336.080328] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3336.081686] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3336.087617] RSP: 0000:ffffaf9fc1b97bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3336.089824] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 3336.092236] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 3336.094421] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 3336.097895] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 3336.100722] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3336.102818] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 3336.105816] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3336.107618] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 3336.109709] Call Trace: [ 3336.110886] [ 3336.111613] ? show_regs.cold.9+0x22/0x2f [ 3336.113633] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3336.115240] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3336.116608] ? watchdog+0x40/0x40 [ 3336.117612] ? hrtimer_interrupt+0x128/0x2c0 [ 3336.118894] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3336.120794] ? apic_timer_interrupt+0xf/0x20 [ 3336.122328] [ 3336.123240] ? native_safe_halt+0x2e/0x50 [ 3336.125274] kvm_wait+0xcd/0xe0 [ 3336.126403] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3336.128354] do_raw_spin_lock+0x177/0x190 [ 3336.130190] _raw_spin_lock+0x21/0x30 [ 3336.131505] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3336.133653] obd_mod_free+0x46/0x140 [ptlrpc] [ 3336.135456] mdc_commit_open+0x91/0xb0 [mdc] [ 3336.137585] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3336.138992] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3336.144482] after_reply+0xe2c/0x1780 [ptlrpc] [ 3336.146425] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3336.152062] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3336.155798] ? do_wait_intr+0xf0/0xf0 [ 3336.158336] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3336.165892] kthread+0x1d1/0x200 [ 3336.168577] ? set_kthread_struct+0x70/0x70 [ 3336.170829] ret_from_fork+0x1f/0x30 [ 3341.791411] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1382s! [ 3341.800518] Showing busy workqueues and worker pools: [ 3341.807532] workqueue events: flags=0x0 [ 3341.810331] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3341.822399] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 3341.826368] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3341.834709] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 3341.846662] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3341.854251] pending: vmstat_shepherd [ 3341.859664] workqueue events_long: flags=0x0 [ 3341.862322] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3341.868608] pending: lu_objects_flush [obdclass] [ 3341.874579] workqueue events_unbound: flags=0x2 [ 3341.876315] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3341.883473] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3341.893579] workqueue events_power_efficient: flags=0x80 [ 3341.903723] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3341.917929] pending: neigh_periodic_work [ 3341.923958] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3341.934971] pending: do_cache_clean, check_lifetime [ 3341.937817] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3341.945206] pending: neigh_periodic_work [ 3341.950871] workqueue writeback: flags=0x4a [ 3341.956677] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3341.962244] pending: wb_workfn, wb_workfn [ 3341.966995] workqueue xprtiod: flags=0xa [ 3341.975323] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3341.985810] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 3341.993616] workqueue ipv6_addrconf: flags=0x40008 [ 3341.995429] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3342.004334] pending: addrconf_verify_work [ 3342.008252] workqueue knbd0-recv: flags=0x1a [ 3342.012782] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3342.020855] in-flight: 657:recv_work [ 3342.023753] workqueue ptlrpc_pinger: flags=0x2 [ 3342.025982] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3342.031995] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 3342.041328] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1383s workers=4 idle: 20831 12642 28859 [ 3342.050319] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3332s workers=2 idle: 659 [ 3348.058012] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [ 3348.067490] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3348.152520] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3348.165194] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3348.171401] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3348.175426] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3348.191529] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3348.195725] RAX: 0000000000007f8a RBX: ffff96848811cae0 RCX: 0000000000000001 [ 3348.204264] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 3348.208735] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3348.213317] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 3348.218384] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 3348.226482] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 3348.233609] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3348.238966] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 3348.246396] Call Trace: [ 3348.255336] [ 3348.256494] ? show_regs.cold.9+0x22/0x2f [ 3348.260363] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3348.264845] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3348.270072] ? watchdog+0x40/0x40 [ 3348.272384] ? hrtimer_interrupt+0x128/0x2c0 [ 3348.277494] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3348.279919] ? apic_timer_interrupt+0xf/0x20 [ 3348.291887] [ 3348.293932] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3348.299044] do_raw_spin_lock+0x177/0x190 [ 3348.301308] _raw_spin_lock+0x21/0x30 [ 3348.302891] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3348.312615] ? do_raw_spin_unlock+0x75/0x190 [ 3348.317048] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3348.323345] ? do_wait_intr+0xf0/0xf0 [ 3348.326994] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3348.331622] mdc_close+0x53b/0xd70 [mdc] [ 3348.337073] lmv_close+0x18d/0x490 [lmv] [ 3348.339400] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 3348.346000] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3348.349244] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3348.352346] ll_md_close+0x274/0x800 [lustre] [ 3348.354573] ll_file_release+0x128/0xc50 [lustre] [ 3348.357716] __fput+0x10d/0x440 [ 3348.359653] ____fput+0x12/0x20 [ 3348.361408] task_work_run+0xc6/0x110 [ 3348.368696] exit_to_usermode_loop+0x1dd/0x1f0 [ 3348.374268] do_syscall_64+0x3d6/0x440 [ 3348.380217] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3348.382092] RIP: 0033:0x7ff54af45cd8 [ 3348.389501] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3348.409122] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3348.415412] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 3348.421941] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 3348.427750] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 3348.432248] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 3348.440386] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3352.061955] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 3352.070597] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3352.119203] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3352.133947] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3352.145452] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3352.151272] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3352.176481] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3352.180759] RAX: 0000000000007952 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 3352.190409] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 3352.206517] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3352.210414] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 3352.215780] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 3352.232242] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 3352.240409] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3352.243631] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 3352.257019] Call Trace: [ 3352.257707] [ 3352.260893] ? show_regs.cold.9+0x22/0x2f [ 3352.265044] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3352.277466] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3352.284728] ? watchdog+0x40/0x40 [ 3352.294191] ? hrtimer_interrupt+0x128/0x2c0 [ 3352.297861] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3352.304433] ? apic_timer_interrupt+0xf/0x20 [ 3352.306967] [ 3352.308871] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3352.311399] do_raw_spin_lock+0x177/0x190 [ 3352.316264] _raw_spin_lock+0x21/0x30 [ 3352.325362] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3352.336347] ? do_raw_spin_unlock+0x75/0x190 [ 3352.347702] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3352.355010] ? do_wait_intr+0xf0/0xf0 [ 3352.356302] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3352.358680] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3352.366561] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3352.370114] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3352.375341] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3352.380446] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3352.385611] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3352.390426] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3352.394506] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3352.401144] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3352.409976] lmv_intent_open+0x338/0x1150 [lmv] [ 3352.414843] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3352.421798] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3352.423909] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3352.433224] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3352.441651] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3352.455020] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3352.459335] ? do_raw_spin_unlock+0x75/0x190 [ 3352.470420] ? _raw_spin_unlock+0x12/0x30 [ 3352.475280] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3352.477857] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3352.481504] ll_atomic_open+0x36a/0xd70 [lustre] [ 3352.493415] lookup_open+0x356/0xb40 [ 3352.494652] do_last+0x313/0xfc0 [ 3352.495860] ? nd_jump_root+0xe5/0x160 [ 3352.505832] ? path_init+0x437/0x520 [ 3352.509076] path_openat+0xf7/0x500 [ 3352.513867] ? do_raw_spin_unlock+0x75/0x190 [ 3352.519662] do_filp_open+0x99/0x140 [ 3352.524678] ? getname_flags+0x6e/0x330 [ 3352.528388] ? __check_object_size+0xff/0x256 [ 3352.532591] ? do_raw_spin_unlock+0x75/0x190 [ 3352.541422] ? _raw_spin_unlock+0x12/0x30 [ 3352.549798] do_sys_openat2+0x2b4/0x410 [ 3352.557327] do_sys_open+0x73/0xa0 [ 3352.561609] __x64_sys_openat+0x24/0x30 [ 3352.563042] do_syscall_64+0xc1/0x440 [ 3352.581775] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3352.583546] RIP: 0033:0x7ff54af45332 [ 3352.584818] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3352.615654] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3352.623755] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 3352.633477] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 3352.638840] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 3352.643445] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 3352.649199] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 3356.059018] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 3356.062532] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3356.090424] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3356.095502] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3356.107505] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3356.116357] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3356.121599] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3356.142772] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3356.149380] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 3356.152918] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 3356.167678] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 3356.176231] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 3356.190333] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 3356.199575] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 3356.205095] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3356.208700] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 3356.214776] Call Trace: [ 3356.215675] [ 3356.216681] ? show_regs.cold.9+0x22/0x2f [ 3356.218220] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3356.221325] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3356.224327] ? watchdog+0x40/0x40 [ 3356.226926] ? hrtimer_interrupt+0x128/0x2c0 [ 3356.229367] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3356.231345] ? apic_timer_interrupt+0xf/0x20 [ 3356.233222] [ 3356.234503] ? native_safe_halt+0x2e/0x50 [ 3356.236017] kvm_wait+0xcd/0xe0 [ 3356.237378] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3356.240382] do_raw_spin_lock+0x177/0x190 [ 3356.242594] _raw_spin_lock+0x21/0x30 [ 3356.244269] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3356.246495] process_one_work+0x2c8/0x700 [ 3356.247646] worker_thread+0x296/0x6e0 [ 3356.248705] ? rescuer_thread+0x570/0x570 [ 3356.250401] kthread+0x1d1/0x200 [ 3356.251382] ? set_kthread_struct+0x70/0x70 [ 3356.253178] ret_from_fork+0x1f/0x30 [ 3368.055026] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 3368.057961] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3368.077533] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3368.089541] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3368.104230] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3368.109106] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3368.120983] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3368.124570] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 3368.132367] RDX: 00000000000004aa RSI: 0000000000000003 RDI: ffff96848811cae0 [ 3368.142570] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 3368.146007] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 3368.149295] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3368.151710] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 3368.164070] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3368.169641] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 3368.172019] Call Trace: [ 3368.173537] [ 3368.175023] ? show_regs.cold.9+0x22/0x2f [ 3368.177296] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3368.182392] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3368.184062] ? watchdog+0x40/0x40 [ 3368.185240] ? hrtimer_interrupt+0x128/0x2c0 [ 3368.192719] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3368.196066] ? apic_timer_interrupt+0xf/0x20 [ 3368.199050] [ 3368.199978] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3368.206075] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3368.216448] do_raw_spin_lock+0x177/0x190 [ 3368.221442] _raw_spin_lock+0x21/0x30 [ 3368.225312] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3368.228039] obd_mod_free+0x46/0x140 [ptlrpc] [ 3368.232706] mdc_commit_open+0x91/0xb0 [mdc] [ 3368.235840] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3368.237907] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3368.245289] after_reply+0xe2c/0x1780 [ptlrpc] [ 3368.247855] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3368.254025] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3368.257812] ? do_wait_intr+0xf0/0xf0 [ 3368.259737] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3368.265218] kthread+0x1d1/0x200 [ 3368.266541] ? set_kthread_struct+0x70/0x70 [ 3368.270226] ret_from_fork+0x1f/0x30 [ 3372.514118] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1413s! [ 3372.529534] Showing busy workqueues and worker pools: [ 3372.531909] workqueue events: flags=0x0 [ 3372.533379] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3372.541156] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 3372.551419] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3372.556489] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 3372.571027] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3372.577069] pending: vmstat_shepherd [ 3372.581159] workqueue events_long: flags=0x0 [ 3372.585272] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3372.596480] pending: lu_objects_flush [obdclass] [ 3372.608173] workqueue events_unbound: flags=0x2 [ 3372.615490] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3372.625125] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3372.633236] workqueue events_power_efficient: flags=0x80 [ 3372.637709] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3372.642895] pending: neigh_periodic_work [ 3372.649014] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3372.655121] pending: do_cache_clean, check_lifetime [ 3372.656874] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3372.666948] pending: neigh_periodic_work [ 3372.670488] workqueue writeback: flags=0x4a [ 3372.672361] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3372.675387] pending: wb_workfn, wb_workfn [ 3372.684865] workqueue xprtiod: flags=0xa [ 3372.689274] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3372.694028] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 3372.700488] workqueue ipv6_addrconf: flags=0x40008 [ 3372.706202] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3372.711434] pending: addrconf_verify_work [ 3372.715164] workqueue knbd0-recv: flags=0x1a [ 3372.721519] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3372.728306] in-flight: 657:recv_work [ 3372.733407] workqueue ptlrpc_pinger: flags=0x2 [ 3372.738351] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3372.741464] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 3372.744644] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1413s workers=4 idle: 20831 12642 28859 [ 3372.753169] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3362s workers=2 idle: 659 [ 3376.060810] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [ 3376.074809] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3376.138785] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3376.154582] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3376.168212] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3376.175836] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3376.204180] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3376.216876] RAX: 000000000000702f RBX: ffff96848811cae0 RCX: 0000000000000001 [ 3376.226946] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 3376.235994] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3376.240702] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 3376.250048] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 3376.252804] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 3376.263477] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3376.270421] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 3376.282099] Call Trace: [ 3376.283266] [ 3376.291425] ? show_regs.cold.9+0x22/0x2f [ 3376.293288] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3376.301261] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3376.304026] ? watchdog+0x40/0x40 [ 3376.307423] ? hrtimer_interrupt+0x128/0x2c0 [ 3376.314449] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3376.318653] ? apic_timer_interrupt+0xf/0x20 [ 3376.320174] [ 3376.326703] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3376.331868] do_raw_spin_lock+0x177/0x190 [ 3376.334260] _raw_spin_lock+0x21/0x30 [ 3376.336249] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3376.341858] ? do_raw_spin_unlock+0x75/0x190 [ 3376.346169] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3376.348402] ? do_wait_intr+0xf0/0xf0 [ 3376.349444] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3376.351095] mdc_close+0x53b/0xd70 [mdc] [ 3376.357314] lmv_close+0x18d/0x490 [lmv] [ 3376.360637] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 3376.365286] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3376.367952] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3376.371617] ll_md_close+0x274/0x800 [lustre] [ 3376.374136] ll_file_release+0x128/0xc50 [lustre] [ 3376.376394] __fput+0x10d/0x440 [ 3376.377768] ____fput+0x12/0x20 [ 3376.379194] task_work_run+0xc6/0x110 [ 3376.380761] exit_to_usermode_loop+0x1dd/0x1f0 [ 3376.388240] do_syscall_64+0x3d6/0x440 [ 3376.390019] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3376.392623] RIP: 0033:0x7ff54af45cd8 [ 3376.393887] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3376.402134] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3376.405123] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 3376.408063] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 3376.412851] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 3376.416779] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 3376.419735] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3380.060045] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 3380.072662] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3380.093333] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3380.096824] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3380.101253] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3380.103492] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3380.111969] RSP: 0000:ffffaf9fc15234f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3380.115313] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 3380.118783] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 3380.122122] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3380.125549] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 3380.130888] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 3380.136780] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 3380.143824] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3380.148384] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 3380.152810] Call Trace: [ 3380.154295] [ 3380.155359] ? show_regs.cold.9+0x22/0x2f [ 3380.158090] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3380.160004] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3380.165437] ? watchdog+0x40/0x40 [ 3380.166850] ? hrtimer_interrupt+0x128/0x2c0 [ 3380.172554] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3380.175384] ? apic_timer_interrupt+0xf/0x20 [ 3380.180035] [ 3380.183737] ? native_safe_halt+0x2e/0x50 [ 3380.185732] kvm_wait+0xcd/0xe0 [ 3380.187031] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3380.189003] do_raw_spin_lock+0x177/0x190 [ 3380.190763] _raw_spin_lock+0x21/0x30 [ 3380.192257] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3380.194633] ? do_raw_spin_unlock+0x75/0x190 [ 3380.196315] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3380.198657] ? do_wait_intr+0xf0/0xf0 [ 3380.199946] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3380.202239] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3380.204390] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3380.206898] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3380.208898] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3380.211222] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3380.212947] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3380.214858] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3380.216864] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3380.219427] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3380.221856] lmv_intent_open+0x338/0x1150 [lmv] [ 3380.223365] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3380.225579] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3380.227394] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3380.228701] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3380.230056] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3380.231923] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3380.233724] ? do_raw_spin_unlock+0x75/0x190 [ 3380.235177] ? _raw_spin_unlock+0x12/0x30 [ 3380.236837] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3380.238531] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3380.240664] ll_atomic_open+0x36a/0xd70 [lustre] [ 3380.242968] lookup_open+0x356/0xb40 [ 3380.244831] do_last+0x313/0xfc0 [ 3380.246225] ? nd_jump_root+0xe5/0x160 [ 3380.247806] ? path_init+0x437/0x520 [ 3380.249331] path_openat+0xf7/0x500 [ 3380.250892] ? do_raw_spin_unlock+0x75/0x190 [ 3380.252274] do_filp_open+0x99/0x140 [ 3380.254039] ? getname_flags+0x6e/0x330 [ 3380.255332] ? __check_object_size+0xff/0x256 [ 3380.257317] ? do_raw_spin_unlock+0x75/0x190 [ 3380.259554] ? _raw_spin_unlock+0x12/0x30 [ 3380.261096] do_sys_openat2+0x2b4/0x410 [ 3380.262756] do_sys_open+0x73/0xa0 [ 3380.264331] __x64_sys_openat+0x24/0x30 [ 3380.266203] do_syscall_64+0xc1/0x440 [ 3380.267816] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3380.269755] RIP: 0033:0x7ff54af45332 [ 3380.271067] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3380.278875] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3380.281846] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 3380.284357] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 3380.286932] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 3380.289429] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 3380.291842] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 3384.059353] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 3384.077759] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3384.123057] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3384.139797] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3384.154816] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3384.166555] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3384.173147] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3384.191282] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3384.200148] RAX: 0000000000004221 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 3384.208240] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 3384.218965] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 3384.228915] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 3384.232289] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 3384.248650] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 3384.260704] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3384.269429] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 3384.277523] Call Trace: [ 3384.281162] [ 3384.285858] ? show_regs.cold.9+0x22/0x2f [ 3384.292332] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3384.296074] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3384.299670] ? watchdog+0x40/0x40 [ 3384.302000] ? hrtimer_interrupt+0x128/0x2c0 [ 3384.305228] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3384.308566] ? apic_timer_interrupt+0xf/0x20 [ 3384.316111] [ 3384.319106] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3384.330915] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3384.335119] do_raw_spin_lock+0x177/0x190 [ 3384.337832] _raw_spin_lock+0x21/0x30 [ 3384.341056] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3384.356569] process_one_work+0x2c8/0x700 [ 3384.358283] worker_thread+0x296/0x6e0 [ 3384.362025] ? rescuer_thread+0x570/0x570 [ 3384.365592] kthread+0x1d1/0x200 [ 3384.371328] ? set_kthread_struct+0x70/0x70 [ 3384.373163] ret_from_fork+0x1f/0x30 [ 3400.055197] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 3400.058667] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3400.081365] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3400.086527] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3400.090759] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3400.092406] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3400.101818] RSP: 0000:ffffaf9fc1b97bb8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 3400.105528] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 3400.108531] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 3400.111790] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 3400.115043] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 3400.120698] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3400.128468] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 3400.134537] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3400.141548] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 3400.150945] Call Trace: [ 3400.154526] [ 3400.158064] ? show_regs.cold.9+0x22/0x2f [ 3400.163186] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3400.167849] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3400.174920] ? watchdog+0x40/0x40 [ 3400.179668] ? hrtimer_interrupt+0x128/0x2c0 [ 3400.183746] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3400.189978] ? apic_timer_interrupt+0xf/0x20 [ 3400.196478] [ 3400.199487] ? native_safe_halt+0x2e/0x50 [ 3400.206337] kvm_wait+0xcd/0xe0 [ 3400.207684] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3400.215372] do_raw_spin_lock+0x177/0x190 [ 3400.220269] _raw_spin_lock+0x21/0x30 [ 3400.224211] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3400.229617] obd_mod_free+0x46/0x140 [ptlrpc] [ 3400.235522] mdc_commit_open+0x91/0xb0 [mdc] [ 3400.238425] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3400.241980] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3400.245907] after_reply+0xe2c/0x1780 [ptlrpc] [ 3400.248706] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3400.253926] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3400.256681] ? do_wait_intr+0xf0/0xf0 [ 3400.258783] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3400.261820] kthread+0x1d1/0x200 [ 3400.264438] ? set_kthread_struct+0x70/0x70 [ 3400.267513] ret_from_fork+0x1f/0x30 [ 3403.233986] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1444s! [ 3403.239747] Showing busy workqueues and worker pools: [ 3403.242837] workqueue events: flags=0x0 [ 3403.244849] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3403.249109] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 3403.254187] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3403.256940] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 3403.271019] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3403.275264] pending: vmstat_shepherd [ 3403.277997] workqueue events_long: flags=0x0 [ 3403.280559] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3403.286044] pending: lu_objects_flush [obdclass] [ 3403.291964] workqueue events_unbound: flags=0x2 [ 3403.298263] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3403.309022] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3403.315193] workqueue events_power_efficient: flags=0x80 [ 3403.318094] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3403.321898] pending: neigh_periodic_work [ 3403.324096] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3403.328038] pending: do_cache_clean, check_lifetime [ 3403.330914] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3403.334343] pending: neigh_periodic_work [ 3403.336969] workqueue writeback: flags=0x4a [ 3403.338522] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3403.341879] pending: wb_workfn, wb_workfn [ 3403.344855] workqueue xprtiod: flags=0xa [ 3403.346879] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3403.350162] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 3403.355911] workqueue ipv6_addrconf: flags=0x40008 [ 3403.359843] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3403.363680] pending: addrconf_verify_work [ 3403.366437] workqueue knbd0-recv: flags=0x1a [ 3403.368529] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3403.372486] in-flight: 657:recv_work [ 3403.374839] workqueue ptlrpc_pinger: flags=0x2 [ 3403.377553] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3403.381520] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 3403.388196] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1444s workers=4 idle: 20831 12642 28859 [ 3403.397825] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3393s workers=2 idle: 659 [ 3404.067443] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [ 3404.083358] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3404.133911] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3404.145152] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3404.161548] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3404.175295] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3404.190993] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 3404.196879] RAX: 0000000000007c11 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 3404.201367] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 3404.204474] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3404.208639] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 3404.212478] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 3404.216419] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 3404.220624] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3404.224533] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 3404.229234] Call Trace: [ 3404.230659] [ 3404.233611] ? show_regs.cold.9+0x22/0x2f [ 3404.237425] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3404.241526] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3404.243614] ? watchdog+0x40/0x40 [ 3404.245078] ? hrtimer_interrupt+0x128/0x2c0 [ 3404.246402] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3404.253487] ? apic_timer_interrupt+0xf/0x20 [ 3404.256439] [ 3404.260522] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3404.268456] do_raw_spin_lock+0x177/0x190 [ 3404.272431] _raw_spin_lock+0x21/0x30 [ 3404.275139] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3404.278547] ? do_raw_spin_unlock+0x75/0x190 [ 3404.281308] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3404.284491] ? do_wait_intr+0xf0/0xf0 [ 3404.286471] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3404.289960] mdc_close+0x53b/0xd70 [mdc] [ 3404.292223] lmv_close+0x18d/0x490 [lmv] [ 3404.294450] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 3404.299906] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3404.303946] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3404.306643] ll_md_close+0x274/0x800 [lustre] [ 3404.315296] ll_file_release+0x128/0xc50 [lustre] [ 3404.317110] __fput+0x10d/0x440 [ 3404.323699] ____fput+0x12/0x20 [ 3404.325680] task_work_run+0xc6/0x110 [ 3404.332149] exit_to_usermode_loop+0x1dd/0x1f0 [ 3404.336713] do_syscall_64+0x3d6/0x440 [ 3404.341908] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3404.348507] RIP: 0033:0x7ff54af45cd8 [ 3404.357293] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3404.371157] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3404.379212] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 3404.382164] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 3404.385518] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 3404.389310] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 3404.392767] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3408.060130] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [ 3408.064601] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3408.099705] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3408.106648] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3408.112781] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3408.115175] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3408.125968] RSP: 0000:ffffaf9fc15234f0 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 3408.130574] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 3408.134821] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 3408.138521] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3408.142499] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 3408.146767] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 3408.150585] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 3408.155726] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3408.162472] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 3408.167028] Call Trace: [ 3408.168430] [ 3408.169705] ? show_regs.cold.9+0x22/0x2f [ 3408.172092] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3408.176098] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3408.179155] ? watchdog+0x40/0x40 [ 3408.181298] ? hrtimer_interrupt+0x128/0x2c0 [ 3408.186157] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3408.190517] ? apic_timer_interrupt+0xf/0x20 [ 3408.196291] [ 3408.198112] ? native_safe_halt+0x2e/0x50 [ 3408.201419] kvm_wait+0xcd/0xe0 [ 3408.203740] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3408.207328] do_raw_spin_lock+0x177/0x190 [ 3408.210176] _raw_spin_lock+0x21/0x30 [ 3408.212612] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3408.216505] ? do_raw_spin_unlock+0x75/0x190 [ 3408.220054] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3408.225165] ? do_wait_intr+0xf0/0xf0 [ 3408.231438] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3408.237030] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3408.240765] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3408.245224] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3408.249389] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3408.252608] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3408.255921] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3408.259517] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3408.262734] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3408.271343] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3408.274493] lmv_intent_open+0x338/0x1150 [lmv] [ 3408.277525] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3408.280564] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3408.284087] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3408.286624] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3408.289640] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3408.292773] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3408.296077] ? do_raw_spin_unlock+0x75/0x190 [ 3408.298782] ? _raw_spin_unlock+0x12/0x30 [ 3408.305665] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3408.308487] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3408.311544] ll_atomic_open+0x36a/0xd70 [lustre] [ 3408.314626] lookup_open+0x356/0xb40 [ 3408.316671] do_last+0x313/0xfc0 [ 3408.318591] ? nd_jump_root+0xe5/0x160 [ 3408.320664] ? path_init+0x437/0x520 [ 3408.322533] path_openat+0xf7/0x500 [ 3408.324725] ? do_raw_spin_unlock+0x75/0x190 [ 3408.326696] do_filp_open+0x99/0x140 [ 3408.328670] ? getname_flags+0x6e/0x330 [ 3408.330706] ? __check_object_size+0xff/0x256 [ 3408.333499] ? do_raw_spin_unlock+0x75/0x190 [ 3408.336152] ? _raw_spin_unlock+0x12/0x30 [ 3408.338120] do_sys_openat2+0x2b4/0x410 [ 3408.340414] do_sys_open+0x73/0xa0 [ 3408.342402] __x64_sys_openat+0x24/0x30 [ 3408.344152] do_syscall_64+0xc1/0x440 [ 3408.346430] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3408.349514] RIP: 0033:0x7ff54af45332 [ 3408.351598] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3408.361671] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3408.366692] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 3408.370553] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 3408.375551] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 3408.381426] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 3408.388065] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 3412.068398] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 3412.074078] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3412.125328] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3412.140379] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3412.146611] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3412.152427] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3412.162449] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3412.180480] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3412.197377] RAX: 0000000000007d7d RBX: ffff96848811cae0 RCX: 0000000000000001 [ 3412.209573] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 3412.218522] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 3412.229361] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 3412.243094] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 3412.246672] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 3412.259604] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3412.266081] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 3412.273446] Call Trace: [ 3412.281534] [ 3412.284714] ? show_regs.cold.9+0x22/0x2f [ 3412.290425] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3412.293508] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3412.296845] ? watchdog+0x40/0x40 [ 3412.300542] ? hrtimer_interrupt+0x128/0x2c0 [ 3412.309594] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3412.322499] ? apic_timer_interrupt+0xf/0x20 [ 3412.330472] [ 3412.332172] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3412.337888] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3412.342914] do_raw_spin_lock+0x177/0x190 [ 3412.346410] _raw_spin_lock+0x21/0x30 [ 3412.348751] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3412.356670] process_one_work+0x2c8/0x700 [ 3412.360051] worker_thread+0x296/0x6e0 [ 3412.362087] ? rescuer_thread+0x570/0x570 [ 3412.365957] kthread+0x1d1/0x200 [ 3412.369061] ? set_kthread_struct+0x70/0x70 [ 3412.371104] ret_from_fork+0x1f/0x30 [ 3428.057607] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 3428.061560] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3428.107996] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3428.117755] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3428.128133] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3428.133411] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3428.153816] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3428.171014] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 3428.184321] RDX: 00000000000020cb RSI: 0000000000000003 RDI: ffff96848811cae0 [ 3428.194083] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 3428.206206] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 3428.217303] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3428.227239] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 3428.239453] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3428.246785] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 3428.254902] Call Trace: [ 3428.255916] [ 3428.260510] ? show_regs.cold.9+0x22/0x2f [ 3428.263219] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3428.264938] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3428.271897] ? watchdog+0x40/0x40 [ 3428.273048] ? hrtimer_interrupt+0x128/0x2c0 [ 3428.277891] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3428.279624] ? apic_timer_interrupt+0xf/0x20 [ 3428.281750] [ 3428.282854] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3428.287490] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3428.290305] do_raw_spin_lock+0x177/0x190 [ 3428.293254] _raw_spin_lock+0x21/0x30 [ 3428.295158] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3428.298630] obd_mod_free+0x46/0x140 [ptlrpc] [ 3428.309338] mdc_commit_open+0x91/0xb0 [mdc] [ 3428.314371] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3428.317033] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3428.319020] after_reply+0xe2c/0x1780 [ptlrpc] [ 3428.330649] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3428.334442] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3428.338256] ? do_wait_intr+0xf0/0xf0 [ 3428.340219] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3428.343987] kthread+0x1d1/0x200 [ 3428.345488] ? set_kthread_struct+0x70/0x70 [ 3428.348192] ret_from_fork+0x1f/0x30 [ 3432.059090] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 3432.064365] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3432.124365] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3432.137626] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3432.143906] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3432.156319] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3432.183580] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3432.194708] RAX: 000000000000614b RBX: ffff96848811cae0 RCX: 0000000000000001 [ 3432.198218] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 3432.204808] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3432.219038] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 3432.225771] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 3432.229849] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 3432.234672] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3432.239335] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 3432.241966] Call Trace: [ 3432.242844] [ 3432.245349] ? show_regs.cold.9+0x22/0x2f [ 3432.252360] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3432.258823] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3432.260412] ? watchdog+0x40/0x40 [ 3432.262686] ? hrtimer_interrupt+0x128/0x2c0 [ 3432.264150] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3432.267505] ? apic_timer_interrupt+0xf/0x20 [ 3432.269096] [ 3432.269832] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3432.278215] do_raw_spin_lock+0x177/0x190 [ 3432.280128] _raw_spin_lock+0x21/0x30 [ 3432.281318] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3432.283944] ? do_raw_spin_unlock+0x75/0x190 [ 3432.285553] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3432.293773] ? do_wait_intr+0xf0/0xf0 [ 3432.297362] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3432.303895] mdc_close+0x53b/0xd70 [mdc] [ 3432.309944] lmv_close+0x18d/0x490 [lmv] [ 3432.316755] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 3432.330358] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3432.339855] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3432.345848] ll_md_close+0x274/0x800 [lustre] [ 3432.351538] ll_file_release+0x128/0xc50 [lustre] [ 3432.361192] __fput+0x10d/0x440 [ 3432.363366] ____fput+0x12/0x20 [ 3432.370505] task_work_run+0xc6/0x110 [ 3432.375880] exit_to_usermode_loop+0x1dd/0x1f0 [ 3432.377465] do_syscall_64+0x3d6/0x440 [ 3432.381888] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3432.389983] RIP: 0033:0x7ff54af45cd8 [ 3432.395586] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3432.414067] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3432.421326] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 3432.428793] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 3432.434261] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 3432.439561] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 3432.442760] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3433.439043] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1474s! [ 3433.441996] Showing busy workqueues and worker pools: [ 3433.443892] workqueue events: flags=0x0 [ 3433.445402] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3433.448885] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 3433.452495] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3433.455265] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 3433.460023] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3433.463513] pending: vmstat_shepherd [ 3433.465565] workqueue events_long: flags=0x0 [ 3433.467491] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3433.470665] pending: lu_objects_flush [obdclass] [ 3433.473229] workqueue events_unbound: flags=0x2 [ 3433.475090] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3433.477601] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3433.482829] workqueue events_power_efficient: flags=0x80 [ 3433.485051] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3433.488190] pending: neigh_periodic_work [ 3433.489914] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3433.492843] pending: do_cache_clean, check_lifetime [ 3433.498314] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3433.501860] pending: neigh_periodic_work [ 3433.504140] workqueue writeback: flags=0x4a [ 3433.506260] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3433.516801] pending: wb_workfn, wb_workfn [ 3433.520056] workqueue xprtiod: flags=0xa [ 3433.522321] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3433.530224] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 3433.539121] workqueue ipv6_addrconf: flags=0x40008 [ 3433.541017] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3433.550115] pending: addrconf_verify_work [ 3433.552555] workqueue knbd0-recv: flags=0x1a [ 3433.554944] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3433.558865] in-flight: 657:recv_work [ 3433.561523] workqueue ptlrpc_pinger: flags=0x2 [ 3433.564341] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3433.570251] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 3433.574497] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1474s workers=4 idle: 20831 12642 28859 [ 3433.578767] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3423s workers=2 idle: 659 [ 3436.063675] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [ 3436.081084] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3436.151342] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3436.169652] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3436.175317] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3436.181828] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3436.201785] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3436.216332] RAX: 0000000000003d88 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 3436.223931] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 3436.236464] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3436.243870] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 3436.248603] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 3436.255915] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 3436.259617] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3436.262467] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 3436.277728] Call Trace: [ 3436.284737] [ 3436.285567] ? show_regs.cold.9+0x22/0x2f [ 3436.288367] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3436.292749] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3436.298738] ? watchdog+0x40/0x40 [ 3436.299925] ? hrtimer_interrupt+0x128/0x2c0 [ 3436.305090] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3436.307084] ? apic_timer_interrupt+0xf/0x20 [ 3436.311703] [ 3436.313703] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3436.319018] do_raw_spin_lock+0x177/0x190 [ 3436.322391] _raw_spin_lock+0x21/0x30 [ 3436.325085] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3436.329935] ? do_raw_spin_unlock+0x75/0x190 [ 3436.331362] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3436.335198] ? do_wait_intr+0xf0/0xf0 [ 3436.337921] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3436.341906] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3436.346061] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3436.349412] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3436.353226] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3436.356606] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3436.359845] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3436.363412] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3436.367620] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3436.372021] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3436.378539] lmv_intent_open+0x338/0x1150 [lmv] [ 3436.382333] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3436.386221] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3436.393890] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3436.399371] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3436.403634] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3436.408365] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3436.412723] ? do_raw_spin_unlock+0x75/0x190 [ 3436.418731] ? _raw_spin_unlock+0x12/0x30 [ 3436.423761] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3436.427914] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3436.431242] ll_atomic_open+0x36a/0xd70 [lustre] [ 3436.435656] lookup_open+0x356/0xb40 [ 3436.438789] do_last+0x313/0xfc0 [ 3436.441672] ? nd_jump_root+0xe5/0x160 [ 3436.444393] ? path_init+0x437/0x520 [ 3436.445448] path_openat+0xf7/0x500 [ 3436.446545] ? do_raw_spin_unlock+0x75/0x190 [ 3436.450194] do_filp_open+0x99/0x140 [ 3436.451897] ? getname_flags+0x6e/0x330 [ 3436.455847] ? __check_object_size+0xff/0x256 [ 3436.458234] ? do_raw_spin_unlock+0x75/0x190 [ 3436.466247] ? _raw_spin_unlock+0x12/0x30 [ 3436.469269] do_sys_openat2+0x2b4/0x410 [ 3436.474927] do_sys_open+0x73/0xa0 [ 3436.476553] __x64_sys_openat+0x24/0x30 [ 3436.480775] do_syscall_64+0xc1/0x440 [ 3436.482560] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3436.490318] RIP: 0033:0x7ff54af45332 [ 3436.492611] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3436.506692] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3436.509422] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 3436.514524] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 3436.518235] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 3436.522656] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 3436.529872] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 3440.059465] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 3440.067930] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3440.134886] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3440.144853] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3440.148614] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3440.152127] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3440.154322] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3440.163093] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3440.170183] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 3440.180348] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 3440.193843] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 3440.197148] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 3440.212291] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 3440.224175] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 3440.234108] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3440.243320] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 3440.246330] Call Trace: [ 3440.248730] [ 3440.252177] ? show_regs.cold.9+0x22/0x2f [ 3440.259691] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3440.264309] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3440.266076] ? watchdog+0x40/0x40 [ 3440.269766] ? hrtimer_interrupt+0x128/0x2c0 [ 3440.275627] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3440.280459] ? apic_timer_interrupt+0xf/0x20 [ 3440.282637] [ 3440.285867] ? native_safe_halt+0x2e/0x50 [ 3440.286732] kvm_wait+0xcd/0xe0 [ 3440.288565] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3440.291081] do_raw_spin_lock+0x177/0x190 [ 3440.292955] _raw_spin_lock+0x21/0x30 [ 3440.294252] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3440.296416] process_one_work+0x2c8/0x700 [ 3440.298287] worker_thread+0x296/0x6e0 [ 3440.299706] ? rescuer_thread+0x570/0x570 [ 3440.301477] kthread+0x1d1/0x200 [ 3440.302725] ? set_kthread_struct+0x70/0x70 [ 3440.304553] ret_from_fork+0x1f/0x30 [ 3453.880010] rcu: INFO: rcu_sched self-detected stall on CPU [ 3453.883155] rcu: 0-...!: (784953 ticks this GP) idle=98a/1/0x4000000000000002 softirq=125154/125154 fqs=959 [ 3453.888021] (t=1500040 jiffies g=253585 q=3230) [ 3453.890053] rcu: rcu_sched kthread starved for 1495000 jiffies! g253585 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=3 [ 3453.895033] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 3453.899803] rcu: RCU grace-period kthread stack dump: [ 3453.902563] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 3453.907146] Call Trace: [ 3453.908542] __schedule+0x351/0xcb0 [ 3453.910181] schedule+0xc0/0x180 [ 3453.911595] schedule_timeout+0xb4/0x190 [ 3453.914239] ? do_raw_spin_unlock+0x75/0x190 [ 3453.916224] ? __next_timer_interrupt+0x160/0x160 [ 3453.918562] ? prepare_to_swait_event+0xf3/0x140 [ 3453.920729] rcu_gp_kthread+0x8dd/0x15b0 [ 3453.922308] ? force_qs_rnp+0x2c0/0x2c0 [ 3453.924058] kthread+0x1d1/0x200 [ 3453.925545] ? set_kthread_struct+0x70/0x70 [ 3453.927681] ret_from_fork+0x1f/0x30 [ 3453.929689] rcu: Stack dump where RCU GP kthread last ran: [ 3453.932224] Sending NMI from CPU 0 to CPUs 3: [ 3453.936057] NMI backtrace for cpu 3 [ 3453.936063] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3453.936065] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3453.936067] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3453.936068] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3453.936070] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 [ 3453.936073] RAX: 000000000000522a RBX: ffff96848811cae0 RCX: 0000000000000001 [ 3453.936075] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 3453.936076] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3453.936077] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 3453.936079] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 3453.936080] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 3453.936081] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3453.936083] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 3453.936084] Call Trace: [ 3453.936085] [ 3453.936086] ? show_regs.cold.9+0x22/0x2f [ 3453.936091] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 3453.936092] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 3453.936093] ? nmi_handle+0xbd/0x210 [ 3453.936094] ? default_do_nmi+0x4b/0x1b0 [ 3453.936095] ? do_nmi+0x280/0x380 [ 3453.936096] ? end_repeat_nmi+0x16/0x1f [ 3453.936097] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3453.936099] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3453.936100] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3453.936101] [ 3453.936102] do_raw_spin_lock+0x177/0x190 [ 3453.936103] _raw_spin_lock+0x21/0x30 [ 3453.936104] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3453.936106] ? do_raw_spin_unlock+0x75/0x190 [ 3453.936107] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3453.936108] ? do_wait_intr+0xf0/0xf0 [ 3453.936112] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3453.936114] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3453.936114] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3453.936116] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3453.936117] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3453.936118] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3453.936119] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3453.936120] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3453.936121] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3453.936122] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3453.936123] lmv_intent_open+0x338/0x1150 [lmv] [ 3453.936124] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3453.936125] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3453.936126] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3453.936128] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3453.936128] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3453.936130] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3453.936134] ? do_raw_spin_unlock+0x75/0x190 [ 3453.936135] ? _raw_spin_unlock+0x12/0x30 [ 3453.936136] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3453.936137] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3453.936138] ll_atomic_open+0x36a/0xd70 [lustre] [ 3453.936140] lookup_open+0x356/0xb40 [ 3453.936141] do_last+0x313/0xfc0 [ 3453.936142] ? nd_jump_root+0xe5/0x160 [ 3453.936143] ? path_init+0x437/0x520 [ 3453.936144] path_openat+0xf7/0x500 [ 3453.936145] ? do_raw_spin_unlock+0x75/0x190 [ 3453.936146] do_filp_open+0x99/0x140 [ 3453.936147] ? getname_flags+0x6e/0x330 [ 3453.936148] ? __check_object_size+0xff/0x256 [ 3453.936149] ? do_raw_spin_unlock+0x75/0x190 [ 3453.936149] ? _raw_spin_unlock+0x12/0x30 [ 3453.936150] do_sys_openat2+0x2b4/0x410 [ 3453.936151] do_sys_open+0x73/0xa0 [ 3453.936156] __x64_sys_openat+0x24/0x30 [ 3453.936157] do_syscall_64+0xc1/0x440 [ 3453.936158] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3453.936159] RIP: 0033:0x7ff54af45332 [ 3453.936160] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3453.936162] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3453.936165] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 3453.936166] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 3453.936167] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 3453.936169] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 3453.936170] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 3453.936967] NMI backtrace for cpu 0 [ 3454.137311] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3454.143556] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3454.147488] Call Trace: [ 3454.148710] [ 3454.149552] ? dump_stack+0xbb/0x10e [ 3454.151045] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 3454.153088] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 3454.155322] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 3454.157395] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 3454.159704] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 3454.162613] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 3454.164660] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 3454.166862] ? update_process_times+0x6d/0xc0 [ 3454.168724] ? tick_sched_handle.isra.18+0x27/0x90 [ 3454.171194] ? tick_sched_timer+0x53/0xb0 [ 3454.173461] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3454.175384] ? tick_sched_handle.isra.18+0x90/0x90 [ 3454.177373] ? hrtimer_interrupt+0x128/0x2c0 [ 3454.178847] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3454.181545] ? apic_timer_interrupt+0xf/0x20 [ 3454.183477] [ 3454.184395] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3454.187000] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3454.189733] ? do_raw_spin_lock+0x177/0x190 [ 3454.191181] ? _raw_spin_lock+0x21/0x30 [ 3454.197058] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3454.201258] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 3454.203397] ? mdc_commit_open+0x91/0xb0 [mdc] [ 3454.205472] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3454.210279] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3454.214350] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 3454.216745] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3454.219964] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3454.225744] ? do_wait_intr+0xf0/0xf0 [ 3454.228814] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3454.232155] ? kthread+0x1d1/0x200 [ 3454.234191] ? set_kthread_struct+0x70/0x70 [ 3454.235713] ? ret_from_fork+0x1f/0x30 [ 3460.060513] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 3460.070132] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3460.118214] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3460.130615] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3460.141177] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3460.147567] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3460.160846] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3460.165823] RAX: 0000000000007a87 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 3460.174941] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 3460.183978] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3460.189020] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 3460.193967] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 3460.201369] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 3460.207187] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3460.208856] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 3460.222346] Call Trace: [ 3460.223989] [ 3460.224980] ? show_regs.cold.9+0x22/0x2f [ 3460.228183] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3460.236082] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3460.239983] ? watchdog+0x40/0x40 [ 3460.242206] ? hrtimer_interrupt+0x128/0x2c0 [ 3460.252665] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3460.267129] ? apic_timer_interrupt+0xf/0x20 [ 3460.271623] [ 3460.272430] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3460.278400] do_raw_spin_lock+0x177/0x190 [ 3460.287131] _raw_spin_lock+0x21/0x30 [ 3460.294056] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3460.301780] ? do_raw_spin_unlock+0x75/0x190 [ 3460.306231] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3460.312182] ? do_wait_intr+0xf0/0xf0 [ 3460.314910] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3460.322166] mdc_close+0x53b/0xd70 [mdc] [ 3460.329047] lmv_close+0x18d/0x490 [lmv] [ 3460.334855] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 3460.343106] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3460.345287] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3460.348683] ll_md_close+0x274/0x800 [lustre] [ 3460.352602] ll_file_release+0x128/0xc50 [lustre] [ 3460.361894] __fput+0x10d/0x440 [ 3460.364132] ____fput+0x12/0x20 [ 3460.366769] task_work_run+0xc6/0x110 [ 3460.370628] exit_to_usermode_loop+0x1dd/0x1f0 [ 3460.376370] do_syscall_64+0x3d6/0x440 [ 3460.379817] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3460.383875] RIP: 0033:0x7ff54af45cd8 [ 3460.386619] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3460.405824] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3460.413136] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 3460.420310] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 3460.425844] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 3460.434135] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 3460.440675] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3463.647029] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1504s! [ 3463.652565] Showing busy workqueues and worker pools: [ 3463.654964] workqueue events: flags=0x0 [ 3463.657468] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3463.662916] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 3463.667576] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3463.671345] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 3463.677445] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3463.680471] pending: vmstat_shepherd [ 3463.683017] workqueue events_long: flags=0x0 [ 3463.685192] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3463.688397] pending: lu_objects_flush [obdclass] [ 3463.691375] workqueue events_unbound: flags=0x2 [ 3463.693572] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3463.697026] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3463.702273] workqueue events_power_efficient: flags=0x80 [ 3463.704396] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3463.709382] pending: neigh_periodic_work [ 3463.713419] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3463.722005] pending: do_cache_clean, check_lifetime [ 3463.727364] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3463.734615] pending: neigh_periodic_work [ 3463.739350] workqueue writeback: flags=0x4a [ 3463.742514] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3463.749383] pending: wb_workfn, wb_workfn [ 3463.753487] workqueue xprtiod: flags=0xa [ 3463.759402] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3463.762318] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 3463.765742] workqueue ipv6_addrconf: flags=0x40008 [ 3463.767927] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3463.770385] pending: addrconf_verify_work [ 3463.773214] workqueue knbd0-recv: flags=0x1a [ 3463.774612] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3463.777532] in-flight: 657:recv_work [ 3463.779269] workqueue ptlrpc_pinger: flags=0x2 [ 3463.781251] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3463.785267] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 3463.787946] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1504s workers=4 idle: 20831 12642 28859 [ 3463.791467] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3454s workers=2 idle: 659 [ 3468.059222] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 3468.063269] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3468.077548] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3468.082936] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3468.086310] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3468.088987] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3468.091181] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3468.095977] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 3468.098564] RAX: 0000000000006cc6 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 3468.101462] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 3468.104229] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 3468.106129] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 3468.107570] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 3468.109432] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 3468.112030] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3468.113557] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 3468.115471] Call Trace: [ 3468.116139] [ 3468.116503] ? show_regs.cold.9+0x22/0x2f [ 3468.117475] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3468.118545] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3468.119570] ? watchdog+0x40/0x40 [ 3468.120436] ? hrtimer_interrupt+0x128/0x2c0 [ 3468.121447] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3468.122501] ? apic_timer_interrupt+0xf/0x20 [ 3468.123506] [ 3468.124397] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3468.126591] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3468.129030] do_raw_spin_lock+0x177/0x190 [ 3468.131024] _raw_spin_lock+0x21/0x30 [ 3468.132524] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3468.135119] process_one_work+0x2c8/0x700 [ 3468.136576] worker_thread+0x296/0x6e0 [ 3468.138158] ? rescuer_thread+0x570/0x570 [ 3468.141377] kthread+0x1d1/0x200 [ 3468.143076] ? set_kthread_struct+0x70/0x70 [ 3468.145115] ret_from_fork+0x1f/0x30 [ 3480.060026] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [ 3480.064459] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3480.093293] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3480.098831] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3480.103549] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3480.105603] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3480.115072] RSP: 0000:ffffaf9fc15234f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3480.118957] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 3480.122453] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 3480.125410] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3480.128808] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 3480.132524] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 3480.136093] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 3480.139908] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3480.142786] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 3480.146186] Call Trace: [ 3480.147374] [ 3480.148228] ? show_regs.cold.9+0x22/0x2f [ 3480.150338] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3480.152326] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3480.154270] ? watchdog+0x40/0x40 [ 3480.156087] ? hrtimer_interrupt+0x128/0x2c0 [ 3480.158310] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3480.160492] ? apic_timer_interrupt+0xf/0x20 [ 3480.162692] [ 3480.163648] ? native_safe_halt+0x2e/0x50 [ 3480.165391] kvm_wait+0xcd/0xe0 [ 3480.166914] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3480.169434] do_raw_spin_lock+0x177/0x190 [ 3480.171167] _raw_spin_lock+0x21/0x30 [ 3480.172829] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3480.176080] ? do_raw_spin_unlock+0x75/0x190 [ 3480.178829] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3480.181780] ? do_wait_intr+0xf0/0xf0 [ 3480.183721] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3480.186279] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3480.189124] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3480.191901] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3480.194634] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3480.197298] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3480.199391] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3480.201839] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3480.204737] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3480.207706] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3480.210178] lmv_intent_open+0x338/0x1150 [lmv] [ 3480.212382] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3480.214518] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3480.217072] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3480.218688] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3480.220875] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3480.223456] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3480.225773] ? do_raw_spin_unlock+0x75/0x190 [ 3480.227979] ? _raw_spin_unlock+0x12/0x30 [ 3480.229766] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3480.232482] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3480.235308] ll_atomic_open+0x36a/0xd70 [lustre] [ 3480.237835] lookup_open+0x356/0xb40 [ 3480.239776] do_last+0x313/0xfc0 [ 3480.241458] ? nd_jump_root+0xe5/0x160 [ 3480.243553] ? path_init+0x437/0x520 [ 3480.245347] path_openat+0xf7/0x500 [ 3480.247196] ? do_raw_spin_unlock+0x75/0x190 [ 3480.249428] do_filp_open+0x99/0x140 [ 3480.251406] ? getname_flags+0x6e/0x330 [ 3480.253638] ? __check_object_size+0xff/0x256 [ 3480.256767] ? do_raw_spin_unlock+0x75/0x190 [ 3480.258876] ? _raw_spin_unlock+0x12/0x30 [ 3480.261024] do_sys_openat2+0x2b4/0x410 [ 3480.263328] do_sys_open+0x73/0xa0 [ 3480.265148] __x64_sys_openat+0x24/0x30 [ 3480.267277] do_syscall_64+0xc1/0x440 [ 3480.269322] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3480.272193] RIP: 0033:0x7ff54af45332 [ 3480.273961] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3480.283729] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3480.287602] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 3480.291008] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 3480.294735] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 3480.298326] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 3480.301766] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 3488.055332] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 3488.058427] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 3488.058585] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3488.058718] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3488.058723] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3488.058726] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3488.058760] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3488.058763] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3488.058767] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 3488.058769] RDX: 0000000000003cd5 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 3488.058772] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 3488.058774] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 3488.058775] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3488.058778] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 3488.058789] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3488.058791] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 3488.058797] Call Trace: [ 3488.058808] [ 3488.058815] ? show_regs.cold.9+0x22/0x2f [ 3488.058823] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3488.058828] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3488.058832] ? watchdog+0x40/0x40 [ 3488.058835] ? hrtimer_interrupt+0x128/0x2c0 [ 3488.058838] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3488.058846] ? apic_timer_interrupt+0xf/0x20 [ 3488.058850] [ 3488.058853] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3488.058857] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3488.058861] do_raw_spin_lock+0x177/0x190 [ 3488.058865] _raw_spin_lock+0x21/0x30 [ 3488.058870] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3488.059154] obd_mod_free+0x46/0x140 [ptlrpc] [ 3488.061858] mdc_commit_open+0x91/0xb0 [mdc] [ 3488.061894] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3488.082400] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3488.082535] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3488.082542] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3488.082545] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3488.082559] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3488.123928] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3488.138999] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 3488.139024] RAX: 0000000000005a0a RBX: ffff96848811cae0 RCX: 0000000000000001 [ 3488.139026] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 3488.139028] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3488.139030] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 3488.139032] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 3488.139034] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 3488.139037] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3488.139039] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 3488.139047] Call Trace: [ 3488.139055] [ 3488.139061] ? show_regs.cold.9+0x22/0x2f [ 3488.139076] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3488.139082] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3488.139086] ? watchdog+0x40/0x40 [ 3488.139091] ? hrtimer_interrupt+0x128/0x2c0 [ 3488.139094] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3488.139103] ? apic_timer_interrupt+0xf/0x20 [ 3488.139109] [ 3488.139112] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3488.139125] do_raw_spin_lock+0x177/0x190 [ 3488.139130] _raw_spin_lock+0x21/0x30 [ 3488.139134] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3488.139997] ? do_raw_spin_unlock+0x75/0x190 [ 3488.140006] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3488.140248] ? do_wait_intr+0xf0/0xf0 [ 3488.140256] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3488.140473] mdc_close+0x53b/0xd70 [mdc] [ 3488.140517] lmv_close+0x18d/0x490 [lmv] [ 3488.140962] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 3488.154744] after_reply+0xe2c/0x1780 [ptlrpc] [ 3488.159120] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3488.159253] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3488.159335] ll_md_close+0x274/0x800 [lustre] [ 3488.159711] ll_file_release+0x128/0xc50 [lustre] [ 3488.159794] __fput+0x10d/0x440 [ 3488.159805] ____fput+0x12/0x20 [ 3488.159808] task_work_run+0xc6/0x110 [ 3488.159936] exit_to_usermode_loop+0x1dd/0x1f0 [ 3488.159944] do_syscall_64+0x3d6/0x440 [ 3488.159949] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3488.159957] RIP: 0033:0x7ff54af45cd8 [ 3488.159964] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3488.159968] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3488.159972] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 3488.160045] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 3488.160047] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 3488.160049] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 3488.160052] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3488.547193] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3488.550227] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3488.554004] ? do_wait_intr+0xf0/0xf0 [ 3488.556478] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3488.559571] kthread+0x1d1/0x200 [ 3488.560778] ? set_kthread_struct+0x70/0x70 [ 3488.561754] ret_from_fork+0x1f/0x30 [ 3493.855392] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1534s! [ 3493.869343] Showing busy workqueues and worker pools: [ 3493.873461] workqueue events: flags=0x0 [ 3493.875469] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3493.877449] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 3493.883279] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3493.886431] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 3493.891879] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3493.897950] pending: vmstat_shepherd [ 3493.904199] workqueue events_long: flags=0x0 [ 3493.908200] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3493.913846] pending: lu_objects_flush [obdclass] [ 3493.917232] workqueue events_unbound: flags=0x2 [ 3493.921323] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3493.925103] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3493.938214] workqueue events_power_efficient: flags=0x80 [ 3493.942917] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3493.946149] pending: neigh_periodic_work [ 3493.948174] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3493.951666] pending: do_cache_clean, check_lifetime [ 3493.955812] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3493.963018] pending: neigh_periodic_work [ 3493.966759] workqueue writeback: flags=0x4a [ 3493.968208] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3493.971375] pending: wb_workfn, wb_workfn [ 3493.976296] workqueue xprtiod: flags=0xa [ 3493.977743] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3493.979508] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 3493.987578] workqueue ipv6_addrconf: flags=0x40008 [ 3493.990918] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3493.992647] pending: addrconf_verify_work [ 3493.993840] workqueue knbd0-recv: flags=0x1a [ 3493.995505] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3493.998287] in-flight: 657:recv_work [ 3494.000017] workqueue ptlrpc_pinger: flags=0x2 [ 3494.002148] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3494.004735] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 3494.007416] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1535s workers=4 idle: 20831 12642 28859 [ 3494.010982] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3484s workers=2 idle: 659 [ 3496.059057] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 3496.062949] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3496.124473] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3496.140035] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3496.143570] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3496.151705] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3496.155080] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3496.171511] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3496.177796] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 3496.184841] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 3496.190832] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 3496.194297] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 3496.209819] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 3496.216797] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 3496.226257] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3496.233009] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 3496.240660] Call Trace: [ 3496.244642] [ 3496.248150] ? show_regs.cold.9+0x22/0x2f [ 3496.249679] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3496.255257] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3496.259848] ? watchdog+0x40/0x40 [ 3496.267077] ? hrtimer_interrupt+0x128/0x2c0 [ 3496.273365] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3496.275164] ? apic_timer_interrupt+0xf/0x20 [ 3496.280274] [ 3496.281587] ? native_safe_halt+0x2e/0x50 [ 3496.284313] kvm_wait+0xcd/0xe0 [ 3496.287842] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3496.295327] do_raw_spin_lock+0x177/0x190 [ 3496.296838] _raw_spin_lock+0x21/0x30 [ 3496.303129] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3496.311881] process_one_work+0x2c8/0x700 [ 3496.315848] worker_thread+0x296/0x6e0 [ 3496.322685] ? rescuer_thread+0x570/0x570 [ 3496.326047] kthread+0x1d1/0x200 [ 3496.327994] ? set_kthread_struct+0x70/0x70 [ 3496.334806] ret_from_fork+0x1f/0x30 [ 3508.062395] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [ 3508.070461] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3508.128372] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3508.140436] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3508.148041] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3508.153747] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3508.177051] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3508.181916] RAX: 0000000000005985 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 3508.188467] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 3508.193804] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3508.200849] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 3508.205508] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 3508.211086] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 3508.216582] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3508.221333] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 3508.227975] Call Trace: [ 3508.231963] [ 3508.233114] ? show_regs.cold.9+0x22/0x2f [ 3508.235611] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3508.244824] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3508.250769] ? watchdog+0x40/0x40 [ 3508.256478] ? hrtimer_interrupt+0x128/0x2c0 [ 3508.261615] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3508.266169] ? apic_timer_interrupt+0xf/0x20 [ 3508.269212] [ 3508.270844] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3508.278070] do_raw_spin_lock+0x177/0x190 [ 3508.282397] _raw_spin_lock+0x21/0x30 [ 3508.286635] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3508.293846] ? do_raw_spin_unlock+0x75/0x190 [ 3508.296918] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3508.300693] ? do_wait_intr+0xf0/0xf0 [ 3508.302927] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3508.306506] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3508.310029] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3508.315215] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3508.319222] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3508.323450] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3508.326599] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3508.333186] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3508.339990] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3508.344365] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3508.348159] lmv_intent_open+0x338/0x1150 [lmv] [ 3508.351245] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3508.358987] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3508.366015] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3508.368586] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3508.372769] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3508.377721] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3508.380030] ? do_raw_spin_unlock+0x75/0x190 [ 3508.382425] ? _raw_spin_unlock+0x12/0x30 [ 3508.384777] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3508.387309] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3508.389645] ll_atomic_open+0x36a/0xd70 [lustre] [ 3508.391740] lookup_open+0x356/0xb40 [ 3508.393040] do_last+0x313/0xfc0 [ 3508.394874] ? nd_jump_root+0xe5/0x160 [ 3508.396727] ? path_init+0x437/0x520 [ 3508.398431] path_openat+0xf7/0x500 [ 3508.400573] ? do_raw_spin_unlock+0x75/0x190 [ 3508.402611] do_filp_open+0x99/0x140 [ 3508.404224] ? getname_flags+0x6e/0x330 [ 3508.406113] ? __check_object_size+0xff/0x256 [ 3508.408336] ? do_raw_spin_unlock+0x75/0x190 [ 3508.410517] ? _raw_spin_unlock+0x12/0x30 [ 3508.412634] do_sys_openat2+0x2b4/0x410 [ 3508.415953] do_sys_open+0x73/0xa0 [ 3508.417890] __x64_sys_openat+0x24/0x30 [ 3508.420189] do_syscall_64+0xc1/0x440 [ 3508.421888] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3508.424812] RIP: 0033:0x7ff54af45332 [ 3508.426420] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3508.436386] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3508.440708] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 3508.446746] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 3508.451682] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 3508.459630] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 3508.465975] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 3516.058029] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 3516.067961] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3516.105762] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3516.109979] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3516.113622] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3516.118562] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3516.130800] RSP: 0018:ffffaf9fc1543ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3516.136825] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 3516.138839] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 3516.150685] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3516.159848] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 3516.168146] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 3516.175909] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 3516.183466] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3516.186986] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 3516.197303] Call Trace: [ 3516.200178] [ 3516.201704] ? show_regs.cold.9+0x22/0x2f [ 3516.205661] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3516.207954] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3516.215904] ? watchdog+0x40/0x40 [ 3516.217208] ? hrtimer_interrupt+0x128/0x2c0 [ 3516.223673] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3516.229972] ? apic_timer_interrupt+0xf/0x20 [ 3516.232439] [ 3516.238828] ? native_safe_halt+0x2e/0x50 [ 3516.241951] kvm_wait+0xcd/0xe0 [ 3516.247849] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3516.250020] do_raw_spin_lock+0x177/0x190 [ 3516.256572] _raw_spin_lock+0x21/0x30 [ 3516.259519] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3516.269242] ? do_raw_spin_unlock+0x75/0x190 [ 3516.271610] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3516.276809] ? do_wait_intr+0xf0/0xf0 [ 3516.279727] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3516.282740] mdc_close+0x53b/0xd70 [mdc] [ 3516.287652] lmv_close+0x18d/0x490 [lmv] [ 3516.290264] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 3516.295534] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3516.299702] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3516.307273] ll_md_close+0x274/0x800 [lustre] [ 3516.311113] ll_file_release+0x128/0xc50 [lustre] [ 3516.317517] __fput+0x10d/0x440 [ 3516.321796] ____fput+0x12/0x20 [ 3516.325801] task_work_run+0xc6/0x110 [ 3516.328364] exit_to_usermode_loop+0x1dd/0x1f0 [ 3516.330034] do_syscall_64+0x3d6/0x440 [ 3516.333919] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3516.336807] RIP: 0033:0x7ff54af45cd8 [ 3516.339179] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3516.354277] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3516.357812] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 3516.360507] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 3516.362878] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 3516.371103] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 3516.381719] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3520.055024] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [ 3520.063735] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3520.184322] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3520.195917] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3520.207809] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3520.220422] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3520.262739] RSP: 0000:ffffaf9fc1b97bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3520.279063] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 3520.291586] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 3520.302634] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 3520.318772] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 3520.330991] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3520.342179] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 3520.370784] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3520.383657] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 3520.408823] Call Trace: [ 3520.412193] [ 3520.421048] ? show_regs.cold.9+0x22/0x2f [ 3520.425400] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3520.431121] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3520.439148] ? watchdog+0x40/0x40 [ 3520.445925] ? hrtimer_interrupt+0x128/0x2c0 [ 3520.454036] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3520.462753] ? apic_timer_interrupt+0xf/0x20 [ 3520.470527] [ 3520.475262] ? native_safe_halt+0x2e/0x50 [ 3520.479366] kvm_wait+0xcd/0xe0 [ 3520.481424] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3520.489065] do_raw_spin_lock+0x177/0x190 [ 3520.496331] _raw_spin_lock+0x21/0x30 [ 3520.501485] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3520.508845] obd_mod_free+0x46/0x140 [ptlrpc] [ 3520.510642] mdc_commit_open+0x91/0xb0 [mdc] [ 3520.515503] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3520.520307] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3520.527847] after_reply+0xe2c/0x1780 [ptlrpc] [ 3520.531783] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3520.536039] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3520.539462] ? do_wait_intr+0xf0/0xf0 [ 3520.541640] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3520.545433] kthread+0x1d1/0x200 [ 3520.547389] ? set_kthread_struct+0x70/0x70 [ 3520.549797] ret_from_fork+0x1f/0x30 [ 3524.059610] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 3524.063072] BUG: workqueue lockup - pool [ 3524.064819] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) [ 3524.066269] cpus=0-3 [ 3524.066277] lfsck(O) [ 3524.068947] flags=0x4 nice=0 stuck for 1565s! [ 3524.068961] Showing busy workqueues and worker pools: [ 3524.068963] workqueue events: flags=0x0 [ 3524.068966] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3524.068976] pending: ldlm_pools_recalc_task [ptlrpc] [ 3524.069806] obdecho(O) [ 3524.070646] , cpu_latency_work [ptlrpc] [ 3524.070866] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3524.070874] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4] [ 3524.075169] mgc(O) [ 3524.076872] , osc_grant_work_handler [osc], key_garbage_collector [ 3524.076956] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3524.076968] pending: vmstat_shepherd [ 3524.076975] workqueue events_long: flags=0x0 [ 3524.076977] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3524.076985] pending: lu_objects_flush [obdclass] [ 3524.077208] workqueue events_unbound: flags=0x2 [ 3524.077210] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3524.077217] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3524.077238] workqueue events_power_efficient: flags=0x80 [ 3524.077240] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3524.077261] pending: neigh_periodic_work [ 3524.077268] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3524.077275] pending: do_cache_clean, check_lifetime [ 3524.077287] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3524.077292] pending: neigh_periodic_work [ 3524.077303] workqueue writeback: flags=0x4a [ 3524.077305] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3524.077310] pending: wb_workfn, wb_workfn [ 3524.077323] workqueue xprtiod: flags=0xa [ 3524.077324] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3524.077329] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 3524.077345] workqueue ipv6_addrconf: flags=0x40008 [ 3524.077348] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3524.077355] pending: addrconf_verify_work [ 3524.077365] workqueue knbd0-recv: flags=0x1a [ 3524.077367] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3524.077373] in-flight: 657:recv_work [ 3524.077382] workqueue ptlrpc_pinger: flags=0x2 [ 3524.077384] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3524.077388] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 3524.077624] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1565s workers=4 idle: 20831 12642 28859 [ 3524.077633] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3514s workers=2 idle: 659 [ 3524.217947] mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3524.240345] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3524.248934] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3524.258157] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3524.262464] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3524.266344] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3524.275565] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3524.278855] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 3524.282596] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 3524.285453] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 3524.288241] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 3524.291168] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 3524.298566] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 3524.303580] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3524.316343] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 3524.320812] Call Trace: [ 3524.321795] [ 3524.322559] ? show_regs.cold.9+0x22/0x2f [ 3524.327562] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3524.334497] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3524.337017] ? watchdog+0x40/0x40 [ 3524.338179] ? hrtimer_interrupt+0x128/0x2c0 [ 3524.343042] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3524.345916] ? apic_timer_interrupt+0xf/0x20 [ 3524.348141] [ 3524.349068] ? native_safe_halt+0x2e/0x50 [ 3524.351149] kvm_wait+0xcd/0xe0 [ 3524.354538] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3524.360595] do_raw_spin_lock+0x177/0x190 [ 3524.367712] _raw_spin_lock+0x21/0x30 [ 3524.368921] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3524.377357] process_one_work+0x2c8/0x700 [ 3524.389955] worker_thread+0x296/0x6e0 [ 3524.394370] ? rescuer_thread+0x570/0x570 [ 3524.397317] kthread+0x1d1/0x200 [ 3524.398265] ? set_kthread_struct+0x70/0x70 [ 3524.399156] ret_from_fork+0x1f/0x30 [ 3536.060756] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [ 3536.066084] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3536.101335] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3536.111418] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3536.118477] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3536.122441] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3536.133317] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3536.138911] RAX: 0000000000003585 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 3536.143894] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 3536.149793] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3536.158891] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 3536.162240] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 3536.167650] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 3536.172506] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3536.175129] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 3536.179939] Call Trace: [ 3536.180841] [ 3536.181461] ? show_regs.cold.9+0x22/0x2f [ 3536.182788] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3536.188793] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3536.190409] ? watchdog+0x40/0x40 [ 3536.193257] ? hrtimer_interrupt+0x128/0x2c0 [ 3536.195518] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3536.199881] ? apic_timer_interrupt+0xf/0x20 [ 3536.202544] [ 3536.203644] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3536.206197] do_raw_spin_lock+0x177/0x190 [ 3536.208040] _raw_spin_lock+0x21/0x30 [ 3536.209599] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3536.212073] ? do_raw_spin_unlock+0x75/0x190 [ 3536.213479] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3536.215528] ? do_wait_intr+0xf0/0xf0 [ 3536.216825] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3536.219003] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3536.223096] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3536.226073] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3536.228882] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3536.231297] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3536.233191] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3536.235176] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3536.237261] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3536.239686] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3536.242151] lmv_intent_open+0x338/0x1150 [lmv] [ 3536.243955] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3536.246164] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3536.248658] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3536.250495] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3536.252825] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3536.255191] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3536.256947] ? do_raw_spin_unlock+0x75/0x190 [ 3536.258688] ? _raw_spin_unlock+0x12/0x30 [ 3536.260374] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3536.263448] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3536.267059] ll_atomic_open+0x36a/0xd70 [lustre] [ 3536.270027] lookup_open+0x356/0xb40 [ 3536.271481] do_last+0x313/0xfc0 [ 3536.273643] ? nd_jump_root+0xe5/0x160 [ 3536.275241] ? path_init+0x437/0x520 [ 3536.276945] path_openat+0xf7/0x500 [ 3536.278364] ? do_raw_spin_unlock+0x75/0x190 [ 3536.281546] do_filp_open+0x99/0x140 [ 3536.282822] ? getname_flags+0x6e/0x330 [ 3536.284763] ? __check_object_size+0xff/0x256 [ 3536.286903] ? do_raw_spin_unlock+0x75/0x190 [ 3536.290314] ? _raw_spin_unlock+0x12/0x30 [ 3536.294025] do_sys_openat2+0x2b4/0x410 [ 3536.295051] do_sys_open+0x73/0xa0 [ 3536.295801] __x64_sys_openat+0x24/0x30 [ 3536.299802] do_syscall_64+0xc1/0x440 [ 3536.301589] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3536.303774] RIP: 0033:0x7ff54af45332 [ 3536.305207] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3536.318309] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3536.323235] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 3536.328187] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 3536.337561] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 3536.351399] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 3536.355437] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 3544.069104] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 3544.080708] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3544.143303] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3544.151510] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3544.156006] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3544.159318] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3544.174507] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3544.183661] RAX: 000000000000750a RBX: ffff96848811cae0 RCX: 0000000000000001 [ 3544.191842] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 3544.203553] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3544.205988] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 3544.220574] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 3544.228827] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 3544.236450] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3544.242408] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 3544.254521] Call Trace: [ 3544.255841] [ 3544.256984] ? show_regs.cold.9+0x22/0x2f [ 3544.259416] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3544.264661] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3544.271432] ? watchdog+0x40/0x40 [ 3544.274832] ? hrtimer_interrupt+0x128/0x2c0 [ 3544.281139] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3544.282537] ? apic_timer_interrupt+0xf/0x20 [ 3544.283879] [ 3544.289485] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3544.291232] do_raw_spin_lock+0x177/0x190 [ 3544.298326] _raw_spin_lock+0x21/0x30 [ 3544.305648] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3544.312562] ? do_raw_spin_unlock+0x75/0x190 [ 3544.321766] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3544.330636] ? do_wait_intr+0xf0/0xf0 [ 3544.334767] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3544.344858] mdc_close+0x53b/0xd70 [mdc] [ 3544.351323] lmv_close+0x18d/0x490 [lmv] [ 3544.357335] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 3544.368555] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3544.381266] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3544.390386] ll_md_close+0x274/0x800 [lustre] [ 3544.398307] ll_file_release+0x128/0xc50 [lustre] [ 3544.408025] __fput+0x10d/0x440 [ 3544.408985] ____fput+0x12/0x20 [ 3544.416319] task_work_run+0xc6/0x110 [ 3544.424186] exit_to_usermode_loop+0x1dd/0x1f0 [ 3544.430686] do_syscall_64+0x3d6/0x440 [ 3544.434962] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3544.440286] RIP: 0033:0x7ff54af45cd8 [ 3544.444451] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3544.465284] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3544.471564] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 3544.477819] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 3544.483595] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 3544.493320] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 3544.497374] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3552.055259] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [ 3552.059006] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 3552.062793] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3552.063007] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3552.063015] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3552.063018] RIP: 0010:__pv_queued_spin_lock_slowpath+0x17c/0x6d0 [ 3552.063033] Code: 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 48 83 05 5c d6 dc 02 01 <83> ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 75 e6 48 83 05 2c d6 dc [ 3552.063037] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 3552.063042] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 3552.063045] RDX: 0000000000001276 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 3552.063047] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 3552.063049] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 3552.063051] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3552.063054] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 3552.063057] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3552.063059] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 3552.063067] Call Trace: [ 3552.063073] [ 3552.063079] ? show_regs.cold.9+0x22/0x2f [ 3552.063089] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3552.063095] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3552.063099] ? watchdog+0x40/0x40 [ 3552.063104] ? hrtimer_interrupt+0x128/0x2c0 [ 3552.063108] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3552.063117] ? apic_timer_interrupt+0xf/0x20 [ 3552.063122] [ 3552.063125] ? __pv_queued_spin_lock_slowpath+0x17c/0x6d0 [ 3552.063131] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3552.063135] do_raw_spin_lock+0x177/0x190 [ 3552.063140] _raw_spin_lock+0x21/0x30 [ 3552.063145] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3552.070797] Modules linked in: [ 3552.096772] obd_mod_free+0x46/0x140 [ptlrpc] [ 3552.097303] mdc_commit_open+0x91/0xb0 [mdc] [ 3552.097344] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3552.097837] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3552.098048] after_reply+0xe2c/0x1780 [ptlrpc] [ 3552.098247] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3552.099596] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3552.100040] ? do_wait_intr+0xf0/0xf0 [ 3552.100053] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3552.100255] kthread+0x1d1/0x200 [ 3552.100261] ? set_kthread_struct+0x70/0x70 [ 3552.100265] ret_from_fork+0x1f/0x30 [ 3552.238905] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3552.256573] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3552.260838] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3552.264718] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3552.267197] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3552.269248] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3552.277284] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3552.280718] RAX: 0000000000007e99 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 3552.284483] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 3552.288279] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 3552.292501] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 3552.296229] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 3552.300404] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 3552.304354] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3552.307586] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 3552.311474] Call Trace: [ 3552.313065] [ 3552.314566] ? show_regs.cold.9+0x22/0x2f [ 3552.316326] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3552.318969] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3552.322173] ? watchdog+0x40/0x40 [ 3552.323815] ? hrtimer_interrupt+0x128/0x2c0 [ 3552.326037] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3552.327928] ? apic_timer_interrupt+0xf/0x20 [ 3552.330127] [ 3552.331215] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3552.334715] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3552.337370] do_raw_spin_lock+0x177/0x190 [ 3552.340196] _raw_spin_lock+0x21/0x30 [ 3552.342669] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3552.346399] process_one_work+0x2c8/0x700 [ 3552.348262] worker_thread+0x296/0x6e0 [ 3552.353132] ? rescuer_thread+0x570/0x570 [ 3552.355163] kthread+0x1d1/0x200 [ 3552.357075] ? set_kthread_struct+0x70/0x70 [ 3552.359746] ret_from_fork+0x1f/0x30 [ 3554.271072] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1595s! [ 3554.275056] Showing busy workqueues and worker pools: [ 3554.278529] workqueue events: flags=0x0 [ 3554.281288] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3554.289099] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 3554.295231] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3554.299555] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 3554.307671] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3554.314647] pending: vmstat_shepherd [ 3554.317300] workqueue events_long: flags=0x0 [ 3554.319984] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3554.323065] pending: lu_objects_flush [obdclass] [ 3554.325314] workqueue events_unbound: flags=0x2 [ 3554.326771] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3554.328689] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3554.333612] workqueue events_power_efficient: flags=0x80 [ 3554.336627] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3554.340438] pending: neigh_periodic_work [ 3554.342669] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3554.346776] pending: do_cache_clean, check_lifetime [ 3554.349909] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3554.353654] pending: neigh_periodic_work [ 3554.356445] workqueue writeback: flags=0x4a [ 3554.359264] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3554.362671] pending: wb_workfn, wb_workfn [ 3554.365297] workqueue xprtiod: flags=0xa [ 3554.367443] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3554.371073] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 3554.375358] workqueue ipv6_addrconf: flags=0x40008 [ 3554.378059] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3554.380827] pending: addrconf_verify_work [ 3554.383375] workqueue knbd0-recv: flags=0x1a [ 3554.385636] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3554.390062] in-flight: 657:recv_work [ 3554.392337] workqueue ptlrpc_pinger: flags=0x2 [ 3554.394582] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3554.398436] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 3554.406262] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1595s workers=4 idle: 20831 12642 28859 [ 3554.416088] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3544s workers=2 idle: 659 [ 3564.060025] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [ 3564.068025] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3564.098828] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3564.105733] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3564.111156] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3564.114234] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3564.128996] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3564.131504] RAX: 0000000000004a63 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 3564.135208] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 3564.141098] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3564.148051] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 3564.150632] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 3564.155348] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 3564.163210] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3564.169013] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 3564.178114] Call Trace: [ 3564.180154] [ 3564.182008] ? show_regs.cold.9+0x22/0x2f [ 3564.188548] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3564.189682] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3564.197011] ? watchdog+0x40/0x40 [ 3564.197783] ? hrtimer_interrupt+0x128/0x2c0 [ 3564.199586] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3564.205281] ? apic_timer_interrupt+0xf/0x20 [ 3564.208434] [ 3564.210210] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3564.212091] do_raw_spin_lock+0x177/0x190 [ 3564.213937] _raw_spin_lock+0x21/0x30 [ 3564.215348] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3564.218619] ? do_raw_spin_unlock+0x75/0x190 [ 3564.221228] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3564.223439] ? do_wait_intr+0xf0/0xf0 [ 3564.224978] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3564.227385] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3564.230228] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3564.236203] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3564.240007] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3564.242560] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3564.244680] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3564.247262] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3564.250182] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3564.257222] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3564.259659] lmv_intent_open+0x338/0x1150 [lmv] [ 3564.263540] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3564.265565] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3564.267547] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3564.270066] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3564.272276] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3564.274658] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3564.276174] ? do_raw_spin_unlock+0x75/0x190 [ 3564.277180] ? _raw_spin_unlock+0x12/0x30 [ 3564.281313] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3564.282567] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3564.284430] ll_atomic_open+0x36a/0xd70 [lustre] [ 3564.286254] lookup_open+0x356/0xb40 [ 3564.287341] do_last+0x313/0xfc0 [ 3564.288385] ? nd_jump_root+0xe5/0x160 [ 3564.290533] ? path_init+0x437/0x520 [ 3564.293292] path_openat+0xf7/0x500 [ 3564.296522] ? do_raw_spin_unlock+0x75/0x190 [ 3564.298675] do_filp_open+0x99/0x140 [ 3564.301475] ? getname_flags+0x6e/0x330 [ 3564.306753] ? __check_object_size+0xff/0x256 [ 3564.310275] ? do_raw_spin_unlock+0x75/0x190 [ 3564.311928] ? _raw_spin_unlock+0x12/0x30 [ 3564.312819] do_sys_openat2+0x2b4/0x410 [ 3564.316642] do_sys_open+0x73/0xa0 [ 3564.318524] __x64_sys_openat+0x24/0x30 [ 3564.320545] do_syscall_64+0xc1/0x440 [ 3564.322316] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3564.324162] RIP: 0033:0x7ff54af45332 [ 3564.325655] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3564.340393] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3564.347499] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 3564.351552] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 3564.357877] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 3564.363581] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 3564.365482] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 3572.064035] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 3572.067261] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3572.089731] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3572.096913] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3572.100906] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3572.103809] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3572.114252] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3572.118823] RAX: 0000000000000b90 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 3572.122819] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 3572.125576] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3572.129862] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 3572.134551] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 3572.138842] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 3572.143350] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3572.145596] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 3572.148973] Call Trace: [ 3572.150560] [ 3572.151384] ? show_regs.cold.9+0x22/0x2f [ 3572.153248] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3572.156052] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3572.159183] ? watchdog+0x40/0x40 [ 3572.162240] ? hrtimer_interrupt+0x128/0x2c0 [ 3572.169439] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3572.176622] ? apic_timer_interrupt+0xf/0x20 [ 3572.178053] [ 3572.183032] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3572.188001] do_raw_spin_lock+0x177/0x190 [ 3572.190442] _raw_spin_lock+0x21/0x30 [ 3572.192149] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3572.195523] ? do_raw_spin_unlock+0x75/0x190 [ 3572.197339] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3572.200053] ? do_wait_intr+0xf0/0xf0 [ 3572.201584] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3572.210530] mdc_close+0x53b/0xd70 [mdc] [ 3572.211903] lmv_close+0x18d/0x490 [lmv] [ 3572.214302] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 3572.227535] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3572.237126] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3572.245653] ll_md_close+0x274/0x800 [lustre] [ 3572.247307] ll_file_release+0x128/0xc50 [lustre] [ 3572.255197] __fput+0x10d/0x440 [ 3572.259272] ____fput+0x12/0x20 [ 3572.266756] task_work_run+0xc6/0x110 [ 3572.272393] exit_to_usermode_loop+0x1dd/0x1f0 [ 3572.282285] do_syscall_64+0x3d6/0x440 [ 3572.284971] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3572.294444] RIP: 0033:0x7ff54af45cd8 [ 3572.296420] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3572.327546] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3572.332163] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 3572.338362] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 3572.342666] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 3572.352029] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 3572.358417] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3580.056721] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [ 3580.060909] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 3580.061999] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3580.062076] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3580.062085] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3580.062088] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3580.062113] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3580.062117] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3580.062122] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 3580.062124] RDX: 000000000000250a RSI: 0000000000000003 RDI: ffff96848811cae0 [ 3580.062127] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 3580.062129] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 3580.062131] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3580.062134] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 3580.062136] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3580.062138] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 3580.062145] Call Trace: [ 3580.062151] [ 3580.062158] ? show_regs.cold.9+0x22/0x2f [ 3580.062167] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3580.062174] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3580.062178] ? watchdog+0x40/0x40 [ 3580.062182] ? hrtimer_interrupt+0x128/0x2c0 [ 3580.062186] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3580.062194] ? apic_timer_interrupt+0xf/0x20 [ 3580.062198] [ 3580.062202] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3580.062207] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3580.062211] do_raw_spin_lock+0x177/0x190 [ 3580.062215] _raw_spin_lock+0x21/0x30 [ 3580.062221] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3580.062524] obd_mod_free+0x46/0x140 [ptlrpc] [ 3580.062735] mdc_commit_open+0x91/0xb0 [mdc] [ 3580.062772] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3580.074526] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3580.074608] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3580.074616] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3580.074622] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3580.112471] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3580.112820] after_reply+0xe2c/0x1780 [ptlrpc] [ 3580.113010] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3580.113212] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3580.113431] ? do_wait_intr+0xf0/0xf0 [ 3580.113443] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3580.113660] kthread+0x1d1/0x200 [ 3580.113665] ? set_kthread_struct+0x70/0x70 [ 3580.113669] ret_from_fork+0x1f/0x30 [ 3580.398932] [ 3580.404962] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3580.413782] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3580.441475] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3580.451865] RAX: 00000000000062a7 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 3580.456785] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 3580.465474] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 3580.470517] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 3580.476444] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 3580.483409] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 3580.488785] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3580.490388] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 3580.496244] Call Trace: [ 3580.498871] [ 3580.499879] ? show_regs.cold.9+0x22/0x2f [ 3580.502103] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3580.504628] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3580.506224] ? watchdog+0x40/0x40 [ 3580.507315] ? hrtimer_interrupt+0x128/0x2c0 [ 3580.508760] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3580.510696] ? apic_timer_interrupt+0xf/0x20 [ 3580.512240] [ 3580.512987] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3580.514886] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3580.517195] do_raw_spin_lock+0x177/0x190 [ 3580.518960] _raw_spin_lock+0x21/0x30 [ 3580.520613] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3580.523223] process_one_work+0x2c8/0x700 [ 3580.524519] worker_thread+0x296/0x6e0 [ 3580.526088] ? rescuer_thread+0x570/0x570 [ 3580.527613] kthread+0x1d1/0x200 [ 3580.528833] ? set_kthread_struct+0x70/0x70 [ 3580.530429] ret_from_fork+0x1f/0x30 [ 3584.484402] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1625s! [ 3584.496922] Showing busy workqueues and worker pools: [ 3584.506370] workqueue events: flags=0x0 [ 3584.513904] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3584.525851] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 3584.534481] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3584.536435] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 3584.542162] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3584.545405] pending: vmstat_shepherd [ 3584.546526] workqueue events_long: flags=0x0 [ 3584.547600] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3584.549334] pending: lu_objects_flush [obdclass] [ 3584.550818] workqueue events_unbound: flags=0x2 [ 3584.552159] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3584.553916] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3584.556391] workqueue events_power_efficient: flags=0x80 [ 3584.557428] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3584.560084] pending: neigh_periodic_work [ 3584.561403] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3584.564324] pending: do_cache_clean, check_lifetime [ 3584.567712] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3584.569559] pending: neigh_periodic_work [ 3584.570806] workqueue writeback: flags=0x4a [ 3584.571946] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3584.573570] pending: wb_workfn, wb_workfn [ 3584.574757] workqueue xprtiod: flags=0xa [ 3584.575808] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3584.577844] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 3584.581193] workqueue ipv6_addrconf: flags=0x40008 [ 3584.582940] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3584.585380] pending: addrconf_verify_work [ 3584.587011] workqueue knbd0-recv: flags=0x1a [ 3584.588678] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3584.591020] in-flight: 657:recv_work [ 3584.592563] workqueue ptlrpc_pinger: flags=0x2 [ 3584.594227] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3584.596437] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 3584.598885] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1625s workers=4 idle: 20831 12642 28859 [ 3584.601780] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3574s workers=2 idle: 659 [ 3592.061408] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 3592.110159] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3592.134869] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3592.140652] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3592.143397] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3592.145491] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3592.151447] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3592.153911] RAX: 0000000000007f83 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 3592.156045] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 3592.158333] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3592.160585] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 3592.162827] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 3592.165040] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 3592.167544] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3592.169348] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 3592.171775] Call Trace: [ 3592.172548] [ 3592.173192] ? show_regs.cold.9+0x22/0x2f [ 3592.174479] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3592.175941] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3592.177333] ? watchdog+0x40/0x40 [ 3592.178404] ? hrtimer_interrupt+0x128/0x2c0 [ 3592.179801] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3592.181252] ? apic_timer_interrupt+0xf/0x20 [ 3592.182630] [ 3592.183303] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3592.184992] do_raw_spin_lock+0x177/0x190 [ 3592.186294] _raw_spin_lock+0x21/0x30 [ 3592.187393] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3592.189234] ? do_raw_spin_unlock+0x75/0x190 [ 3592.190536] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3592.192247] ? do_wait_intr+0xf0/0xf0 [ 3592.193379] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3592.195224] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3592.197058] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3592.199163] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3592.200934] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3592.202754] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3592.204227] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3592.205732] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3592.207517] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3592.209702] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3592.211483] lmv_intent_open+0x338/0x1150 [lmv] [ 3592.212986] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3592.214813] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3592.216625] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3592.218059] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3592.219672] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3592.221345] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3592.222904] ? do_raw_spin_unlock+0x75/0x190 [ 3592.224250] ? _raw_spin_unlock+0x12/0x30 [ 3592.225624] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3592.227315] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3592.228971] ll_atomic_open+0x36a/0xd70 [lustre] [ 3592.230722] lookup_open+0x356/0xb40 [ 3592.231823] do_last+0x313/0xfc0 [ 3592.232870] ? nd_jump_root+0xe5/0x160 [ 3592.234045] ? path_init+0x437/0x520 [ 3592.235294] path_openat+0xf7/0x500 [ 3592.236387] ? do_raw_spin_unlock+0x75/0x190 [ 3592.237892] do_filp_open+0x99/0x140 [ 3592.239187] ? getname_flags+0x6e/0x330 [ 3592.240377] ? __check_object_size+0xff/0x256 [ 3592.241837] ? do_raw_spin_unlock+0x75/0x190 [ 3592.243243] ? _raw_spin_unlock+0x12/0x30 [ 3592.244607] do_sys_openat2+0x2b4/0x410 [ 3592.245909] do_sys_open+0x73/0xa0 [ 3592.246985] __x64_sys_openat+0x24/0x30 [ 3592.248240] do_syscall_64+0xc1/0x440 [ 3592.249479] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3592.251335] RIP: 0033:0x7ff54af45332 [ 3592.252711] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3592.258931] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3592.261462] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 3592.263953] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 3592.266372] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 3592.268923] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 3592.271374] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 3600.064136] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 3600.077079] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3600.145858] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3600.156860] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3600.165742] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3600.173108] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3600.197374] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3600.210203] RAX: 000000000000138d RBX: ffff96848811cae0 RCX: 0000000000000001 [ 3600.216329] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 3600.223036] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3600.232831] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 3600.245824] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 3600.255822] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 3600.273022] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3600.275872] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 3600.288491] Call Trace: [ 3600.290799] [ 3600.292254] ? show_regs.cold.9+0x22/0x2f [ 3600.300168] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3600.302316] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3600.307198] ? watchdog+0x40/0x40 [ 3600.310406] ? hrtimer_interrupt+0x128/0x2c0 [ 3600.313497] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3600.319042] ? apic_timer_interrupt+0xf/0x20 [ 3600.326822] [ 3600.328795] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3600.337747] do_raw_spin_lock+0x177/0x190 [ 3600.345736] _raw_spin_lock+0x21/0x30 [ 3600.354347] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3600.368208] ? do_raw_spin_unlock+0x75/0x190 [ 3600.382836] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3600.396265] ? do_wait_intr+0xf0/0xf0 [ 3600.401160] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3600.417828] mdc_close+0x53b/0xd70 [mdc] [ 3600.422006] lmv_close+0x18d/0x490 [lmv] [ 3600.424907] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 3600.432304] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3600.439023] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3600.446187] ll_md_close+0x274/0x800 [lustre] [ 3600.458243] ll_file_release+0x128/0xc50 [lustre] [ 3600.474824] __fput+0x10d/0x440 [ 3600.483295] ____fput+0x12/0x20 [ 3600.492360] task_work_run+0xc6/0x110 [ 3600.494293] exit_to_usermode_loop+0x1dd/0x1f0 [ 3600.499051] do_syscall_64+0x3d6/0x440 [ 3600.500879] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3600.503790] RIP: 0033:0x7ff54af45cd8 [ 3600.507652] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3600.523946] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3600.531484] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 3600.543235] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 3600.556040] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 3600.564308] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 3600.580601] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3608.059013] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 3608.063823] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3608.087201] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3608.093938] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3608.098459] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3608.102556] RIP: 0010:__pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 3608.106207] Code: d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 05 18 d4 dc 02 01 e0 48 83 05 06 d4 dc 02 01 48 83 05 be d5 dc 02 01 8b 45 08 85 [ 3608.118053] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3608.122404] RAX: 0000000000006e81 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 3608.126801] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 3608.131227] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 3608.134646] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 3608.138366] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 3608.142928] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 3608.148239] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3608.152924] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 3608.157401] Call Trace: [ 3608.158785] [ 3608.159861] ? show_regs.cold.9+0x22/0x2f [ 3608.161843] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3608.165181] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3608.167453] ? watchdog+0x40/0x40 [ 3608.169381] ? hrtimer_interrupt+0x128/0x2c0 [ 3608.171932] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3608.174028] ? apic_timer_interrupt+0xf/0x20 [ 3608.176236] [ 3608.177003] ? __pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 3608.179608] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3608.181984] do_raw_spin_lock+0x177/0x190 [ 3608.183983] _raw_spin_lock+0x21/0x30 [ 3608.186120] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3608.189716] process_one_work+0x2c8/0x700 [ 3608.191712] worker_thread+0x296/0x6e0 [ 3608.193415] ? rescuer_thread+0x570/0x570 [ 3608.195021] kthread+0x1d1/0x200 [ 3608.197089] ? set_kthread_struct+0x70/0x70 [ 3608.198821] ret_from_fork+0x1f/0x30 [ 3612.057283] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 3612.064299] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3612.110892] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3612.118371] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3612.125830] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3612.128787] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3612.140599] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3612.144925] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 3612.151510] RDX: 0000000000007ca0 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 3612.162202] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 3612.169660] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 3612.174847] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3612.181808] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 3612.186462] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3612.191483] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 3612.194415] Call Trace: [ 3612.196578] [ 3612.199513] ? show_regs.cold.9+0x22/0x2f [ 3612.203693] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3612.206617] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3612.210715] ? watchdog+0x40/0x40 [ 3612.212696] ? hrtimer_interrupt+0x128/0x2c0 [ 3612.217400] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3612.221928] ? apic_timer_interrupt+0xf/0x20 [ 3612.228000] [ 3612.230005] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3612.238278] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3612.240787] do_raw_spin_lock+0x177/0x190 [ 3612.245292] _raw_spin_lock+0x21/0x30 [ 3612.247461] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3612.252137] obd_mod_free+0x46/0x140 [ptlrpc] [ 3612.257180] mdc_commit_open+0x91/0xb0 [mdc] [ 3612.260871] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3612.265386] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3612.269168] after_reply+0xe2c/0x1780 [ptlrpc] [ 3612.276388] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3612.284753] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3612.287481] ? do_wait_intr+0xf0/0xf0 [ 3612.289562] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3612.294933] kthread+0x1d1/0x200 [ 3612.297979] ? set_kthread_struct+0x70/0x70 [ 3612.300534] ret_from_fork+0x1f/0x30 [ 3614.687022] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1655s! [ 3614.690041] Showing busy workqueues and worker pools: [ 3614.693015] workqueue events: flags=0x0 [ 3614.695219] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3614.698202] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 3614.702352] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3614.705597] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 3614.711390] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3614.714534] pending: vmstat_shepherd [ 3614.716696] workqueue events_long: flags=0x0 [ 3614.718893] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3614.722070] pending: lu_objects_flush [obdclass] [ 3614.724333] workqueue events_unbound: flags=0x2 [ 3614.726696] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3614.729751] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3614.734555] workqueue events_power_efficient: flags=0x80 [ 3614.736735] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3614.739984] pending: neigh_periodic_work [ 3614.742281] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3614.745442] pending: do_cache_clean, check_lifetime [ 3614.748032] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3614.751330] pending: neigh_periodic_work [ 3614.753471] workqueue writeback: flags=0x4a [ 3614.755103] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3614.758268] pending: wb_workfn, wb_workfn [ 3614.760574] workqueue xprtiod: flags=0xa [ 3614.762112] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3614.765190] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 3614.768956] workqueue ipv6_addrconf: flags=0x40008 [ 3614.771511] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3614.774559] pending: addrconf_verify_work [ 3614.776804] workqueue knbd0-recv: flags=0x1a [ 3614.778624] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3614.780800] in-flight: 657:recv_work [ 3614.782447] workqueue ptlrpc_pinger: flags=0x2 [ 3614.784126] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3614.787347] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 3614.790367] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1655s workers=4 idle: 20831 12642 28859 [ 3614.794682] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3605s workers=2 idle: 659 [ 3620.066555] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 3620.074914] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3620.125630] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3620.134047] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3620.141830] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3620.145580] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3620.155976] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3620.160648] RAX: 000000000000454e RBX: ffff96848811cae0 RCX: 0000000000000001 [ 3620.164610] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 3620.168736] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3620.172734] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 3620.175069] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 3620.178698] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 3620.183357] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3620.186600] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 3620.190943] Call Trace: [ 3620.192755] [ 3620.194529] ? show_regs.cold.9+0x22/0x2f [ 3620.197721] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3620.200551] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3620.202765] ? watchdog+0x40/0x40 [ 3620.204627] ? hrtimer_interrupt+0x128/0x2c0 [ 3620.207828] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3620.210998] ? apic_timer_interrupt+0xf/0x20 [ 3620.214941] [ 3620.215959] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3620.219105] do_raw_spin_lock+0x177/0x190 [ 3620.222268] _raw_spin_lock+0x21/0x30 [ 3620.224550] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3620.228491] ? do_raw_spin_unlock+0x75/0x190 [ 3620.230821] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3620.234471] ? do_wait_intr+0xf0/0xf0 [ 3620.236768] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3620.240568] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3620.243786] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3620.248526] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3620.252090] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3620.256247] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3620.258594] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3620.260766] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3620.264551] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3620.268050] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3620.271541] lmv_intent_open+0x338/0x1150 [lmv] [ 3620.274343] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3620.277352] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3620.280444] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3620.282671] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3620.285706] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3620.289453] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3620.292959] ? do_raw_spin_unlock+0x75/0x190 [ 3620.297712] ? _raw_spin_unlock+0x12/0x30 [ 3620.300541] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3620.303699] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3620.306659] ll_atomic_open+0x36a/0xd70 [lustre] [ 3620.309906] lookup_open+0x356/0xb40 [ 3620.312015] do_last+0x313/0xfc0 [ 3620.313506] ? nd_jump_root+0xe5/0x160 [ 3620.314846] ? path_init+0x437/0x520 [ 3620.317911] path_openat+0xf7/0x500 [ 3620.319795] ? do_raw_spin_unlock+0x75/0x190 [ 3620.322136] do_filp_open+0x99/0x140 [ 3620.323966] ? getname_flags+0x6e/0x330 [ 3620.326680] ? __check_object_size+0xff/0x256 [ 3620.328279] ? do_raw_spin_unlock+0x75/0x190 [ 3620.336307] ? _raw_spin_unlock+0x12/0x30 [ 3620.342055] do_sys_openat2+0x2b4/0x410 [ 3620.346059] do_sys_open+0x73/0xa0 [ 3620.348527] __x64_sys_openat+0x24/0x30 [ 3620.351453] do_syscall_64+0xc1/0x440 [ 3620.354012] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3620.358944] RIP: 0033:0x7ff54af45332 [ 3620.361129] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3620.380499] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3620.393758] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 3620.400838] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 3620.411038] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 3620.417349] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 3620.423479] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 3628.058027] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 3628.062588] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3628.090238] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3628.095457] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3628.110035] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3628.118177] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3628.129270] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3628.131335] RAX: 0000000000000e0f RBX: ffff96848811cae0 RCX: 0000000000000001 [ 3628.133688] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 3628.137177] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3628.140391] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 3628.144158] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 3628.148318] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 3628.153005] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3628.156035] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 3628.162293] Call Trace: [ 3628.165687] [ 3628.167273] ? show_regs.cold.9+0x22/0x2f [ 3628.169414] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3628.175041] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3628.179184] ? watchdog+0x40/0x40 [ 3628.181465] ? hrtimer_interrupt+0x128/0x2c0 [ 3628.185405] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3628.188596] ? apic_timer_interrupt+0xf/0x20 [ 3628.195661] [ 3628.197048] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3628.200248] do_raw_spin_lock+0x177/0x190 [ 3628.202475] _raw_spin_lock+0x21/0x30 [ 3628.205679] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3628.209609] ? do_raw_spin_unlock+0x75/0x190 [ 3628.214307] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3628.217476] ? do_wait_intr+0xf0/0xf0 [ 3628.220092] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3628.223399] mdc_close+0x53b/0xd70 [mdc] [ 3628.226793] lmv_close+0x18d/0x490 [lmv] [ 3628.229027] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 3628.234113] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3628.241501] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3628.243983] ll_md_close+0x274/0x800 [lustre] [ 3628.250232] ll_file_release+0x128/0xc50 [lustre] [ 3628.253937] __fput+0x10d/0x440 [ 3628.255195] ____fput+0x12/0x20 [ 3628.255976] task_work_run+0xc6/0x110 [ 3628.257211] exit_to_usermode_loop+0x1dd/0x1f0 [ 3628.263045] do_syscall_64+0x3d6/0x440 [ 3628.267784] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3628.270632] RIP: 0033:0x7ff54af45cd8 [ 3628.272653] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3628.294443] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3628.299969] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 3628.304159] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 3628.307792] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 3628.311232] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 3628.314527] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3633.883205] rcu: INFO: rcu_sched self-detected stall on CPU [ 3633.887351] rcu: 0-...!: (891930 ticks this GP) idle=98a/1/0x4000000000000002 softirq=125154/125154 fqs=959 [ 3633.895933] (t=1680043 jiffies g=253585 q=3276) [ 3633.904462] rcu: rcu_sched kthread starved for 1675003 jiffies! g253585 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [ 3633.920146] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 3633.930454] rcu: RCU grace-period kthread stack dump: [ 3633.942964] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 3633.957774] Call Trace: [ 3633.959417] __schedule+0x351/0xcb0 [ 3633.968380] schedule+0xc0/0x180 [ 3633.971966] schedule_timeout+0xb4/0x190 [ 3633.979275] ? do_raw_spin_unlock+0x75/0x190 [ 3633.989503] ? __next_timer_interrupt+0x160/0x160 [ 3633.995407] ? prepare_to_swait_event+0xf3/0x140 [ 3634.014933] rcu_gp_kthread+0x8dd/0x15b0 [ 3634.020921] ? force_qs_rnp+0x2c0/0x2c0 [ 3634.025437] kthread+0x1d1/0x200 [ 3634.031423] ? set_kthread_struct+0x70/0x70 [ 3634.036771] ret_from_fork+0x1f/0x30 [ 3634.038136] rcu: Stack dump where RCU GP kthread last ran: [ 3634.057071] Sending NMI from CPU 0 to CPUs 2: [ 3634.063293] NMI backtrace for cpu 2 skipped: idling at native_safe_halt+0x2e/0x50 [ 3634.064264] NMI backtrace for cpu 0 [ 3634.074347] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3634.080787] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3634.086974] Call Trace: [ 3634.091205] [ 3634.092958] ? dump_stack+0xbb/0x10e [ 3634.097238] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 3634.110796] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 3634.119277] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 3634.127997] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 3634.138821] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 3634.147740] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 3634.151572] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 3634.153315] ? update_process_times+0x6d/0xc0 [ 3634.159479] ? tick_sched_handle.isra.18+0x27/0x90 [ 3634.166265] ? tick_sched_timer+0x53/0xb0 [ 3634.169193] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3634.173016] ? tick_sched_handle.isra.18+0x90/0x90 [ 3634.176671] ? hrtimer_interrupt+0x128/0x2c0 [ 3634.182319] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3634.186159] ? apic_timer_interrupt+0xf/0x20 [ 3634.189198] [ 3634.190272] ? native_safe_halt+0x2e/0x50 [ 3634.192738] ? kvm_wait+0xcd/0xe0 [ 3634.194317] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3634.196847] ? do_raw_spin_lock+0x177/0x190 [ 3634.198852] ? _raw_spin_lock+0x21/0x30 [ 3634.200935] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3634.203542] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 3634.205399] ? mdc_commit_open+0x91/0xb0 [mdc] [ 3634.206986] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3634.209465] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3634.214726] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 3634.216415] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3634.217969] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3634.219435] ? do_wait_intr+0xf0/0xf0 [ 3634.220436] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3634.222426] ? kthread+0x1d1/0x200 [ 3634.223709] ? set_kthread_struct+0x70/0x70 [ 3634.225266] ? ret_from_fork+0x1f/0x30 [ 3644.902682] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1686s! [ 3644.910808] Showing busy workqueues and worker pools: [ 3644.916922] workqueue events: flags=0x0 [ 3644.922428] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3644.926825] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 3644.938671] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3644.946855] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 3644.955568] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3644.963552] pending: vmstat_shepherd [ 3644.966109] workqueue events_long: flags=0x0 [ 3644.969416] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3644.976826] pending: lu_objects_flush [obdclass] [ 3644.984514] workqueue events_unbound: flags=0x2 [ 3644.991286] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3644.996046] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3645.000849] workqueue events_power_efficient: flags=0x80 [ 3645.003399] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3645.006177] pending: neigh_periodic_work [ 3645.008435] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3645.012362] pending: do_cache_clean, check_lifetime [ 3645.014534] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3645.018599] pending: neigh_periodic_work [ 3645.021721] workqueue writeback: flags=0x4a [ 3645.024072] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3645.026822] pending: wb_workfn, wb_workfn [ 3645.029236] workqueue xprtiod: flags=0xa [ 3645.031608] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3645.034538] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 3645.040842] workqueue ipv6_addrconf: flags=0x40008 [ 3645.044391] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3645.047671] pending: addrconf_verify_work [ 3645.051775] workqueue knbd0-recv: flags=0x1a [ 3645.057360] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3645.063800] in-flight: 657:recv_work [ 3645.066232] workqueue ptlrpc_pinger: flags=0x2 [ 3645.068848] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3645.073680] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 3645.077840] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1686s workers=4 idle: 20831 12642 28859 [ 3645.087526] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3635s workers=2 idle: 659 [ 3648.060401] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 3648.068627] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3648.112087] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3648.118016] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3648.123364] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3648.125900] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3648.136547] RSP: 0000:ffffaf9fc15234f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3648.141024] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 3648.145084] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 3648.149383] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3648.154131] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 3648.158482] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 3648.162422] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 3648.167425] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3648.170427] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 3648.174632] Call Trace: [ 3648.175991] [ 3648.177156] ? show_regs.cold.9+0x22/0x2f [ 3648.180071] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3648.184300] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3648.189007] ? watchdog+0x40/0x40 [ 3648.191998] ? hrtimer_interrupt+0x128/0x2c0 [ 3648.195408] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3648.200178] ? apic_timer_interrupt+0xf/0x20 [ 3648.205184] [ 3648.206495] ? native_safe_halt+0x2e/0x50 [ 3648.211118] kvm_wait+0xcd/0xe0 [ 3648.214139] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3648.220407] do_raw_spin_lock+0x177/0x190 [ 3648.223969] _raw_spin_lock+0x21/0x30 [ 3648.229221] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3648.241393] ? do_raw_spin_unlock+0x75/0x190 [ 3648.244806] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3648.248980] ? do_wait_intr+0xf0/0xf0 [ 3648.256748] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3648.260556] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3648.264307] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3648.270057] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3648.273982] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3648.279602] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3648.283058] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3648.286717] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3648.288787] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3648.296332] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3648.299994] lmv_intent_open+0x338/0x1150 [lmv] [ 3648.302211] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3648.304619] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3648.307659] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3648.309813] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3648.311748] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3648.314414] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3648.317029] ? do_raw_spin_unlock+0x75/0x190 [ 3648.319016] ? _raw_spin_unlock+0x12/0x30 [ 3648.320585] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3648.323060] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3648.325400] ll_atomic_open+0x36a/0xd70 [lustre] [ 3648.327767] lookup_open+0x356/0xb40 [ 3648.329111] do_last+0x313/0xfc0 [ 3648.330416] ? nd_jump_root+0xe5/0x160 [ 3648.332398] ? path_init+0x437/0x520 [ 3648.334359] path_openat+0xf7/0x500 [ 3648.336802] ? do_raw_spin_unlock+0x75/0x190 [ 3648.339449] do_filp_open+0x99/0x140 [ 3648.341948] ? getname_flags+0x6e/0x330 [ 3648.343960] ? __check_object_size+0xff/0x256 [ 3648.346816] ? do_raw_spin_unlock+0x75/0x190 [ 3648.348849] ? _raw_spin_unlock+0x12/0x30 [ 3648.351061] do_sys_openat2+0x2b4/0x410 [ 3648.353181] do_sys_open+0x73/0xa0 [ 3648.355304] __x64_sys_openat+0x24/0x30 [ 3648.357180] do_syscall_64+0xc1/0x440 [ 3648.359676] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3648.362758] RIP: 0033:0x7ff54af45332 [ 3648.364373] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3648.375165] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3648.378323] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 3648.381477] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 3648.385112] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 3648.387784] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 3648.390373] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 3656.059310] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 3656.071521] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3656.132481] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3656.152901] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3656.169531] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3656.177753] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3656.212531] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3656.220320] RAX: 0000000000007f86 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 3656.235548] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 3656.241664] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3656.254490] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 3656.264753] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 3656.275799] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 3656.283600] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3656.289139] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 3656.295115] Call Trace: [ 3656.295686] [ 3656.300494] ? show_regs.cold.9+0x22/0x2f [ 3656.308720] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3656.315332] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3656.322937] ? watchdog+0x40/0x40 [ 3656.328154] ? hrtimer_interrupt+0x128/0x2c0 [ 3656.338246] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3656.339888] ? apic_timer_interrupt+0xf/0x20 [ 3656.346387] [ 3656.351391] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3656.360484] do_raw_spin_lock+0x177/0x190 [ 3656.362048] _raw_spin_lock+0x21/0x30 [ 3656.365022] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3656.368509] ? do_raw_spin_unlock+0x75/0x190 [ 3656.372038] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3656.379056] ? do_wait_intr+0xf0/0xf0 [ 3656.382567] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3656.389669] mdc_close+0x53b/0xd70 [mdc] [ 3656.391989] lmv_close+0x18d/0x490 [lmv] [ 3656.395544] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 3656.399988] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3656.404328] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3656.407724] ll_md_close+0x274/0x800 [lustre] [ 3656.412739] ll_file_release+0x128/0xc50 [lustre] [ 3656.419689] __fput+0x10d/0x440 [ 3656.422785] ____fput+0x12/0x20 [ 3656.424337] task_work_run+0xc6/0x110 [ 3656.432917] exit_to_usermode_loop+0x1dd/0x1f0 [ 3656.441994] do_syscall_64+0x3d6/0x440 [ 3656.449785] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3656.455296] RIP: 0033:0x7ff54af45cd8 [ 3656.464502] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3656.489233] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3656.497922] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 3656.504654] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 3656.519889] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 3656.535294] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 3656.544130] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3660.059494] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 3660.068199] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3660.097813] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3660.102477] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3660.106341] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3660.111497] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3660.114297] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3660.120695] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3660.123557] RAX: 0000000000004183 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 3660.130924] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 3660.136182] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 3660.142711] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 3660.145382] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 3660.149998] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 3660.157142] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3660.162802] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 3660.170518] Call Trace: [ 3660.171517] [ 3660.175751] ? show_regs.cold.9+0x22/0x2f [ 3660.185156] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3660.193651] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3660.198752] ? watchdog+0x40/0x40 [ 3660.202465] ? hrtimer_interrupt+0x128/0x2c0 [ 3660.206562] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3660.210602] ? apic_timer_interrupt+0xf/0x20 [ 3660.214332] [ 3660.216277] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3660.220311] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3660.225850] do_raw_spin_lock+0x177/0x190 [ 3660.227461] _raw_spin_lock+0x21/0x30 [ 3660.230642] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3660.235588] process_one_work+0x2c8/0x700 [ 3660.241117] worker_thread+0x296/0x6e0 [ 3660.250271] ? rescuer_thread+0x570/0x570 [ 3660.251465] kthread+0x1d1/0x200 [ 3660.252405] ? set_kthread_struct+0x70/0x70 [ 3660.256652] ret_from_fork+0x1f/0x30 [ 3672.057550] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 3672.065908] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3672.098178] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3672.108574] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3672.117160] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3672.124056] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3672.133767] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3672.139581] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 3672.145296] RDX: 0000000000001a90 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 3672.151701] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 3672.154364] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 3672.163238] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3672.169511] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 3672.174934] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3672.183428] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 3672.187783] Call Trace: [ 3672.191748] [ 3672.192563] ? show_regs.cold.9+0x22/0x2f [ 3672.194193] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3672.199552] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3672.204964] ? watchdog+0x40/0x40 [ 3672.206279] ? hrtimer_interrupt+0x128/0x2c0 [ 3672.211384] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3672.213038] ? apic_timer_interrupt+0xf/0x20 [ 3672.219082] [ 3672.220110] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3672.222608] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3672.225527] do_raw_spin_lock+0x177/0x190 [ 3672.229749] _raw_spin_lock+0x21/0x30 [ 3672.231158] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3672.236286] obd_mod_free+0x46/0x140 [ptlrpc] [ 3672.238338] mdc_commit_open+0x91/0xb0 [mdc] [ 3672.246175] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3672.249691] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3672.255474] after_reply+0xe2c/0x1780 [ptlrpc] [ 3672.259273] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3672.265284] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3672.270911] ? do_wait_intr+0xf0/0xf0 [ 3672.272437] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3672.277780] kthread+0x1d1/0x200 [ 3672.279038] ? set_kthread_struct+0x70/0x70 [ 3672.286899] ret_from_fork+0x1f/0x30 [ 3675.103374] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1716s! [ 3675.110455] Showing busy workqueues and worker pools: [ 3675.119167] workqueue events: flags=0x0 [ 3675.122499] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3675.126374] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 3675.130479] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3675.133436] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 3675.138814] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3675.141839] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 3675.145066] workqueue events_long: flags=0x0 [ 3675.147061] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3675.149895] pending: lu_objects_flush [obdclass] [ 3675.152185] workqueue events_unbound: flags=0x2 [ 3675.153841] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3675.156315] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3675.166760] workqueue events_power_efficient: flags=0x80 [ 3675.169478] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3675.172825] pending: neigh_periodic_work [ 3675.176707] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3675.180861] pending: do_cache_clean, check_lifetime [ 3675.183861] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3675.187550] pending: neigh_periodic_work [ 3675.192424] workqueue writeback: flags=0x4a [ 3675.196148] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3675.201065] pending: wb_workfn, wb_workfn [ 3675.205554] workqueue xprtiod: flags=0xa [ 3675.208052] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3675.220357] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 3675.228155] workqueue ipv6_addrconf: flags=0x40008 [ 3675.232490] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3675.237532] pending: addrconf_verify_work [ 3675.241813] workqueue knbd0-recv: flags=0x1a [ 3675.244536] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3675.248439] in-flight: 657:recv_work [ 3675.250490] workqueue ptlrpc_pinger: flags=0x2 [ 3675.251921] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3675.253682] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 3675.260207] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1716s workers=4 idle: 20831 12642 28859 [ 3675.268515] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3665s workers=2 idle: 659 [ 3676.061403] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 3676.065454] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3676.101996] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3676.116475] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3676.125756] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3676.133466] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3676.146843] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3676.155607] RAX: 0000000000002b1f RBX: ffff96848811cae0 RCX: 0000000000000001 [ 3676.164450] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 3676.171822] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3676.175398] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 3676.187469] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 3676.191959] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 3676.196803] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3676.200162] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 3676.204152] Call Trace: [ 3676.205992] [ 3676.207038] ? show_regs.cold.9+0x22/0x2f [ 3676.209061] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3676.211438] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3676.213663] ? watchdog+0x40/0x40 [ 3676.215154] ? hrtimer_interrupt+0x128/0x2c0 [ 3676.216925] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3676.219633] ? apic_timer_interrupt+0xf/0x20 [ 3676.221674] [ 3676.222333] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3676.224659] do_raw_spin_lock+0x177/0x190 [ 3676.226104] _raw_spin_lock+0x21/0x30 [ 3676.227727] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3676.231437] ? do_raw_spin_unlock+0x75/0x190 [ 3676.232896] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3676.235039] ? do_wait_intr+0xf0/0xf0 [ 3676.237411] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3676.239992] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3676.243991] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3676.252083] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3676.255744] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3676.259383] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3676.262349] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3676.265812] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3676.273113] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3676.278559] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3676.284942] lmv_intent_open+0x338/0x1150 [lmv] [ 3676.292320] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3676.298014] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3676.303422] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3676.306410] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3676.312310] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3676.314808] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3676.318883] ? do_raw_spin_unlock+0x75/0x190 [ 3676.322686] ? _raw_spin_unlock+0x12/0x30 [ 3676.324534] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3676.331854] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3676.334553] ll_atomic_open+0x36a/0xd70 [lustre] [ 3676.338701] lookup_open+0x356/0xb40 [ 3676.340657] do_last+0x313/0xfc0 [ 3676.342224] ? nd_jump_root+0xe5/0x160 [ 3676.344118] ? path_init+0x437/0x520 [ 3676.345806] path_openat+0xf7/0x500 [ 3676.350021] ? do_raw_spin_unlock+0x75/0x190 [ 3676.351388] do_filp_open+0x99/0x140 [ 3676.355222] ? getname_flags+0x6e/0x330 [ 3676.356433] ? __check_object_size+0xff/0x256 [ 3676.357712] ? do_raw_spin_unlock+0x75/0x190 [ 3676.362347] ? _raw_spin_unlock+0x12/0x30 [ 3676.371174] do_sys_openat2+0x2b4/0x410 [ 3676.372510] do_sys_open+0x73/0xa0 [ 3676.373762] __x64_sys_openat+0x24/0x30 [ 3676.376422] do_syscall_64+0xc1/0x440 [ 3676.377590] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3676.389804] RIP: 0033:0x7ff54af45332 [ 3676.391191] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3676.410454] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3676.415246] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 3676.423001] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 3676.431999] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 3676.438973] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 3676.450714] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 3684.061574] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [ 3684.066838] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3684.116970] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3684.124262] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3684.127838] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3684.134259] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3684.148485] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3684.156449] RAX: 000000000000785a RBX: ffff96848811cae0 RCX: 0000000000000001 [ 3684.162433] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 3684.171090] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3684.180858] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 3684.192139] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 3684.198702] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 3684.210088] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3684.212105] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 3684.224826] Call Trace: [ 3684.226952] [ 3684.230238] ? show_regs.cold.9+0x22/0x2f [ 3684.231469] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3684.237059] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3684.241050] ? watchdog+0x40/0x40 [ 3684.242984] ? hrtimer_interrupt+0x128/0x2c0 [ 3684.247890] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3684.254392] ? apic_timer_interrupt+0xf/0x20 [ 3684.261786] [ 3684.262530] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3684.264488] do_raw_spin_lock+0x177/0x190 [ 3684.272939] _raw_spin_lock+0x21/0x30 [ 3684.277138] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3684.286163] ? do_raw_spin_unlock+0x75/0x190 [ 3684.287673] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3684.298837] ? do_wait_intr+0xf0/0xf0 [ 3684.304151] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3684.310155] mdc_close+0x53b/0xd70 [mdc] [ 3684.314209] lmv_close+0x18d/0x490 [lmv] [ 3684.319256] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 3684.321528] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3684.327010] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3684.331088] ll_md_close+0x274/0x800 [lustre] [ 3684.334920] ll_file_release+0x128/0xc50 [lustre] [ 3684.341068] __fput+0x10d/0x440 [ 3684.343871] ____fput+0x12/0x20 [ 3684.348798] task_work_run+0xc6/0x110 [ 3684.351965] exit_to_usermode_loop+0x1dd/0x1f0 [ 3684.358194] do_syscall_64+0x3d6/0x440 [ 3684.361876] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3684.366840] RIP: 0033:0x7ff54af45cd8 [ 3684.370429] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3684.390201] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3684.397770] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 3684.406989] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 3684.411942] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 3684.415819] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 3684.420376] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3688.061826] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 3688.070291] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3688.107285] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3688.120381] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3688.127836] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3688.133148] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3688.135465] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3688.149265] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3688.151216] RAX: 0000000000007d71 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 3688.164916] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 3688.173287] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 3688.176911] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 3688.186450] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 3688.191104] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 3688.197604] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3688.200436] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 3688.211295] Call Trace: [ 3688.213467] [ 3688.216605] ? show_regs.cold.9+0x22/0x2f [ 3688.223272] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3688.226211] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3688.231306] ? watchdog+0x40/0x40 [ 3688.234625] ? hrtimer_interrupt+0x128/0x2c0 [ 3688.238573] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3688.241418] ? apic_timer_interrupt+0xf/0x20 [ 3688.246371] [ 3688.248122] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3688.251929] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3688.258208] do_raw_spin_lock+0x177/0x190 [ 3688.260924] _raw_spin_lock+0x21/0x30 [ 3688.266356] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3688.272072] process_one_work+0x2c8/0x700 [ 3688.273313] worker_thread+0x296/0x6e0 [ 3688.276450] ? rescuer_thread+0x570/0x570 [ 3688.283292] kthread+0x1d1/0x200 [ 3688.286198] ? set_kthread_struct+0x70/0x70 [ 3688.291794] ret_from_fork+0x1f/0x30 [ 3700.055432] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 3700.065232] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3700.088064] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3700.095025] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3700.099350] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3700.103235] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3700.115117] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3700.127211] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 3700.133169] RDX: 0000000000001594 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 3700.144323] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 3700.149238] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 3700.154284] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3700.165565] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 3700.177633] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3700.186219] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 3700.199243] Call Trace: [ 3700.202578] [ 3700.203228] ? show_regs.cold.9+0x22/0x2f [ 3700.204254] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3700.207233] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3700.213130] ? watchdog+0x40/0x40 [ 3700.219585] ? hrtimer_interrupt+0x128/0x2c0 [ 3700.221116] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3700.226156] ? apic_timer_interrupt+0xf/0x20 [ 3700.232488] [ 3700.233319] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3700.237338] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3700.241148] do_raw_spin_lock+0x177/0x190 [ 3700.243035] _raw_spin_lock+0x21/0x30 [ 3700.244834] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3700.247215] obd_mod_free+0x46/0x140 [ptlrpc] [ 3700.256785] mdc_commit_open+0x91/0xb0 [mdc] [ 3700.258246] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3700.261936] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3700.264829] after_reply+0xe2c/0x1780 [ptlrpc] [ 3700.269780] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3700.272861] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3700.276320] ? do_wait_intr+0xf0/0xf0 [ 3700.280364] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3700.284188] kthread+0x1d1/0x200 [ 3700.291072] ? set_kthread_struct+0x70/0x70 [ 3700.297308] ret_from_fork+0x1f/0x30 [ 3704.060225] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 3704.062872] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3704.090483] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3704.095676] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3704.098524] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3704.100111] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3704.106113] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3704.108646] RAX: 000000000000680d RBX: ffff96848811cae0 RCX: 0000000000000001 [ 3704.110724] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 3704.114944] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3704.123157] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 3704.130763] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 3704.135961] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 3704.138707] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3704.142304] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 3704.146529] Call Trace: [ 3704.147941] [ 3704.148681] ? show_regs.cold.9+0x22/0x2f [ 3704.153907] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3704.159871] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3704.164107] ? watchdog+0x40/0x40 [ 3704.165296] ? hrtimer_interrupt+0x128/0x2c0 [ 3704.169645] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3704.173619] ? apic_timer_interrupt+0xf/0x20 [ 3704.177804] [ 3704.178458] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3704.180290] do_raw_spin_lock+0x177/0x190 [ 3704.182765] _raw_spin_lock+0x21/0x30 [ 3704.183942] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3704.188597] ? do_raw_spin_unlock+0x75/0x190 [ 3704.191417] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3704.201463] ? do_wait_intr+0xf0/0xf0 [ 3704.204776] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3704.210682] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3704.212937] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3704.219230] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3704.225635] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3704.230371] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3704.231807] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3704.234160] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3704.242960] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3704.248620] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3704.254559] lmv_intent_open+0x338/0x1150 [lmv] [ 3704.255981] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3704.261674] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3704.263837] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3704.265970] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3704.272676] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3704.281553] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3704.286516] ? do_raw_spin_unlock+0x75/0x190 [ 3704.289634] ? _raw_spin_unlock+0x12/0x30 [ 3704.291950] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3704.295865] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3704.309488] ll_atomic_open+0x36a/0xd70 [lustre] [ 3704.313263] lookup_open+0x356/0xb40 [ 3704.315727] do_last+0x313/0xfc0 [ 3704.317654] ? nd_jump_root+0xe5/0x160 [ 3704.321019] ? path_init+0x437/0x520 [ 3704.322000] path_openat+0xf7/0x500 [ 3704.324593] ? do_raw_spin_unlock+0x75/0x190 [ 3704.326529] do_filp_open+0x99/0x140 [ 3704.328627] ? getname_flags+0x6e/0x330 [ 3704.330741] ? __check_object_size+0xff/0x256 [ 3704.333317] ? do_raw_spin_unlock+0x75/0x190 [ 3704.335834] ? _raw_spin_unlock+0x12/0x30 [ 3704.338647] do_sys_openat2+0x2b4/0x410 [ 3704.340982] do_sys_open+0x73/0xa0 [ 3704.344439] __x64_sys_openat+0x24/0x30 [ 3704.347690] do_syscall_64+0xc1/0x440 [ 3704.350017] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3704.352739] RIP: 0033:0x7ff54af45332 [ 3704.354852] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3704.367418] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3704.371900] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 3704.376590] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 3704.380612] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 3704.384571] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 3704.388745] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 3705.311050] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1746s! [ 3705.323273] Showing busy workqueues and worker pools: [ 3705.328655] workqueue events: flags=0x0 [ 3705.331710] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3705.338572] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 3705.345903] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3705.356485] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 3705.369620] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3705.383855] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 3705.396119] workqueue events_long: flags=0x0 [ 3705.404573] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3705.409154] pending: lu_objects_flush [obdclass] [ 3705.414996] workqueue events_unbound: flags=0x2 [ 3705.418012] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3705.423135] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3705.434777] workqueue events_power_efficient: flags=0x80 [ 3705.437984] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3705.441988] pending: neigh_periodic_work [ 3705.446398] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3705.453886] pending: do_cache_clean, check_lifetime [ 3705.459721] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3705.464062] pending: neigh_periodic_work [ 3705.466716] workqueue writeback: flags=0x4a [ 3705.468647] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3705.474804] pending: wb_workfn, wb_workfn [ 3705.478540] workqueue xprtiod: flags=0xa [ 3705.488605] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3705.499986] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 3705.512302] workqueue ipv6_addrconf: flags=0x40008 [ 3705.518565] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3705.521129] pending: addrconf_verify_work [ 3705.535387] workqueue knbd0-recv: flags=0x1a [ 3705.538211] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3705.548305] in-flight: 657:recv_work [ 3705.552658] workqueue ptlrpc_pinger: flags=0x2 [ 3705.561746] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3705.563975] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 3705.577009] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1746s workers=4 idle: 20831 12642 28859 [ 3705.581940] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3695s workers=2 idle: 659 [ 3712.058633] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [ 3712.064648] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3712.097344] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3712.103993] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3712.110739] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3712.113421] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3712.127369] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3712.130018] RAX: 0000000000007e9f RBX: ffff96848811cae0 RCX: 0000000000000001 [ 3712.137900] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 3712.145274] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3712.148747] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 3712.153208] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 3712.156171] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 3712.165060] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3712.177251] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 3712.183640] Call Trace: [ 3712.184682] [ 3712.187818] ? show_regs.cold.9+0x22/0x2f [ 3712.197231] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3712.198989] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3712.207652] ? watchdog+0x40/0x40 [ 3712.209189] ? hrtimer_interrupt+0x128/0x2c0 [ 3712.212525] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3712.214880] ? apic_timer_interrupt+0xf/0x20 [ 3712.216920] [ 3712.217705] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3712.220570] do_raw_spin_lock+0x177/0x190 [ 3712.223125] _raw_spin_lock+0x21/0x30 [ 3712.224867] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3712.227261] ? do_raw_spin_unlock+0x75/0x190 [ 3712.229069] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3712.231683] ? do_wait_intr+0xf0/0xf0 [ 3712.240054] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3712.244595] mdc_close+0x53b/0xd70 [mdc] [ 3712.247689] lmv_close+0x18d/0x490 [lmv] [ 3712.249700] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 3712.251980] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3712.254239] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3712.257726] ll_md_close+0x274/0x800 [lustre] [ 3712.260587] ll_file_release+0x128/0xc50 [lustre] [ 3712.262670] __fput+0x10d/0x440 [ 3712.263997] ____fput+0x12/0x20 [ 3712.265238] task_work_run+0xc6/0x110 [ 3712.267196] exit_to_usermode_loop+0x1dd/0x1f0 [ 3712.269966] do_syscall_64+0x3d6/0x440 [ 3712.273016] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3712.275055] RIP: 0033:0x7ff54af45cd8 [ 3712.276684] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3712.283981] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3712.288915] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 3712.291796] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 3712.302549] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 3712.310546] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 3712.326625] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3716.059075] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 3716.063694] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3716.103882] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3716.118949] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3716.126800] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3716.130087] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3716.131702] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3716.162082] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3716.178643] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 3716.187304] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 3716.199074] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 3716.214871] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 3716.222079] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 3716.229825] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 3716.237820] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3716.242291] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 3716.251699] Call Trace: [ 3716.253360] [ 3716.254684] ? show_regs.cold.9+0x22/0x2f [ 3716.256888] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3716.259641] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3716.262665] ? watchdog+0x40/0x40 [ 3716.264111] ? hrtimer_interrupt+0x128/0x2c0 [ 3716.266420] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3716.268858] ? apic_timer_interrupt+0xf/0x20 [ 3716.270938] [ 3716.271999] ? native_safe_halt+0x2e/0x50 [ 3716.274004] kvm_wait+0xcd/0xe0 [ 3716.275845] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3716.278659] do_raw_spin_lock+0x177/0x190 [ 3716.280833] _raw_spin_lock+0x21/0x30 [ 3716.283344] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3716.289187] process_one_work+0x2c8/0x700 [ 3716.294527] worker_thread+0x296/0x6e0 [ 3716.297513] ? rescuer_thread+0x570/0x570 [ 3716.300999] kthread+0x1d1/0x200 [ 3716.302161] ? set_kthread_struct+0x70/0x70 [ 3716.308518] ret_from_fork+0x1f/0x30 [ 3732.055063] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 3732.058404] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3732.060203] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 3732.077486] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3732.077499] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3732.077502] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3732.077515] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3732.077519] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3732.077525] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 3732.077527] RDX: 0000000000005b1f RSI: 0000000000000003 RDI: ffff96848811cae0 [ 3732.077530] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 3732.077532] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 3732.077534] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3732.077538] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 3732.077541] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3732.077543] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 3732.077551] Call Trace: [ 3732.077555] [ 3732.077561] ? show_regs.cold.9+0x22/0x2f [ 3732.077569] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3732.077576] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3732.077579] ? watchdog+0x40/0x40 [ 3732.077584] ? hrtimer_interrupt+0x128/0x2c0 [ 3732.077588] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3732.077596] ? apic_timer_interrupt+0xf/0x20 [ 3732.077600] [ 3732.077603] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3732.077609] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3732.077614] do_raw_spin_lock+0x177/0x190 [ 3732.077618] _raw_spin_lock+0x21/0x30 [ 3732.077626] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3732.078035] obd_mod_free+0x46/0x140 [ptlrpc] [ 3732.078243] mdc_commit_open+0x91/0xb0 [mdc] [ 3732.078281] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3732.078476] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3732.079031] after_reply+0xe2c/0x1780 [ptlrpc] [ 3732.079252] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3732.079452] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3732.079684] ? do_wait_intr+0xf0/0xf0 [ 3732.079695] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3732.084333] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3732.084433] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3732.084441] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3732.084444] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3732.084461] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3732.084466] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3732.084471] RAX: 00000000000064fb RBX: ffff96848811cae0 RCX: 0000000000000001 [ 3732.084474] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 3732.084476] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3732.084478] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 3732.084481] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 3732.084484] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 3732.084487] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3732.084490] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 3732.084498] Call Trace: [ 3732.084504] [ 3732.084510] ? show_regs.cold.9+0x22/0x2f [ 3732.084520] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3732.084526] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3732.084532] ? watchdog+0x40/0x40 [ 3732.084536] ? hrtimer_interrupt+0x128/0x2c0 [ 3732.084541] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3732.084549] ? apic_timer_interrupt+0xf/0x20 [ 3732.084568] [ 3732.084571] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3732.084578] do_raw_spin_lock+0x177/0x190 [ 3732.084584] _raw_spin_lock+0x21/0x30 [ 3732.084589] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3732.085090] ? do_raw_spin_unlock+0x75/0x190 [ 3732.085096] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3732.085308] ? do_wait_intr+0xf0/0xf0 [ 3732.085316] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3732.085510] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3732.086026] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3732.092622] kthread+0x1d1/0x200 [ 3732.092655] ? set_kthread_struct+0x70/0x70 [ 3732.092661] ret_from_fork+0x1f/0x30 [ 3732.318043] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3732.320550] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3732.323391] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3732.325845] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3732.328034] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3732.331110] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3732.333792] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3732.336076] lmv_intent_open+0x338/0x1150 [lmv] [ 3732.338371] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3732.341183] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3732.343697] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3732.346424] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3732.349194] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3732.351498] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3732.354174] ? do_raw_spin_unlock+0x75/0x190 [ 3732.357140] ? _raw_spin_unlock+0x12/0x30 [ 3732.359043] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3732.361726] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3732.364108] ll_atomic_open+0x36a/0xd70 [lustre] [ 3732.366501] lookup_open+0x356/0xb40 [ 3732.368426] do_last+0x313/0xfc0 [ 3732.369999] ? nd_jump_root+0xe5/0x160 [ 3732.371826] ? path_init+0x437/0x520 [ 3732.373340] path_openat+0xf7/0x500 [ 3732.374553] ? do_raw_spin_unlock+0x75/0x190 [ 3732.376523] do_filp_open+0x99/0x140 [ 3732.378414] ? getname_flags+0x6e/0x330 [ 3732.380418] ? __check_object_size+0xff/0x256 [ 3732.382502] ? do_raw_spin_unlock+0x75/0x190 [ 3732.384412] ? _raw_spin_unlock+0x12/0x30 [ 3732.386695] do_sys_openat2+0x2b4/0x410 [ 3732.388462] do_sys_open+0x73/0xa0 [ 3732.390027] __x64_sys_openat+0x24/0x30 [ 3732.391925] do_syscall_64+0xc1/0x440 [ 3732.393670] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3732.395683] RIP: 0033:0x7ff54af45332 [ 3732.397482] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3732.406860] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3732.410485] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 3732.413604] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 3732.416936] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 3732.420488] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 3732.423703] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 3736.031137] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1777s! [ 3736.035789] Showing busy workqueues and worker pools: [ 3736.039437] workqueue events: flags=0x0 [ 3736.041547] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3736.045575] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 3736.052710] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3736.062899] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 3736.077147] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3736.084685] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 3736.092261] workqueue events_long: flags=0x0 [ 3736.094881] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3736.101935] pending: lu_objects_flush [obdclass] [ 3736.105173] workqueue events_unbound: flags=0x2 [ 3736.108836] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3736.112127] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3736.117847] workqueue events_power_efficient: flags=0x80 [ 3736.120381] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3736.124002] pending: neigh_periodic_work [ 3736.126312] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3736.129006] pending: do_cache_clean, check_lifetime [ 3736.138128] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3736.141447] pending: neigh_periodic_work [ 3736.144045] workqueue writeback: flags=0x4a [ 3736.146344] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3736.152035] pending: wb_workfn, wb_workfn [ 3736.154053] workqueue xprtiod: flags=0xa [ 3736.164559] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3736.170565] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 3736.180481] workqueue ipv6_addrconf: flags=0x40008 [ 3736.185078] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3736.187860] pending: addrconf_verify_work [ 3736.190252] workqueue knbd0-recv: flags=0x1a [ 3736.192150] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3736.194961] in-flight: 657:recv_work [ 3736.197011] workqueue ptlrpc_pinger: flags=0x2 [ 3736.199142] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3736.201903] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 3736.204830] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1777s workers=4 idle: 20831 12642 28859 [ 3736.207888] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3726s workers=2 idle: 659 [ 3740.058025] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [ 3740.064838] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3740.118344] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3740.129926] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3740.143301] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3740.144927] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3740.151978] RSP: 0018:ffffaf9fc1543ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3740.155065] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 3740.157755] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 3740.159830] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3740.162321] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 3740.164553] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 3740.167251] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 3740.170102] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3740.171674] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 3740.174348] Call Trace: [ 3740.175369] [ 3740.176272] ? show_regs.cold.9+0x22/0x2f [ 3740.177553] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3740.179676] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3740.181558] ? watchdog+0x40/0x40 [ 3740.182829] ? hrtimer_interrupt+0x128/0x2c0 [ 3740.184420] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3740.186138] ? apic_timer_interrupt+0xf/0x20 [ 3740.187490] [ 3740.188212] ? native_safe_halt+0x2e/0x50 [ 3740.189522] kvm_wait+0xcd/0xe0 [ 3740.190576] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3740.193062] do_raw_spin_lock+0x177/0x190 [ 3740.194753] _raw_spin_lock+0x21/0x30 [ 3740.198948] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3740.202772] ? do_raw_spin_unlock+0x75/0x190 [ 3740.204962] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3740.208401] ? do_wait_intr+0xf0/0xf0 [ 3740.209897] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3740.221213] mdc_close+0x53b/0xd70 [mdc] [ 3740.223734] lmv_close+0x18d/0x490 [lmv] [ 3740.225358] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 3740.232345] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3740.235423] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3740.237766] ll_md_close+0x274/0x800 [lustre] [ 3740.240183] ll_file_release+0x128/0xc50 [lustre] [ 3740.242531] __fput+0x10d/0x440 [ 3740.244301] ____fput+0x12/0x20 [ 3740.245513] task_work_run+0xc6/0x110 [ 3740.247308] exit_to_usermode_loop+0x1dd/0x1f0 [ 3740.249386] do_syscall_64+0x3d6/0x440 [ 3740.251234] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3740.253352] RIP: 0033:0x7ff54af45cd8 [ 3740.255712] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3740.263894] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3740.270957] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 3740.273895] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 3740.275715] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 3740.279919] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 3740.291646] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3744.059462] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 3744.064922] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3744.107357] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3744.111831] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3744.118833] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3744.125266] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3744.134880] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3744.151454] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3744.160110] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 3744.167811] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 3744.177110] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 3744.182486] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 3744.191320] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 3744.198391] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 3744.205613] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3744.208489] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 3744.212193] Call Trace: [ 3744.214712] [ 3744.215537] ? show_regs.cold.9+0x22/0x2f [ 3744.218181] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3744.226444] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3744.231741] ? watchdog+0x40/0x40 [ 3744.237849] ? hrtimer_interrupt+0x128/0x2c0 [ 3744.240652] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3744.242690] ? apic_timer_interrupt+0xf/0x20 [ 3744.254159] [ 3744.254914] ? native_safe_halt+0x2e/0x50 [ 3744.263789] kvm_wait+0xcd/0xe0 [ 3744.268844] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3744.273201] do_raw_spin_lock+0x177/0x190 [ 3744.275226] _raw_spin_lock+0x21/0x30 [ 3744.282217] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3744.288435] process_one_work+0x2c8/0x700 [ 3744.289658] worker_thread+0x296/0x6e0 [ 3744.297768] ? rescuer_thread+0x570/0x570 [ 3744.309930] kthread+0x1d1/0x200 [ 3744.311183] ? set_kthread_struct+0x70/0x70 [ 3744.312647] ret_from_fork+0x1f/0x30 [ 3760.063472] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 3760.070195] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3760.096141] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3760.100991] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3760.105074] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3760.107695] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3760.116245] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3760.123741] RAX: 00000000000075aa RBX: ffff96848811cae0 RCX: 0000000000000001 [ 3760.129354] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 3760.137291] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3760.143191] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 3760.149929] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 3760.158004] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 3760.168130] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3760.172346] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 3760.181509] Call Trace: [ 3760.185564] [ 3760.188672] ? show_regs.cold.9+0x22/0x2f [ 3760.191188] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3760.197118] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3760.199371] ? watchdog+0x40/0x40 [ 3760.201738] ? hrtimer_interrupt+0x128/0x2c0 [ 3760.204101] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3760.209274] ? apic_timer_interrupt+0xf/0x20 [ 3760.212207] [ 3760.214343] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3760.220370] do_raw_spin_lock+0x177/0x190 [ 3760.223025] _raw_spin_lock+0x21/0x30 [ 3760.226377] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3760.229408] ? do_raw_spin_unlock+0x75/0x190 [ 3760.234938] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3760.240923] ? do_wait_intr+0xf0/0xf0 [ 3760.244396] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3760.256577] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3760.262921] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3760.272332] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3760.278836] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3760.286361] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3760.289590] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3760.295374] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3760.299988] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3760.304207] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3760.311743] lmv_intent_open+0x338/0x1150 [lmv] [ 3760.314570] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3760.318673] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3760.322947] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3760.328214] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3760.331213] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3760.337308] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3760.339253] ? do_raw_spin_unlock+0x75/0x190 [ 3760.340821] ? _raw_spin_unlock+0x12/0x30 [ 3760.342390] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3760.344377] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3760.346504] ll_atomic_open+0x36a/0xd70 [lustre] [ 3760.348473] lookup_open+0x356/0xb40 [ 3760.349880] do_last+0x313/0xfc0 [ 3760.351217] ? nd_jump_root+0xe5/0x160 [ 3760.352424] ? path_init+0x437/0x520 [ 3760.353404] path_openat+0xf7/0x500 [ 3760.354498] ? do_raw_spin_unlock+0x75/0x190 [ 3760.356244] do_filp_open+0x99/0x140 [ 3760.357668] ? getname_flags+0x6e/0x330 [ 3760.359142] ? __check_object_size+0xff/0x256 [ 3760.360105] ? do_raw_spin_unlock+0x75/0x190 [ 3760.361382] ? _raw_spin_unlock+0x12/0x30 [ 3760.362865] do_sys_openat2+0x2b4/0x410 [ 3760.364147] do_sys_open+0x73/0xa0 [ 3760.365492] __x64_sys_openat+0x24/0x30 [ 3760.366753] do_syscall_64+0xc1/0x440 [ 3760.368078] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3760.370025] RIP: 0033:0x7ff54af45332 [ 3760.371403] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3760.377927] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3760.380598] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 3760.383321] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 3760.385618] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 3760.388613] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 3760.390554] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 3764.059056] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 3764.066063] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3764.119095] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3764.129854] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3764.135792] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3764.141198] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3764.153571] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3764.157918] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 3764.162937] RDX: 00000000000008ce RSI: 0000000000000003 RDI: ffff96848811cae0 [ 3764.169363] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 3764.176216] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 3764.181116] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3764.189433] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 3764.195374] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3764.199533] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 3764.206495] Call Trace: [ 3764.208601] [ 3764.210136] ? show_regs.cold.9+0x22/0x2f [ 3764.213103] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3764.218064] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3764.222189] ? watchdog+0x40/0x40 [ 3764.226944] ? hrtimer_interrupt+0x128/0x2c0 [ 3764.231974] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3764.235951] ? apic_timer_interrupt+0xf/0x20 [ 3764.246565] [ 3764.253646] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3764.255612] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3764.267954] do_raw_spin_lock+0x177/0x190 [ 3764.271974] _raw_spin_lock+0x21/0x30 [ 3764.275995] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3764.280182] obd_mod_free+0x46/0x140 [ptlrpc] [ 3764.283903] mdc_commit_open+0x91/0xb0 [mdc] [ 3764.292191] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3764.307336] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3764.312092] after_reply+0xe2c/0x1780 [ptlrpc] [ 3764.320513] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3764.329382] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3764.340481] ? do_wait_intr+0xf0/0xf0 [ 3764.346843] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3764.356675] kthread+0x1d1/0x200 [ 3764.361879] ? set_kthread_struct+0x70/0x70 [ 3764.364846] ret_from_fork+0x1f/0x30 [ 3766.239716] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1807s! [ 3766.251199] Showing busy workqueues and worker pools: [ 3766.256426] workqueue events: flags=0x0 [ 3766.262714] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3766.269810] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 3766.279095] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3766.285199] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 3766.294666] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3766.302189] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 3766.307593] workqueue events_long: flags=0x0 [ 3766.310106] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3766.315525] pending: lu_objects_flush [obdclass] [ 3766.319698] workqueue events_unbound: flags=0x2 [ 3766.322621] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3766.327943] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3766.334433] workqueue events_power_efficient: flags=0x80 [ 3766.337121] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3766.340579] pending: neigh_periodic_work [ 3766.342951] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3766.346428] pending: do_cache_clean, check_lifetime [ 3766.349089] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3766.354065] pending: neigh_periodic_work [ 3766.358912] workqueue writeback: flags=0x4a [ 3766.366650] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3766.374088] pending: wb_workfn, wb_workfn [ 3766.377790] workqueue xprtiod: flags=0xa [ 3766.379985] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3766.383542] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 3766.388116] workqueue ipv6_addrconf: flags=0x40008 [ 3766.391095] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3766.396126] pending: addrconf_verify_work [ 3766.398287] workqueue knbd0-recv: flags=0x1a [ 3766.400156] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3766.402102] in-flight: 657:recv_work [ 3766.403280] workqueue ptlrpc_pinger: flags=0x2 [ 3766.405560] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3766.407960] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 3766.412641] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1807s workers=4 idle: 20831 12642 28859 [ 3766.416198] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3756s workers=2 idle: 659 [ 3768.058225] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [ 3768.067064] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3768.097279] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3768.106418] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3768.111616] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3768.115261] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3768.128870] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3768.135560] RAX: 0000000000007d81 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 3768.139950] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 3768.146622] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3768.151847] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 3768.155600] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 3768.159698] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 3768.163416] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3768.165426] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 3768.169311] Call Trace: [ 3768.170703] [ 3768.172155] ? show_regs.cold.9+0x22/0x2f [ 3768.174854] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3768.182221] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3768.187017] ? watchdog+0x40/0x40 [ 3768.190815] ? hrtimer_interrupt+0x128/0x2c0 [ 3768.193862] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3768.197271] ? apic_timer_interrupt+0xf/0x20 [ 3768.203532] [ 3768.204517] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3768.216952] do_raw_spin_lock+0x177/0x190 [ 3768.221957] _raw_spin_lock+0x21/0x30 [ 3768.225896] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3768.231306] ? do_raw_spin_unlock+0x75/0x190 [ 3768.235784] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3768.239203] ? do_wait_intr+0xf0/0xf0 [ 3768.241233] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3768.245093] mdc_close+0x53b/0xd70 [mdc] [ 3768.247000] lmv_close+0x18d/0x490 [lmv] [ 3768.260473] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 3768.267975] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3768.275067] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3768.282323] ll_md_close+0x274/0x800 [lustre] [ 3768.287831] ll_file_release+0x128/0xc50 [lustre] [ 3768.290150] __fput+0x10d/0x440 [ 3768.292007] ____fput+0x12/0x20 [ 3768.293420] task_work_run+0xc6/0x110 [ 3768.297931] exit_to_usermode_loop+0x1dd/0x1f0 [ 3768.303076] do_syscall_64+0x3d6/0x440 [ 3768.306035] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3768.311688] RIP: 0033:0x7ff54af45cd8 [ 3768.313632] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3768.328642] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3768.335275] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 3768.343021] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 3768.348958] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 3768.352701] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 3768.356991] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3772.059159] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 3772.063596] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3772.103691] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3772.113620] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3772.120210] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3772.125600] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3772.131209] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3772.151162] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3772.155590] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 3772.160272] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 3772.163408] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 3772.168501] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 3772.173274] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 3772.178550] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 3772.182751] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3772.188504] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 3772.195395] Call Trace: [ 3772.198117] [ 3772.198883] ? show_regs.cold.9+0x22/0x2f [ 3772.200553] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3772.205050] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3772.209388] ? watchdog+0x40/0x40 [ 3772.212176] ? hrtimer_interrupt+0x128/0x2c0 [ 3772.213969] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3772.216091] ? apic_timer_interrupt+0xf/0x20 [ 3772.219166] [ 3772.221041] ? native_safe_halt+0x2e/0x50 [ 3772.225818] kvm_wait+0xcd/0xe0 [ 3772.228755] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3772.232116] do_raw_spin_lock+0x177/0x190 [ 3772.233904] _raw_spin_lock+0x21/0x30 [ 3772.237217] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3772.243157] process_one_work+0x2c8/0x700 [ 3772.245267] worker_thread+0x296/0x6e0 [ 3772.247318] ? rescuer_thread+0x570/0x570 [ 3772.249367] kthread+0x1d1/0x200 [ 3772.251013] ? set_kthread_struct+0x70/0x70 [ 3772.255960] ret_from_fork+0x1f/0x30 [ 3788.060210] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 3788.064015] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3788.083763] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3788.089192] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3788.098536] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3788.099729] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3788.119922] RSP: 0000:ffffaf9fc15234f0 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 3788.127282] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 3788.134668] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 3788.136499] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3788.141778] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 3788.144813] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 3788.147058] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 3788.151003] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3788.157787] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 3788.162918] Call Trace: [ 3788.168356] [ 3788.174502] ? show_regs.cold.9+0x22/0x2f [ 3788.175520] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3788.176684] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3788.183094] ? watchdog+0x40/0x40 [ 3788.184425] ? hrtimer_interrupt+0x128/0x2c0 [ 3788.187344] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3788.189667] ? apic_timer_interrupt+0xf/0x20 [ 3788.191822] [ 3788.192685] ? native_safe_halt+0x2e/0x50 [ 3788.196432] kvm_wait+0xcd/0xe0 [ 3788.198653] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3788.205373] do_raw_spin_lock+0x177/0x190 [ 3788.208868] _raw_spin_lock+0x21/0x30 [ 3788.209917] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3788.218753] ? do_raw_spin_unlock+0x75/0x190 [ 3788.221034] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3788.222957] ? do_wait_intr+0xf0/0xf0 [ 3788.227715] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3788.231777] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3788.241253] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3788.246823] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3788.252984] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3788.256049] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3788.258625] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3788.260582] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3788.265462] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3788.269079] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3788.272189] lmv_intent_open+0x338/0x1150 [lmv] [ 3788.274427] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3788.277467] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3788.281653] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3788.284342] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3788.286955] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3788.291775] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3788.294361] ? do_raw_spin_unlock+0x75/0x190 [ 3788.296624] ? _raw_spin_unlock+0x12/0x30 [ 3788.299417] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3788.302395] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3788.304838] ll_atomic_open+0x36a/0xd70 [lustre] [ 3788.307421] lookup_open+0x356/0xb40 [ 3788.309463] do_last+0x313/0xfc0 [ 3788.311413] ? nd_jump_root+0xe5/0x160 [ 3788.313450] ? path_init+0x437/0x520 [ 3788.315615] path_openat+0xf7/0x500 [ 3788.317391] ? do_raw_spin_unlock+0x75/0x190 [ 3788.319936] do_filp_open+0x99/0x140 [ 3788.321483] ? getname_flags+0x6e/0x330 [ 3788.324429] ? __check_object_size+0xff/0x256 [ 3788.327163] ? do_raw_spin_unlock+0x75/0x190 [ 3788.330124] ? _raw_spin_unlock+0x12/0x30 [ 3788.332240] do_sys_openat2+0x2b4/0x410 [ 3788.334904] do_sys_open+0x73/0xa0 [ 3788.336610] __x64_sys_openat+0x24/0x30 [ 3788.339797] do_syscall_64+0xc1/0x440 [ 3788.341771] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3788.347638] RIP: 0033:0x7ff54af45332 [ 3788.353220] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3788.364780] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3788.369378] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 3788.373307] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 3788.378417] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 3788.382257] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 3788.387493] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 3792.059712] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 3792.069995] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3792.100663] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3792.115809] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3792.122805] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3792.125893] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3792.137814] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3792.144198] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 3792.152920] RDX: 0000000000007664 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 3792.160443] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 3792.165197] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 3792.169871] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3792.174175] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 3792.179835] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3792.182876] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 3792.187747] Call Trace: [ 3792.189157] [ 3792.190456] ? show_regs.cold.9+0x22/0x2f [ 3792.193089] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3792.195762] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3792.198302] ? watchdog+0x40/0x40 [ 3792.202990] ? hrtimer_interrupt+0x128/0x2c0 [ 3792.206130] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3792.209689] ? apic_timer_interrupt+0xf/0x20 [ 3792.212583] [ 3792.213811] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3792.216955] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3792.219661] do_raw_spin_lock+0x177/0x190 [ 3792.221744] _raw_spin_lock+0x21/0x30 [ 3792.223861] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3792.226524] obd_mod_free+0x46/0x140 [ptlrpc] [ 3792.231621] mdc_commit_open+0x91/0xb0 [mdc] [ 3792.236674] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3792.243714] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3792.246840] after_reply+0xe2c/0x1780 [ptlrpc] [ 3792.253221] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3792.256637] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3792.259960] ? do_wait_intr+0xf0/0xf0 [ 3792.261767] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3792.264961] kthread+0x1d1/0x200 [ 3792.266793] ? set_kthread_struct+0x70/0x70 [ 3792.268916] ret_from_fork+0x1f/0x30 [ 3796.058896] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 3796.064875] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3796.103857] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3796.111490] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3796.118900] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3796.123044] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3796.142277] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3796.147204] RAX: 0000000000007f8a RBX: ffff96848811cae0 RCX: 0000000000000001 [ 3796.151768] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 3796.157733] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3796.166270] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 3796.170802] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 3796.175625] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 3796.181596] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3796.189377] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 3796.203972] Call Trace: [ 3796.205138] [ 3796.206294] ? show_regs.cold.9+0x22/0x2f [ 3796.221102] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3796.230709] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3796.232583] ? watchdog+0x40/0x40 [ 3796.246330] ? hrtimer_interrupt+0x128/0x2c0 [ 3796.255190] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3796.264268] ? apic_timer_interrupt+0xf/0x20 [ 3796.270647] [ 3796.273176] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3796.284543] do_raw_spin_lock+0x177/0x190 [ 3796.298171] _raw_spin_lock+0x21/0x30 [ 3796.299777] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3796.314086] ? do_raw_spin_unlock+0x75/0x190 [ 3796.316004] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3796.326482] ? do_wait_intr+0xf0/0xf0 [ 3796.329575] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3796.339650] mdc_close+0x53b/0xd70 [mdc] [ 3796.345167] lmv_close+0x18d/0x490 [lmv] [ 3796.347808] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 3796.350428] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3796.358706] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3796.361958] ll_md_close+0x274/0x800 [lustre] [ 3796.363723] ll_file_release+0x128/0xc50 [lustre] [ 3796.367092] __fput+0x10d/0x440 [ 3796.368268] ____fput+0x12/0x20 [ 3796.369536] task_work_run+0xc6/0x110 [ 3796.375403] exit_to_usermode_loop+0x1dd/0x1f0 [ 3796.385841] do_syscall_64+0x3d6/0x440 [ 3796.388518] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3796.390537] RIP: 0033:0x7ff54af45cd8 [ 3796.399877] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3796.418328] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3796.423472] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 3796.429389] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 3796.433190] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 3796.438287] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 3796.441378] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3796.447251] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1837s! [ 3796.450807] Showing busy workqueues and worker pools: [ 3796.453420] workqueue events: flags=0x0 [ 3796.455276] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3796.458780] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 3796.463494] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3796.466373] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 3796.472519] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3796.476176] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 3796.480322] workqueue events_long: flags=0x0 [ 3796.482632] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3796.490291] pending: lu_objects_flush [obdclass] [ 3796.494715] workqueue events_unbound: flags=0x2 [ 3796.497469] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3796.506541] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3796.514640] workqueue events_power_efficient: flags=0x80 [ 3796.517886] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3796.521514] pending: neigh_periodic_work [ 3796.524223] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3796.528230] pending: do_cache_clean, check_lifetime [ 3796.531187] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3796.535281] pending: neigh_periodic_work [ 3796.538101] workqueue writeback: flags=0x4a [ 3796.540212] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3796.547623] pending: wb_workfn, wb_workfn [ 3796.553333] workqueue xprtiod: flags=0xa [ 3796.555392] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3796.565450] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 3796.571570] workqueue ipv6_addrconf: flags=0x40008 [ 3796.575069] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3796.579206] pending: addrconf_verify_work [ 3796.587901] workqueue knbd0-recv: flags=0x1a [ 3796.593562] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3796.599416] in-flight: 657:recv_work [ 3796.605207] workqueue ptlrpc_pinger: flags=0x2 [ 3796.611505] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3796.618434] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 3796.626308] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1837s workers=4 idle: 20831 12642 28859 [ 3796.639914] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3786s workers=2 idle: 659 [ 3800.066879] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 3800.078372] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3800.103296] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3800.108020] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3800.111465] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3800.114243] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3800.116529] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3800.123910] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3800.127738] RAX: 0000000000007cd1 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 3800.131111] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 3800.134915] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 3800.138279] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 3800.141731] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 3800.146180] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 3800.155843] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3800.159040] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 3800.170740] Call Trace: [ 3800.174906] [ 3800.176280] ? show_regs.cold.9+0x22/0x2f [ 3800.177783] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3800.181546] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3800.183647] ? watchdog+0x40/0x40 [ 3800.188347] ? hrtimer_interrupt+0x128/0x2c0 [ 3800.198189] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3800.208850] ? apic_timer_interrupt+0xf/0x20 [ 3800.210383] [ 3800.215253] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3800.222150] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3800.226784] do_raw_spin_lock+0x177/0x190 [ 3800.232901] _raw_spin_lock+0x21/0x30 [ 3800.235243] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3800.241453] process_one_work+0x2c8/0x700 [ 3800.244329] worker_thread+0x296/0x6e0 [ 3800.245875] ? rescuer_thread+0x570/0x570 [ 3800.248720] kthread+0x1d1/0x200 [ 3800.250292] ? set_kthread_struct+0x70/0x70 [ 3800.252183] ret_from_fork+0x1f/0x30 [ 3813.888547] rcu: INFO: rcu_sched self-detected stall on CPU [ 3813.899239] rcu: 0-...!: (989911 ticks this GP) idle=98a/1/0x4000000000000002 softirq=125154/125154 fqs=959 [ 3813.913271] (t=1860048 jiffies g=253585 q=3310) [ 3813.917475] rcu: rcu_sched kthread starved for 1855008 jiffies! g253585 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=3 [ 3813.932544] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 3813.943907] rcu: RCU grace-period kthread stack dump: [ 3813.948970] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 3813.964774] Call Trace: [ 3813.971945] __schedule+0x351/0xcb0 [ 3813.978827] schedule+0xc0/0x180 [ 3813.981192] schedule_timeout+0xb4/0x190 [ 3813.985792] ? do_raw_spin_unlock+0x75/0x190 [ 3813.996555] ? __next_timer_interrupt+0x160/0x160 [ 3814.006255] ? prepare_to_swait_event+0xf3/0x140 [ 3814.012938] rcu_gp_kthread+0x8dd/0x15b0 [ 3814.018152] ? force_qs_rnp+0x2c0/0x2c0 [ 3814.031734] kthread+0x1d1/0x200 [ 3814.033719] ? set_kthread_struct+0x70/0x70 [ 3814.040989] ret_from_fork+0x1f/0x30 [ 3814.050280] rcu: Stack dump where RCU GP kthread last ran: [ 3814.061509] Sending NMI from CPU 0 to CPUs 3: [ 3814.073874] INFO: NMI handler (nmi_cpu_backtrace_handler) took too long to run: 5.939 msecs [ 3814.079612] NMI backtrace for cpu 3 [ 3814.079636] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3814.079639] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3814.079640] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3814.079642] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3814.079645] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000216 [ 3814.079659] RAX: 00000000000072ae RBX: ffff96848811cae0 RCX: 0000000000000001 [ 3814.079661] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 3814.079662] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3814.079664] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 3814.079665] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 3814.079666] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 3814.079681] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3814.079683] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 3814.079684] Call Trace: [ 3814.079685] [ 3814.079686] ? show_regs.cold.9+0x22/0x2f [ 3814.079688] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 3814.079689] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 3814.079689] ? nmi_handle+0xbd/0x210 [ 3814.079691] ? default_do_nmi+0x4b/0x1b0 [ 3814.079692] ? do_nmi+0x280/0x380 [ 3814.079693] ? end_repeat_nmi+0x16/0x1f [ 3814.079694] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3814.079696] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3814.079697] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3814.079698] [ 3814.079699] do_raw_spin_lock+0x177/0x190 [ 3814.079700] _raw_spin_lock+0x21/0x30 [ 3814.079724] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3814.079726] ? do_raw_spin_unlock+0x75/0x190 [ 3814.079728] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3814.079728] ? do_wait_intr+0xf0/0xf0 [ 3814.079730] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3814.079731] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3814.079732] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3814.079733] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3814.079735] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3814.079736] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3814.079737] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3814.079739] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3814.079740] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3814.079741] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3814.079743] lmv_intent_open+0x338/0x1150 [lmv] [ 3814.079755] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3814.079756] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3814.079757] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3814.079758] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3814.079759] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3814.079760] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3814.079761] ? do_raw_spin_unlock+0x75/0x190 [ 3814.079762] ? _raw_spin_unlock+0x12/0x30 [ 3814.079763] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3814.079764] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3814.079766] ll_atomic_open+0x36a/0xd70 [lustre] [ 3814.079767] lookup_open+0x356/0xb40 [ 3814.079768] do_last+0x313/0xfc0 [ 3814.079769] ? nd_jump_root+0xe5/0x160 [ 3814.079770] ? path_init+0x437/0x520 [ 3814.079771] path_openat+0xf7/0x500 [ 3814.079772] ? do_raw_spin_unlock+0x75/0x190 [ 3814.079773] do_filp_open+0x99/0x140 [ 3814.079774] ? getname_flags+0x6e/0x330 [ 3814.079775] ? __check_object_size+0xff/0x256 [ 3814.079776] ? do_raw_spin_unlock+0x75/0x190 [ 3814.079778] ? _raw_spin_unlock+0x12/0x30 [ 3814.079779] do_sys_openat2+0x2b4/0x410 [ 3814.079780] do_sys_open+0x73/0xa0 [ 3814.079781] __x64_sys_openat+0x24/0x30 [ 3814.079782] do_syscall_64+0xc1/0x440 [ 3814.079783] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3814.079783] RIP: 0033:0x7ff54af45332 [ 3814.079785] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3814.079787] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3814.079789] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 3814.079791] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 3814.079792] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 3814.079793] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 3814.079795] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 3814.079801] NMI backtrace for cpu 0 [ 3814.618107] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3814.624953] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3814.633782] Call Trace: [ 3814.635070] [ 3814.636105] ? dump_stack+0xbb/0x10e [ 3814.638621] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 3814.645243] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 3814.648591] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 3814.650763] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 3814.655832] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 3814.659410] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 3814.664168] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 3814.667505] ? update_process_times+0x6d/0xc0 [ 3814.670239] ? tick_sched_handle.isra.18+0x27/0x90 [ 3814.673389] ? tick_sched_timer+0x53/0xb0 [ 3814.675941] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3814.679064] ? tick_sched_handle.isra.18+0x90/0x90 [ 3814.682368] ? hrtimer_interrupt+0x128/0x2c0 [ 3814.685901] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3814.689939] ? apic_timer_interrupt+0xf/0x20 [ 3814.694316] [ 3814.695750] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3814.699062] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3814.702566] ? do_raw_spin_lock+0x177/0x190 [ 3814.707490] ? _raw_spin_lock+0x21/0x30 [ 3814.710203] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3814.714922] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 3814.717874] ? mdc_commit_open+0x91/0xb0 [mdc] [ 3814.721160] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3814.726739] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3814.730751] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 3814.733834] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3814.739470] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3814.747213] ? do_wait_intr+0xf0/0xf0 [ 3814.750151] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3814.754186] ? kthread+0x1d1/0x200 [ 3814.756759] ? set_kthread_struct+0x70/0x70 [ 3814.762156] ? ret_from_fork+0x1f/0x30 [ 3824.058069] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 3824.062636] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3824.100720] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3824.115730] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3824.123941] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3824.129396] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3824.149401] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3824.158861] RAX: 0000000000007085 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 3824.164460] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 3824.172752] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3824.181255] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 3824.187351] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 3824.191778] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 3824.197479] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3824.204642] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 3824.210792] Call Trace: [ 3824.230566] [ 3824.235717] ? show_regs.cold.9+0x22/0x2f [ 3824.247477] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3824.249552] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3824.258733] ? watchdog+0x40/0x40 [ 3824.261721] ? hrtimer_interrupt+0x128/0x2c0 [ 3824.265295] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3824.285758] ? apic_timer_interrupt+0xf/0x20 [ 3824.287625] [ 3824.306778] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3824.309148] do_raw_spin_lock+0x177/0x190 [ 3824.327391] _raw_spin_lock+0x21/0x30 [ 3824.329004] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3824.341513] ? do_raw_spin_unlock+0x75/0x190 [ 3824.348569] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3824.356512] ? do_wait_intr+0xf0/0xf0 [ 3824.358079] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3824.377248] mdc_close+0x53b/0xd70 [mdc] [ 3824.378993] lmv_close+0x18d/0x490 [lmv] [ 3824.380794] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 3824.383305] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3824.385867] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3824.388111] ll_md_close+0x274/0x800 [lustre] [ 3824.390255] ll_file_release+0x128/0xc50 [lustre] [ 3824.392518] __fput+0x10d/0x440 [ 3824.393937] ____fput+0x12/0x20 [ 3824.395401] task_work_run+0xc6/0x110 [ 3824.397026] exit_to_usermode_loop+0x1dd/0x1f0 [ 3824.398821] do_syscall_64+0x3d6/0x440 [ 3824.400614] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3824.402910] RIP: 0033:0x7ff54af45cd8 [ 3824.404554] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3824.413049] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3824.416594] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 3824.419920] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 3824.422983] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 3824.426099] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 3824.429492] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3826.655131] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1867s! [ 3826.659671] Showing busy workqueues and worker pools: [ 3826.662429] workqueue events: flags=0x0 [ 3826.664629] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3826.668608] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 3826.673776] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3826.680407] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 3826.692260] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3826.699227] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 3826.704665] workqueue events_long: flags=0x0 [ 3826.708564] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3826.714406] pending: lu_objects_flush [obdclass] [ 3826.719974] workqueue events_unbound: flags=0x2 [ 3826.724184] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3826.728034] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3826.733675] workqueue events_power_efficient: flags=0x80 [ 3826.737226] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3826.747654] pending: neigh_periodic_work [ 3826.752805] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3826.759053] pending: do_cache_clean, check_lifetime [ 3826.764661] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3826.772868] pending: neigh_periodic_work [ 3826.777767] workqueue writeback: flags=0x4a [ 3826.781846] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3826.786947] pending: wb_workfn, wb_workfn [ 3826.788727] workqueue xprtiod: flags=0xa [ 3826.790216] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3826.797771] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 3826.802798] workqueue ipv6_addrconf: flags=0x40008 [ 3826.805822] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3826.810405] pending: addrconf_verify_work [ 3826.813346] workqueue knbd0-recv: flags=0x1a [ 3826.818222] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3826.822545] in-flight: 657:recv_work [ 3826.824799] workqueue ptlrpc_pinger: flags=0x2 [ 3826.827626] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3826.831082] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 3826.838585] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1867s workers=4 idle: 20831 12642 28859 [ 3826.844467] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3817s workers=2 idle: 659 [ 3828.059417] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 3828.065361] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3828.088190] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3828.093281] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3828.098138] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3828.101162] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3828.104215] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3828.112794] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3828.116703] RAX: 0000000000004cb7 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 3828.120480] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 3828.124850] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 3828.128214] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 3828.131537] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 3828.135539] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 3828.139915] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3828.142885] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 3828.147076] Call Trace: [ 3828.148353] [ 3828.149189] ? show_regs.cold.9+0x22/0x2f [ 3828.151442] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3828.154568] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3828.157604] ? watchdog+0x40/0x40 [ 3828.159440] ? hrtimer_interrupt+0x128/0x2c0 [ 3828.161791] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3828.164119] ? apic_timer_interrupt+0xf/0x20 [ 3828.166381] [ 3828.168144] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3828.170619] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3828.173316] do_raw_spin_lock+0x177/0x190 [ 3828.174592] _raw_spin_lock+0x21/0x30 [ 3828.176089] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3828.178996] process_one_work+0x2c8/0x700 [ 3828.180685] worker_thread+0x296/0x6e0 [ 3828.183004] ? rescuer_thread+0x570/0x570 [ 3828.185114] kthread+0x1d1/0x200 [ 3828.187033] ? set_kthread_struct+0x70/0x70 [ 3828.188911] ret_from_fork+0x1f/0x30 [ 3840.062603] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [ 3840.068542] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3840.130124] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3840.137924] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3840.143255] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3840.146852] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3840.156501] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3840.160361] RAX: 0000000000003698 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 3840.163941] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 3840.167712] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3840.171221] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 3840.175535] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 3840.179248] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 3840.185434] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3840.188355] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 3840.192592] Call Trace: [ 3840.194387] [ 3840.195171] ? show_regs.cold.9+0x22/0x2f [ 3840.198830] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3840.201978] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3840.204951] ? watchdog+0x40/0x40 [ 3840.207454] ? hrtimer_interrupt+0x128/0x2c0 [ 3840.209894] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3840.212828] ? apic_timer_interrupt+0xf/0x20 [ 3840.215771] [ 3840.216913] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3840.219939] do_raw_spin_lock+0x177/0x190 [ 3840.222314] _raw_spin_lock+0x21/0x30 [ 3840.224611] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3840.227792] ? do_raw_spin_unlock+0x75/0x190 [ 3840.230116] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3840.233078] ? do_wait_intr+0xf0/0xf0 [ 3840.235061] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3840.238513] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3840.242591] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3840.246874] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3840.250003] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3840.253299] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3840.255754] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3840.258853] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3840.261517] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3840.265745] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3840.268926] lmv_intent_open+0x338/0x1150 [lmv] [ 3840.270540] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3840.274015] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3840.276883] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3840.279140] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3840.281350] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3840.285883] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3840.289821] ? do_raw_spin_unlock+0x75/0x190 [ 3840.296459] ? _raw_spin_unlock+0x12/0x30 [ 3840.299450] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3840.302342] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3840.305608] ll_atomic_open+0x36a/0xd70 [lustre] [ 3840.308101] lookup_open+0x356/0xb40 [ 3840.309621] do_last+0x313/0xfc0 [ 3840.311559] ? nd_jump_root+0xe5/0x160 [ 3840.313738] ? path_init+0x437/0x520 [ 3840.315865] path_openat+0xf7/0x500 [ 3840.317868] ? do_raw_spin_unlock+0x75/0x190 [ 3840.320247] do_filp_open+0x99/0x140 [ 3840.322499] ? getname_flags+0x6e/0x330 [ 3840.324957] ? __check_object_size+0xff/0x256 [ 3840.327282] ? do_raw_spin_unlock+0x75/0x190 [ 3840.329611] ? _raw_spin_unlock+0x12/0x30 [ 3840.331753] do_sys_openat2+0x2b4/0x410 [ 3840.333783] do_sys_open+0x73/0xa0 [ 3840.335483] __x64_sys_openat+0x24/0x30 [ 3840.337745] do_syscall_64+0xc1/0x440 [ 3840.339718] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3840.342632] RIP: 0033:0x7ff54af45332 [ 3840.344765] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3840.355360] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3840.358979] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 3840.363132] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 3840.367020] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 3840.370339] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 3840.374917] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 3852.055044] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 3852.060363] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 3852.062793] Modules linked in: [ 3852.067113] Modules linked in: lustre(O) osp(O) [ 3852.069219] lustre(O) osp(O) ofd(O) lod(O) [ 3852.071319] ofd(O) lod(O) mdt(O) [ 3852.083171] mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3852.083241] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3852.083248] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3852.083251] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3852.083266] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3852.083271] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3852.083276] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 3852.083278] RDX: 0000000000001d3a RSI: 0000000000000003 RDI: ffff96848811cae0 [ 3852.083281] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 3852.083283] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 3852.083285] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3852.083289] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 3852.083292] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3852.083294] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 3852.083303] Call Trace: [ 3852.083309] [ 3852.083315] ? show_regs.cold.9+0x22/0x2f [ 3852.083324] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3852.083330] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3852.083334] ? watchdog+0x40/0x40 [ 3852.083339] ? hrtimer_interrupt+0x128/0x2c0 [ 3852.083343] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3852.083351] ? apic_timer_interrupt+0xf/0x20 [ 3852.083356] [ 3852.083359] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3852.083364] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3852.083368] do_raw_spin_lock+0x177/0x190 [ 3852.083373] _raw_spin_lock+0x21/0x30 [ 3852.083379] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3852.083663] obd_mod_free+0x46/0x140 [ptlrpc] [ 3852.083850] mdc_commit_open+0x91/0xb0 [mdc] [ 3852.083885] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3852.084091] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3852.084283] after_reply+0xe2c/0x1780 [ptlrpc] [ 3852.084476] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3852.084672] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3852.084870] ? do_wait_intr+0xf0/0xf0 [ 3852.084880] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3852.129637] mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl [ 3852.146996] kthread+0x1d1/0x200 [ 3852.147014] ? set_kthread_struct+0x70/0x70 [ 3852.147019] ret_from_fork+0x1f/0x30 [ 3852.285788] pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3852.295885] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3852.302833] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3852.307002] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3852.309372] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3852.317977] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3852.321338] RAX: 0000000000004084 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 3852.324342] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 3852.327805] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3852.330940] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 3852.334125] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 3852.337167] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 3852.340549] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3852.344385] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 3852.349342] Call Trace: [ 3852.350764] [ 3852.351709] ? show_regs.cold.9+0x22/0x2f [ 3852.353224] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3852.355186] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3852.357039] ? watchdog+0x40/0x40 [ 3852.358448] ? hrtimer_interrupt+0x128/0x2c0 [ 3852.360449] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3852.362341] ? apic_timer_interrupt+0xf/0x20 [ 3852.364206] [ 3852.365147] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3852.367107] do_raw_spin_lock+0x177/0x190 [ 3852.368702] _raw_spin_lock+0x21/0x30 [ 3852.370192] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3852.372571] ? do_raw_spin_unlock+0x75/0x190 [ 3852.374231] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3852.376662] ? do_wait_intr+0xf0/0xf0 [ 3852.378172] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3852.381240] mdc_close+0x53b/0xd70 [mdc] [ 3852.382971] lmv_close+0x18d/0x490 [lmv] [ 3852.384699] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 3852.387229] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3852.389646] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3852.391665] ll_md_close+0x274/0x800 [lustre] [ 3852.393639] ll_file_release+0x128/0xc50 [lustre] [ 3852.395723] __fput+0x10d/0x440 [ 3852.396988] ____fput+0x12/0x20 [ 3852.398207] task_work_run+0xc6/0x110 [ 3852.400052] exit_to_usermode_loop+0x1dd/0x1f0 [ 3852.401647] do_syscall_64+0x3d6/0x440 [ 3852.402858] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3852.405056] RIP: 0033:0x7ff54af45cd8 [ 3852.406452] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3852.414135] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3852.417346] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 3852.420260] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 3852.423358] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 3852.426560] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 3852.429867] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3856.059165] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 3856.062232] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3856.094023] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3856.103298] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3856.107938] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3856.111658] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3856.118990] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3856.147816] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3856.166065] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 3856.168557] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 3856.183555] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 3856.199665] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 3856.210656] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 3856.217711] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 3856.226056] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3856.234603] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 3856.242210] Call Trace: [ 3856.243241] [ 3856.244605] ? show_regs.cold.9+0x22/0x2f [ 3856.246671] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3856.248375] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3856.249914] ? watchdog+0x40/0x40 [ 3856.252801] ? hrtimer_interrupt+0x128/0x2c0 [ 3856.256319] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3856.258823] ? apic_timer_interrupt+0xf/0x20 [ 3856.261217] [ 3856.262311] ? native_safe_halt+0x2e/0x50 [ 3856.264639] kvm_wait+0xcd/0xe0 [ 3856.266331] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3856.268838] do_raw_spin_lock+0x177/0x190 [ 3856.271109] _raw_spin_lock+0x21/0x30 [ 3856.273285] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3856.275844] process_one_work+0x2c8/0x700 [ 3856.277770] worker_thread+0x296/0x6e0 [ 3856.279559] ? rescuer_thread+0x570/0x570 [ 3856.281227] kthread+0x1d1/0x200 [ 3856.283677] ? set_kthread_struct+0x70/0x70 [ 3856.285592] ret_from_fork+0x1f/0x30 [ 3856.863309] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1897s! [ 3856.870053] Showing busy workqueues and worker pools: [ 3856.875261] workqueue events: flags=0x0 [ 3856.880238] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3856.886825] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 3856.899334] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3856.904326] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 3856.913769] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3856.920484] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 3856.924720] workqueue events_long: flags=0x0 [ 3856.926838] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3856.930174] pending: lu_objects_flush [obdclass] [ 3856.934781] workqueue events_unbound: flags=0x2 [ 3856.938533] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3856.940721] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3856.947269] workqueue events_power_efficient: flags=0x80 [ 3856.950641] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3856.954795] pending: neigh_periodic_work [ 3856.957155] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3856.963438] pending: do_cache_clean, check_lifetime [ 3856.973068] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3856.978371] pending: neigh_periodic_work [ 3856.983355] workqueue writeback: flags=0x4a [ 3856.987249] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3856.991279] pending: wb_workfn, wb_workfn [ 3857.000841] workqueue xprtiod: flags=0xa [ 3857.006014] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3857.010438] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 3857.018303] workqueue ipv6_addrconf: flags=0x40008 [ 3857.022547] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3857.028371] pending: addrconf_verify_work [ 3857.032482] workqueue knbd0-recv: flags=0x1a [ 3857.035099] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3857.040097] in-flight: 657:recv_work [ 3857.043777] workqueue ptlrpc_pinger: flags=0x2 [ 3857.047323] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3857.050182] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 3857.054827] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1898s workers=4 idle: 20831 12642 28859 [ 3857.061299] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3847s workers=2 idle: 659 [ 3868.064096] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 3868.070053] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3868.172172] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3868.177850] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3868.199900] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3868.205778] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3868.237376] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3868.239740] RAX: 00000000000010d5 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 3868.255190] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 3868.269877] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3868.277014] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 3868.300474] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 3868.303084] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 3868.323682] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3868.332849] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 3868.341596] Call Trace: [ 3868.342482] [ 3868.343172] ? show_regs.cold.9+0x22/0x2f [ 3868.344469] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3868.351397] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3868.359502] ? watchdog+0x40/0x40 [ 3868.360699] ? hrtimer_interrupt+0x128/0x2c0 [ 3868.365800] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3868.373686] ? apic_timer_interrupt+0xf/0x20 [ 3868.379405] [ 3868.382082] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3868.387221] do_raw_spin_lock+0x177/0x190 [ 3868.388527] _raw_spin_lock+0x21/0x30 [ 3868.390142] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3868.396637] ? do_raw_spin_unlock+0x75/0x190 [ 3868.398713] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3868.408685] ? do_wait_intr+0xf0/0xf0 [ 3868.414234] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3868.420313] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3868.422613] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3868.429915] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3868.433191] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3868.436003] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3868.441947] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3868.445148] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3868.448784] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3868.454273] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3868.458007] lmv_intent_open+0x338/0x1150 [lmv] [ 3868.462104] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3868.467224] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3868.476111] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3868.481838] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3868.486125] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3868.495753] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3868.504107] ? do_raw_spin_unlock+0x75/0x190 [ 3868.506392] ? _raw_spin_unlock+0x12/0x30 [ 3868.513171] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3868.522070] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3868.525399] ll_atomic_open+0x36a/0xd70 [lustre] [ 3868.534244] lookup_open+0x356/0xb40 [ 3868.537111] do_last+0x313/0xfc0 [ 3868.541670] ? nd_jump_root+0xe5/0x160 [ 3868.544305] ? path_init+0x437/0x520 [ 3868.550637] path_openat+0xf7/0x500 [ 3868.552326] ? do_raw_spin_unlock+0x75/0x190 [ 3868.558703] do_filp_open+0x99/0x140 [ 3868.561067] ? getname_flags+0x6e/0x330 [ 3868.564825] ? __check_object_size+0xff/0x256 [ 3868.567082] ? do_raw_spin_unlock+0x75/0x190 [ 3868.569219] ? _raw_spin_unlock+0x12/0x30 [ 3868.571124] do_sys_openat2+0x2b4/0x410 [ 3868.575132] do_sys_open+0x73/0xa0 [ 3868.581765] __x64_sys_openat+0x24/0x30 [ 3868.584068] do_syscall_64+0xc1/0x440 [ 3868.589649] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3868.593321] RIP: 0033:0x7ff54af45332 [ 3868.595959] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3868.605697] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3868.609174] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 3868.616793] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 3868.624175] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 3868.632035] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 3868.642528] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 3880.058058] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 3880.065255] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3880.083544] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3880.088259] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3880.092220] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3880.094213] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3880.102502] RSP: 0018:ffffaf9fc1543ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3880.106177] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 3880.109285] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 3880.112386] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3880.115379] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 3880.118318] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 3880.121116] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 3880.124341] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3880.126566] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 3880.129827] Call Trace: [ 3880.130882] [ 3880.132066] ? show_regs.cold.9+0x22/0x2f [ 3880.133581] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3880.135402] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3880.137502] ? watchdog+0x40/0x40 [ 3880.138842] ? hrtimer_interrupt+0x128/0x2c0 [ 3880.140897] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3880.142854] ? apic_timer_interrupt+0xf/0x20 [ 3880.145201] [ 3880.146147] ? native_safe_halt+0x2e/0x50 [ 3880.147891] kvm_wait+0xcd/0xe0 [ 3880.149358] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3880.151126] do_raw_spin_lock+0x177/0x190 [ 3880.152725] _raw_spin_lock+0x21/0x30 [ 3880.154251] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3880.157176] ? do_raw_spin_unlock+0x75/0x190 [ 3880.159069] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3880.161432] ? do_wait_intr+0xf0/0xf0 [ 3880.163182] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3880.166885] mdc_close+0x53b/0xd70 [mdc] [ 3880.169153] lmv_close+0x18d/0x490 [lmv] [ 3880.171070] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 3880.174064] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3880.176433] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3880.179369] ll_md_close+0x274/0x800 [lustre] [ 3880.181346] ll_file_release+0x128/0xc50 [lustre] [ 3880.183810] __fput+0x10d/0x440 [ 3880.185162] ____fput+0x12/0x20 [ 3880.186942] task_work_run+0xc6/0x110 [ 3880.188523] exit_to_usermode_loop+0x1dd/0x1f0 [ 3880.190586] do_syscall_64+0x3d6/0x440 [ 3880.192455] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3880.194836] RIP: 0033:0x7ff54af45cd8 [ 3880.196411] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3880.205398] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3880.208860] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 3880.211926] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 3880.215358] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 3880.219273] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 3880.222552] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3884.055014] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [ 3884.059025] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 3884.059912] Modules linked in: [ 3884.066480] Modules linked in: [ 3884.066482] lustre(O) osp(O) ofd(O) [ 3884.071242] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3884.071311] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3884.071318] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3884.071325] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3884.072903] lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3884.073067] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3884.073074] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3884.073077] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3884.073091] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3884.073095] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3884.073099] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 3884.073102] RDX: 0000000000007792 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 3884.073105] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 3884.073107] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 3884.073109] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3884.073113] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 3884.073116] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3884.073118] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 3884.073126] Call Trace: [ 3884.073131] [ 3884.073136] ? show_regs.cold.9+0x22/0x2f [ 3884.073146] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3884.073153] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3884.073157] ? watchdog+0x40/0x40 [ 3884.073162] ? hrtimer_interrupt+0x128/0x2c0 [ 3884.073166] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3884.073188] ? apic_timer_interrupt+0xf/0x20 [ 3884.073193] [ 3884.073196] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3884.073201] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3884.073206] do_raw_spin_lock+0x177/0x190 [ 3884.073210] _raw_spin_lock+0x21/0x30 [ 3884.073216] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3884.073597] obd_mod_free+0x46/0x140 [ptlrpc] [ 3884.073899] mdc_commit_open+0x91/0xb0 [mdc] [ 3884.074021] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3884.074405] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3884.074607] after_reply+0xe2c/0x1780 [ptlrpc] [ 3884.077272] [ 3884.077279] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3884.102782] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3884.103363] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3884.103710] ? do_wait_intr+0xf0/0xf0 [ 3884.103723] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3884.104012] kthread+0x1d1/0x200 [ 3884.104018] ? set_kthread_struct+0x70/0x70 [ 3884.104022] ret_from_fork+0x1f/0x30 [ 3884.251902] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3884.259098] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3884.262028] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 3884.264522] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 3884.267380] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 3884.270260] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 3884.273021] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 3884.275852] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 3884.278897] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3884.281050] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 3884.283764] Call Trace: [ 3884.284577] [ 3884.285421] ? show_regs.cold.9+0x22/0x2f [ 3884.287043] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3884.288899] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3884.290527] ? watchdog+0x40/0x40 [ 3884.292060] ? hrtimer_interrupt+0x128/0x2c0 [ 3884.293738] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3884.295474] ? apic_timer_interrupt+0xf/0x20 [ 3884.297165] [ 3884.298031] ? native_safe_halt+0x2e/0x50 [ 3884.299484] kvm_wait+0xcd/0xe0 [ 3884.300790] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3884.302774] do_raw_spin_lock+0x177/0x190 [ 3884.304287] _raw_spin_lock+0x21/0x30 [ 3884.305854] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3884.308067] process_one_work+0x2c8/0x700 [ 3884.309495] worker_thread+0x296/0x6e0 [ 3884.310716] ? rescuer_thread+0x570/0x570 [ 3884.312209] kthread+0x1d1/0x200 [ 3884.313447] ? set_kthread_struct+0x70/0x70 [ 3884.315241] ret_from_fork+0x1f/0x30 [ 3887.074403] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1928s! [ 3887.082773] Showing busy workqueues and worker pools: [ 3887.084344] workqueue events: flags=0x0 [ 3887.085349] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3887.086999] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 3887.099769] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3887.101776] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 3887.106875] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3887.109312] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 3887.112109] workqueue events_long: flags=0x0 [ 3887.113710] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3887.116094] pending: lu_objects_flush [obdclass] [ 3887.117760] workqueue events_unbound: flags=0x2 [ 3887.119628] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3887.122266] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3887.126816] workqueue events_power_efficient: flags=0x80 [ 3887.129454] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3887.135357] pending: neigh_periodic_work [ 3887.138213] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3887.142281] pending: do_cache_clean, check_lifetime [ 3887.144093] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3887.145684] pending: neigh_periodic_work [ 3887.146794] workqueue writeback: flags=0x4a [ 3887.148343] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3887.150555] pending: wb_workfn, wb_workfn [ 3887.152189] workqueue xprtiod: flags=0xa [ 3887.153128] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3887.156930] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 3887.159348] workqueue ipv6_addrconf: flags=0x40008 [ 3887.161288] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3887.164245] pending: addrconf_verify_work [ 3887.165928] workqueue knbd0-recv: flags=0x1a [ 3887.167677] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3887.170621] in-flight: 657:recv_work [ 3887.177857] workqueue ptlrpc_pinger: flags=0x2 [ 3887.179849] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3887.182309] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 3887.185981] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1928s workers=4 idle: 20831 12642 28859 [ 3887.189348] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3877s workers=2 idle: 659 [ 3896.062693] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 3896.075006] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3896.112772] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3896.133250] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3896.142244] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3896.149200] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3896.178013] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3896.192397] RAX: 0000000000007318 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 3896.197737] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 3896.206017] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3896.214861] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 3896.218733] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 3896.232276] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 3896.242038] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3896.252352] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 3896.256727] Call Trace: [ 3896.258926] [ 3896.260910] ? show_regs.cold.9+0x22/0x2f [ 3896.267686] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3896.271388] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3896.276190] ? watchdog+0x40/0x40 [ 3896.279730] ? hrtimer_interrupt+0x128/0x2c0 [ 3896.282989] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3896.289117] ? apic_timer_interrupt+0xf/0x20 [ 3896.300674] [ 3896.301469] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3896.307535] do_raw_spin_lock+0x177/0x190 [ 3896.310540] _raw_spin_lock+0x21/0x30 [ 3896.315735] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3896.320329] ? do_raw_spin_unlock+0x75/0x190 [ 3896.324019] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3896.327116] ? do_wait_intr+0xf0/0xf0 [ 3896.330036] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3896.334119] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3896.343842] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3896.356658] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3896.364556] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3896.376615] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3896.391033] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3896.404824] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3896.412053] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3896.422060] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3896.426932] lmv_intent_open+0x338/0x1150 [lmv] [ 3896.431270] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3896.433999] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3896.444844] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3896.464992] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3896.471393] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3896.479130] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3896.488819] ? do_raw_spin_unlock+0x75/0x190 [ 3896.492572] ? _raw_spin_unlock+0x12/0x30 [ 3896.498935] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3896.504404] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3896.513361] ll_atomic_open+0x36a/0xd70 [lustre] [ 3896.523326] lookup_open+0x356/0xb40 [ 3896.532060] do_last+0x313/0xfc0 [ 3896.538556] ? nd_jump_root+0xe5/0x160 [ 3896.545901] ? path_init+0x437/0x520 [ 3896.548925] path_openat+0xf7/0x500 [ 3896.559488] ? do_raw_spin_unlock+0x75/0x190 [ 3896.563023] do_filp_open+0x99/0x140 [ 3896.567005] ? getname_flags+0x6e/0x330 [ 3896.571469] ? __check_object_size+0xff/0x256 [ 3896.575827] ? do_raw_spin_unlock+0x75/0x190 [ 3896.578564] ? _raw_spin_unlock+0x12/0x30 [ 3896.581627] do_sys_openat2+0x2b4/0x410 [ 3896.585867] do_sys_open+0x73/0xa0 [ 3896.589908] __x64_sys_openat+0x24/0x30 [ 3896.591075] do_syscall_64+0xc1/0x440 [ 3896.592321] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3896.598388] RIP: 0033:0x7ff54af45332 [ 3896.601364] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3896.616577] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3896.619814] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 3896.623925] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 3896.638951] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 3896.649488] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 3896.653783] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 3908.060300] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 3908.068013] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3908.116543] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3908.122893] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3908.131808] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3908.136925] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3908.151153] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3908.158222] RAX: 0000000000005732 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 3908.160509] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 3908.170014] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3908.176924] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 3908.183400] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 3908.189652] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 3908.195994] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3908.199949] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 3908.205942] Call Trace: [ 3908.208183] [ 3908.209701] ? show_regs.cold.9+0x22/0x2f [ 3908.215089] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3908.226316] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3908.231605] ? watchdog+0x40/0x40 [ 3908.235858] ? hrtimer_interrupt+0x128/0x2c0 [ 3908.238660] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3908.241606] ? apic_timer_interrupt+0xf/0x20 [ 3908.243124] [ 3908.243925] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3908.250726] do_raw_spin_lock+0x177/0x190 [ 3908.254035] _raw_spin_lock+0x21/0x30 [ 3908.259700] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3908.264420] ? do_raw_spin_unlock+0x75/0x190 [ 3908.265771] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3908.273607] ? do_wait_intr+0xf0/0xf0 [ 3908.278760] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3908.287964] mdc_close+0x53b/0xd70 [mdc] [ 3908.292307] lmv_close+0x18d/0x490 [lmv] [ 3908.295856] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 3908.302995] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3908.309586] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3908.317999] ll_md_close+0x274/0x800 [lustre] [ 3908.321921] ll_file_release+0x128/0xc50 [lustre] [ 3908.325569] __fput+0x10d/0x440 [ 3908.329001] ____fput+0x12/0x20 [ 3908.333640] task_work_run+0xc6/0x110 [ 3908.336349] exit_to_usermode_loop+0x1dd/0x1f0 [ 3908.339662] do_syscall_64+0x3d6/0x440 [ 3908.347978] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3908.356897] RIP: 0033:0x7ff54af45cd8 [ 3908.360645] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3908.386700] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3908.393335] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 3908.406029] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 3908.418908] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 3908.425705] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 3908.438783] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3912.059053] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 3912.065029] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 3912.066905] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3912.066990] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3912.066997] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3912.067004] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3912.069995] RIP: 0010:__pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 3912.070005] Code: d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 05 18 d4 dc 02 01 e0 48 83 05 06 d4 dc 02 01 48 83 05 be d5 dc 02 01 8b 45 08 85 [ 3912.070008] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3912.070011] RAX: 0000000000007c07 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 3912.070013] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 3912.070014] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 3912.070016] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 3912.070017] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 3912.070020] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 3912.070021] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3912.070022] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 3912.070029] Call Trace: [ 3912.070033] [ 3912.070037] ? show_regs.cold.9+0x22/0x2f [ 3912.070044] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3912.070049] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3912.070052] ? watchdog+0x40/0x40 [ 3912.070054] ? hrtimer_interrupt+0x128/0x2c0 [ 3912.070057] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3912.070063] ? apic_timer_interrupt+0xf/0x20 [ 3912.070066] [ 3912.070068] ? __pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 3912.070071] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3912.070074] do_raw_spin_lock+0x177/0x190 [ 3912.070076] _raw_spin_lock+0x21/0x30 [ 3912.070080] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3912.070220] process_one_work+0x2c8/0x700 [ 3912.070224] worker_thread+0x296/0x6e0 [ 3912.070227] ? rescuer_thread+0x570/0x570 [ 3912.070229] kthread+0x1d1/0x200 [ 3912.070232] ? set_kthread_struct+0x70/0x70 [ 3912.070235] ret_from_fork+0x1f/0x30 [ 3912.437070] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3912.506731] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3912.525689] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3912.533958] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3912.549712] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3912.575673] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3912.584967] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 3912.594220] RDX: 00000000000012f6 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 3912.608072] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 3912.617363] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 3912.624677] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3912.630386] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 3912.637018] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3912.642834] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 3912.650904] Call Trace: [ 3912.652550] [ 3912.654939] ? show_regs.cold.9+0x22/0x2f [ 3912.664281] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3912.672736] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3912.678042] ? watchdog+0x40/0x40 [ 3912.680155] ? hrtimer_interrupt+0x128/0x2c0 [ 3912.682082] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3912.686404] ? apic_timer_interrupt+0xf/0x20 [ 3912.690442] [ 3912.691287] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3912.693669] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3912.696199] do_raw_spin_lock+0x177/0x190 [ 3912.698958] _raw_spin_lock+0x21/0x30 [ 3912.701604] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3912.705773] obd_mod_free+0x46/0x140 [ptlrpc] [ 3912.707807] mdc_commit_open+0x91/0xb0 [mdc] [ 3912.709290] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3912.711681] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3912.717126] after_reply+0xe2c/0x1780 [ptlrpc] [ 3912.720885] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3912.724012] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3912.726253] ? do_wait_intr+0xf0/0xf0 [ 3912.728274] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3912.731612] kthread+0x1d1/0x200 [ 3912.733634] ? set_kthread_struct+0x70/0x70 [ 3912.735443] ret_from_fork+0x1f/0x30 [ 3917.279089] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1958s! [ 3917.281651] Showing busy workqueues and worker pools: [ 3917.287642] workqueue events: flags=0x0 [ 3917.288948] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3917.292343] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 3917.300371] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3917.310398] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 3917.317607] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3917.320585] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 3917.323968] workqueue events_long: flags=0x0 [ 3917.328835] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3917.333714] pending: lu_objects_flush [obdclass] [ 3917.338870] workqueue events_unbound: flags=0x2 [ 3917.342769] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3917.349422] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3917.357197] workqueue events_power_efficient: flags=0x80 [ 3917.359184] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3917.370512] pending: neigh_periodic_work [ 3917.372111] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3917.374717] pending: do_cache_clean, check_lifetime [ 3917.376908] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3917.379192] pending: neigh_periodic_work [ 3917.380944] workqueue writeback: flags=0x4a [ 3917.383622] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3917.386761] pending: wb_workfn, wb_workfn [ 3917.389248] workqueue xprtiod: flags=0xa [ 3917.404995] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3917.410637] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 3917.415176] workqueue ipv6_addrconf: flags=0x40008 [ 3917.417676] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3917.420913] pending: addrconf_verify_work [ 3917.423163] workqueue knbd0-recv: flags=0x1a [ 3917.425043] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3917.433011] in-flight: 657:recv_work [ 3917.440028] workqueue ptlrpc_pinger: flags=0x2 [ 3917.444416] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3917.448884] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 3917.456052] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1958s workers=4 idle: 20831 12642 28859 [ 3917.464154] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3907s workers=2 idle: 659 [ 3924.060036] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 3924.063366] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3924.099512] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3924.106443] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3924.114831] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3924.118938] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3924.132117] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3924.135394] RAX: 0000000000003e81 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 3924.138413] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 3924.141691] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3924.145473] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 3924.149829] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 3924.153805] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 3924.157570] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3924.160767] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 3924.165324] Call Trace: [ 3924.167682] [ 3924.168534] ? show_regs.cold.9+0x22/0x2f [ 3924.170755] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3924.173842] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3924.180149] ? watchdog+0x40/0x40 [ 3924.182429] ? hrtimer_interrupt+0x128/0x2c0 [ 3924.185137] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3924.189441] ? apic_timer_interrupt+0xf/0x20 [ 3924.192042] [ 3924.193469] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3924.196480] do_raw_spin_lock+0x177/0x190 [ 3924.199563] _raw_spin_lock+0x21/0x30 [ 3924.201543] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3924.206632] ? do_raw_spin_unlock+0x75/0x190 [ 3924.211670] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3924.215860] ? do_wait_intr+0xf0/0xf0 [ 3924.218458] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3924.221940] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3924.229248] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3924.233822] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3924.239135] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3924.241927] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3924.246052] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3924.248800] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3924.254092] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3924.257710] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3924.260397] lmv_intent_open+0x338/0x1150 [lmv] [ 3924.263752] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3924.268669] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3924.271946] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3924.274099] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3924.278236] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3924.280562] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3924.282877] ? do_raw_spin_unlock+0x75/0x190 [ 3924.285059] ? _raw_spin_unlock+0x12/0x30 [ 3924.288567] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3924.291856] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3924.295340] ll_atomic_open+0x36a/0xd70 [lustre] [ 3924.302394] lookup_open+0x356/0xb40 [ 3924.307526] do_last+0x313/0xfc0 [ 3924.312111] ? nd_jump_root+0xe5/0x160 [ 3924.314397] ? path_init+0x437/0x520 [ 3924.319898] path_openat+0xf7/0x500 [ 3924.323484] ? do_raw_spin_unlock+0x75/0x190 [ 3924.325827] do_filp_open+0x99/0x140 [ 3924.329864] ? getname_flags+0x6e/0x330 [ 3924.332051] ? __check_object_size+0xff/0x256 [ 3924.334174] ? do_raw_spin_unlock+0x75/0x190 [ 3924.338014] ? _raw_spin_unlock+0x12/0x30 [ 3924.341064] do_sys_openat2+0x2b4/0x410 [ 3924.343668] do_sys_open+0x73/0xa0 [ 3924.347698] __x64_sys_openat+0x24/0x30 [ 3924.349694] do_syscall_64+0xc1/0x440 [ 3924.352343] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3924.355701] RIP: 0033:0x7ff54af45332 [ 3924.358511] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3924.379280] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3924.387621] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 3924.396920] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 3924.401848] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 3924.411156] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 3924.415440] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 3936.058015] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 3936.062033] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3936.098521] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3936.111949] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3936.120921] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3936.128040] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3936.153598] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3936.163610] RAX: 00000000000013ce RBX: ffff96848811cae0 RCX: 0000000000000001 [ 3936.171025] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 3936.176275] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3936.185056] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 3936.194493] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 3936.199549] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 3936.211565] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3936.222006] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 3936.228711] Call Trace: [ 3936.235883] [ 3936.237273] ? show_regs.cold.9+0x22/0x2f [ 3936.242366] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3936.246949] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3936.252944] ? watchdog+0x40/0x40 [ 3936.259534] ? hrtimer_interrupt+0x128/0x2c0 [ 3936.268464] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3936.293187] ? apic_timer_interrupt+0xf/0x20 [ 3936.300693] [ 3936.301530] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3936.310537] do_raw_spin_lock+0x177/0x190 [ 3936.323302] _raw_spin_lock+0x21/0x30 [ 3936.334562] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3936.346566] ? do_raw_spin_unlock+0x75/0x190 [ 3936.363406] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3936.368747] ? do_wait_intr+0xf0/0xf0 [ 3936.371978] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3936.387270] mdc_close+0x53b/0xd70 [mdc] [ 3936.392233] lmv_close+0x18d/0x490 [lmv] [ 3936.402435] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 3936.423209] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3936.429733] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3936.442783] ll_md_close+0x274/0x800 [lustre] [ 3936.451566] ll_file_release+0x128/0xc50 [lustre] [ 3936.468849] __fput+0x10d/0x440 [ 3936.470692] ____fput+0x12/0x20 [ 3936.478065] task_work_run+0xc6/0x110 [ 3936.483774] exit_to_usermode_loop+0x1dd/0x1f0 [ 3936.492121] do_syscall_64+0x3d6/0x440 [ 3936.500568] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3936.518020] RIP: 0033:0x7ff54af45cd8 [ 3936.522736] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3936.545629] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3936.564685] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 3936.575619] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 3936.589526] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 3936.603662] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 3936.617579] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3940.059026] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 3940.061241] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3940.075097] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3940.078129] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3940.080534] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3940.082544] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3940.084015] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3940.089460] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3940.091660] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 3940.093624] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 3940.095490] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 3940.097561] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 3940.100019] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 3940.102591] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 3940.104787] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3940.106424] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 3940.108406] Call Trace: [ 3940.109156] [ 3940.109739] ? show_regs.cold.9+0x22/0x2f [ 3940.111544] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3940.113189] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3940.114561] ? watchdog+0x40/0x40 [ 3940.116127] ? hrtimer_interrupt+0x128/0x2c0 [ 3940.118206] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3940.119471] ? apic_timer_interrupt+0xf/0x20 [ 3940.120615] [ 3940.121264] ? native_safe_halt+0x2e/0x50 [ 3940.122349] kvm_wait+0xcd/0xe0 [ 3940.123281] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3940.124779] do_raw_spin_lock+0x177/0x190 [ 3940.125811] _raw_spin_lock+0x21/0x30 [ 3940.126827] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3940.129168] process_one_work+0x2c8/0x700 [ 3940.130168] worker_thread+0x296/0x6e0 [ 3940.131177] ? rescuer_thread+0x570/0x570 [ 3940.132159] kthread+0x1d1/0x200 [ 3940.132921] ? set_kthread_struct+0x70/0x70 [ 3940.134018] ret_from_fork+0x1f/0x30 [ 3944.055355] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [ 3944.069393] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3944.166506] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3944.179505] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3944.205044] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3944.208014] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3944.254864] RSP: 0000:ffffaf9fc1b97bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3944.264917] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 3944.278672] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 3944.289976] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 3944.298284] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 3944.309541] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3944.324733] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 3944.333433] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3944.340697] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 3944.355600] Call Trace: [ 3944.357878] [ 3944.360355] ? show_regs.cold.9+0x22/0x2f [ 3944.363993] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3944.368367] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3944.377534] ? watchdog+0x40/0x40 [ 3944.381718] ? hrtimer_interrupt+0x128/0x2c0 [ 3944.387039] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3944.389099] ? apic_timer_interrupt+0xf/0x20 [ 3944.402261] [ 3944.409516] ? native_safe_halt+0x2e/0x50 [ 3944.420428] kvm_wait+0xcd/0xe0 [ 3944.432237] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3944.440893] do_raw_spin_lock+0x177/0x190 [ 3944.442658] _raw_spin_lock+0x21/0x30 [ 3944.459495] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3944.461779] obd_mod_free+0x46/0x140 [ptlrpc] [ 3944.476732] mdc_commit_open+0x91/0xb0 [mdc] [ 3944.486728] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3944.510632] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3944.541422] after_reply+0xe2c/0x1780 [ptlrpc] [ 3944.543679] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3944.566528] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3944.576520] ? do_wait_intr+0xf0/0xf0 [ 3944.578101] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3944.606618] kthread+0x1d1/0x200 [ 3944.620528] ? set_kthread_struct+0x70/0x70 [ 3944.650604] ret_from_fork+0x1f/0x30 [ 3947.487047] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1988s! [ 3947.489174] Showing busy workqueues and worker pools: [ 3947.490439] workqueue events: flags=0x0 [ 3947.491433] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3947.493817] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 3947.519169] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3947.522091] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 3947.531702] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3947.534798] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 3947.543385] workqueue events_long: flags=0x0 [ 3947.544834] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3947.546512] pending: lu_objects_flush [obdclass] [ 3947.549790] workqueue events_unbound: flags=0x2 [ 3947.551395] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3947.556425] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3947.570672] workqueue events_power_efficient: flags=0x80 [ 3947.577473] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3947.597442] pending: neigh_periodic_work [ 3947.602308] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3947.605960] pending: do_cache_clean, check_lifetime [ 3947.608174] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3947.615585] pending: neigh_periodic_work [ 3947.616935] workqueue writeback: flags=0x4a [ 3947.635786] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3947.655937] pending: wb_workfn, wb_workfn [ 3947.657510] workqueue xprtiod: flags=0xa [ 3947.658883] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3947.676792] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 3947.684339] workqueue ipv6_addrconf: flags=0x40008 [ 3947.694081] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3947.698821] pending: addrconf_verify_work [ 3947.702975] workqueue knbd0-recv: flags=0x1a [ 3947.705689] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3947.708706] in-flight: 657:recv_work [ 3947.710179] workqueue ptlrpc_pinger: flags=0x2 [ 3947.715654] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3947.719604] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 3947.724744] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1988s workers=4 idle: 20831 12642 28859 [ 3947.729602] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3937s workers=2 idle: 659 [ 3952.063682] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 3952.067841] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3952.105122] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3952.120383] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3952.129704] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3952.133574] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3952.147228] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3952.151704] RAX: 0000000000005786 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 3952.156386] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 3952.160678] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3952.164307] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 3952.172252] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 3952.181300] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 3952.189876] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3952.196528] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 3952.202582] Call Trace: [ 3952.206262] [ 3952.208856] ? show_regs.cold.9+0x22/0x2f [ 3952.213888] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3952.218414] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3952.225913] ? watchdog+0x40/0x40 [ 3952.226893] ? hrtimer_interrupt+0x128/0x2c0 [ 3952.230809] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3952.237550] ? apic_timer_interrupt+0xf/0x20 [ 3952.242682] [ 3952.246124] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3952.251590] do_raw_spin_lock+0x177/0x190 [ 3952.258365] _raw_spin_lock+0x21/0x30 [ 3952.264545] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3952.281704] ? do_raw_spin_unlock+0x75/0x190 [ 3952.292567] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3952.302662] ? do_wait_intr+0xf0/0xf0 [ 3952.303847] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3952.317237] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3952.328026] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3952.338581] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3952.347283] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3952.357927] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3952.361140] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3952.366121] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3952.372034] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3952.379368] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3952.386304] lmv_intent_open+0x338/0x1150 [lmv] [ 3952.391247] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3952.398695] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3952.402333] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3952.403842] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3952.411137] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3952.418240] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3952.425896] ? do_raw_spin_unlock+0x75/0x190 [ 3952.431793] ? _raw_spin_unlock+0x12/0x30 [ 3952.435836] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3952.441284] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3952.448166] ll_atomic_open+0x36a/0xd70 [lustre] [ 3952.454576] lookup_open+0x356/0xb40 [ 3952.456137] do_last+0x313/0xfc0 [ 3952.467248] ? nd_jump_root+0xe5/0x160 [ 3952.477739] ? path_init+0x437/0x520 [ 3952.488618] path_openat+0xf7/0x500 [ 3952.493306] ? do_raw_spin_unlock+0x75/0x190 [ 3952.494879] do_filp_open+0x99/0x140 [ 3952.510335] ? getname_flags+0x6e/0x330 [ 3952.523614] ? __check_object_size+0xff/0x256 [ 3952.524937] ? do_raw_spin_unlock+0x75/0x190 [ 3952.526405] ? _raw_spin_unlock+0x12/0x30 [ 3952.543861] do_sys_openat2+0x2b4/0x410 [ 3952.545225] do_sys_open+0x73/0xa0 [ 3952.558425] __x64_sys_openat+0x24/0x30 [ 3952.562711] do_syscall_64+0xc1/0x440 [ 3952.567863] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3952.574644] RIP: 0033:0x7ff54af45332 [ 3952.581231] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3952.590060] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3952.595692] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 3952.601589] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 3952.616203] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 3952.632666] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 3952.648087] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 3964.071467] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 3964.091172] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3964.137867] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3964.146620] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3964.152088] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3964.156372] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3964.172650] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3964.177876] RAX: 0000000000007403 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 3964.180706] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 3964.184359] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3964.193522] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 3964.200559] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 3964.204538] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 3964.208585] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3964.214801] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 3964.219107] Call Trace: [ 3964.220895] [ 3964.221655] ? show_regs.cold.9+0x22/0x2f [ 3964.226362] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3964.228589] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3964.230750] ? watchdog+0x40/0x40 [ 3964.232744] ? hrtimer_interrupt+0x128/0x2c0 [ 3964.235053] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3964.237643] ? apic_timer_interrupt+0xf/0x20 [ 3964.239713] [ 3964.240746] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3964.242620] do_raw_spin_lock+0x177/0x190 [ 3964.244713] _raw_spin_lock+0x21/0x30 [ 3964.246535] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3964.249681] ? do_raw_spin_unlock+0x75/0x190 [ 3964.252014] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3964.255492] ? do_wait_intr+0xf0/0xf0 [ 3964.260941] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3964.268820] mdc_close+0x53b/0xd70 [mdc] [ 3964.272713] lmv_close+0x18d/0x490 [lmv] [ 3964.274871] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 3964.277366] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3964.278903] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3964.280154] ll_md_close+0x274/0x800 [lustre] [ 3964.281756] ll_file_release+0x128/0xc50 [lustre] [ 3964.283309] __fput+0x10d/0x440 [ 3964.284323] ____fput+0x12/0x20 [ 3964.286681] task_work_run+0xc6/0x110 [ 3964.288593] exit_to_usermode_loop+0x1dd/0x1f0 [ 3964.290083] do_syscall_64+0x3d6/0x440 [ 3964.291423] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3964.293092] RIP: 0033:0x7ff54af45cd8 [ 3964.294392] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3964.303726] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3964.306043] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 3964.310083] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 3964.320340] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 3964.323703] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 3964.330095] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3968.059272] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 3968.064568] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3968.112742] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3968.121954] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3968.129314] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3968.133240] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3968.147516] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3968.160443] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3968.167265] RAX: 0000000000007f0a RBX: ffff96848811cae0 RCX: 0000000000000001 [ 3968.170999] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 3968.176474] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 3968.181928] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 3968.188521] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 3968.194305] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 3968.201895] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3968.205883] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 3968.210115] Call Trace: [ 3968.211510] [ 3968.212864] ? show_regs.cold.9+0x22/0x2f [ 3968.219200] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3968.223947] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3968.225216] ? watchdog+0x40/0x40 [ 3968.233292] ? hrtimer_interrupt+0x128/0x2c0 [ 3968.237363] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3968.240976] ? apic_timer_interrupt+0xf/0x20 [ 3968.244329] [ 3968.246348] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3968.252410] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3968.258721] do_raw_spin_lock+0x177/0x190 [ 3968.264532] _raw_spin_lock+0x21/0x30 [ 3968.268471] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3968.275476] process_one_work+0x2c8/0x700 [ 3968.281657] worker_thread+0x296/0x6e0 [ 3968.284048] ? rescuer_thread+0x570/0x570 [ 3968.291846] kthread+0x1d1/0x200 [ 3968.299319] ? set_kthread_struct+0x70/0x70 [ 3968.306261] ret_from_fork+0x1f/0x30 [ 3972.055058] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [ 3972.060518] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3972.104036] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3972.120317] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3972.129416] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3972.135242] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3972.153516] RSP: 0000:ffffaf9fc1b97bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3972.160612] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 3972.164476] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 3972.168130] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 3972.175547] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 3972.180185] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3972.184744] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 3972.193136] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3972.197336] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 3972.204972] Call Trace: [ 3972.206043] [ 3972.207205] ? show_regs.cold.9+0x22/0x2f [ 3972.210061] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3972.218169] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3972.224553] ? watchdog+0x40/0x40 [ 3972.225655] ? hrtimer_interrupt+0x128/0x2c0 [ 3972.227572] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3972.229678] ? apic_timer_interrupt+0xf/0x20 [ 3972.231524] [ 3972.235347] ? native_safe_halt+0x2e/0x50 [ 3972.242688] kvm_wait+0xcd/0xe0 [ 3972.252400] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3972.255366] do_raw_spin_lock+0x177/0x190 [ 3972.261251] _raw_spin_lock+0x21/0x30 [ 3972.266026] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3972.274527] obd_mod_free+0x46/0x140 [ptlrpc] [ 3972.277841] mdc_commit_open+0x91/0xb0 [mdc] [ 3972.280160] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3972.283624] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3972.287202] after_reply+0xe2c/0x1780 [ptlrpc] [ 3972.291031] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3972.295653] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3972.303162] ? do_wait_intr+0xf0/0xf0 [ 3972.307748] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3972.311362] kthread+0x1d1/0x200 [ 3972.313877] ? set_kthread_struct+0x70/0x70 [ 3972.315955] ret_from_fork+0x1f/0x30 [ 3978.207548] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2019s! [ 3978.220603] Showing busy workqueues and worker pools: [ 3978.229751] workqueue events: flags=0x0 [ 3978.231308] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3978.233745] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 3978.258664] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3978.267926] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 3978.290243] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3978.299478] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 3978.311838] workqueue events_long: flags=0x0 [ 3978.313367] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3978.318581] pending: lu_objects_flush [obdclass] [ 3978.320757] workqueue events_unbound: flags=0x2 [ 3978.324447] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3978.329535] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3978.335913] workqueue events_power_efficient: flags=0x80 [ 3978.339449] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3978.342815] pending: neigh_periodic_work [ 3978.344567] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3978.347633] pending: do_cache_clean, check_lifetime [ 3978.350640] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3978.354492] pending: neigh_periodic_work [ 3978.357391] workqueue writeback: flags=0x4a [ 3978.361335] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3978.363995] pending: wb_workfn, wb_workfn [ 3978.365754] workqueue xprtiod: flags=0xa [ 3978.371382] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3978.374639] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 3978.380177] workqueue ipv6_addrconf: flags=0x40008 [ 3978.383207] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3978.387388] pending: addrconf_verify_work [ 3978.389959] workqueue knbd0-recv: flags=0x1a [ 3978.391623] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3978.395598] in-flight: 657:recv_work [ 3978.398103] workqueue ptlrpc_pinger: flags=0x2 [ 3978.401059] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3978.405071] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 3978.411249] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2019s workers=4 idle: 20831 12642 28859 [ 3978.420438] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3968s workers=2 idle: 659 [ 3980.060315] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 3980.065645] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3980.095435] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3980.103541] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3980.106526] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3980.108974] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3980.116633] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3980.132391] RAX: 0000000000006567 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 3980.138342] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 3980.145013] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3980.148761] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 3980.154942] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 3980.158186] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 3980.167637] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3980.178159] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 3980.179992] Call Trace: [ 3980.181659] [ 3980.182192] ? show_regs.cold.9+0x22/0x2f [ 3980.183211] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3980.185236] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3980.186436] ? watchdog+0x40/0x40 [ 3980.187343] ? hrtimer_interrupt+0x128/0x2c0 [ 3980.188522] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3980.190163] ? apic_timer_interrupt+0xf/0x20 [ 3980.191760] [ 3980.192549] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3980.194809] do_raw_spin_lock+0x177/0x190 [ 3980.196147] _raw_spin_lock+0x21/0x30 [ 3980.198126] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3980.200757] ? do_raw_spin_unlock+0x75/0x190 [ 3980.202511] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3980.204206] ? do_wait_intr+0xf0/0xf0 [ 3980.205433] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3980.207027] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3980.208414] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3980.210648] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3980.212166] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3980.213536] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3980.215192] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3980.216402] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3980.218435] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3980.223791] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3980.232473] lmv_intent_open+0x338/0x1150 [lmv] [ 3980.234127] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3980.242644] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3980.251580] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3980.254464] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3980.257523] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3980.263106] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3980.265893] ? do_raw_spin_unlock+0x75/0x190 [ 3980.268094] ? _raw_spin_unlock+0x12/0x30 [ 3980.270204] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3980.272431] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3980.274787] ll_atomic_open+0x36a/0xd70 [lustre] [ 3980.276274] lookup_open+0x356/0xb40 [ 3980.277096] do_last+0x313/0xfc0 [ 3980.277872] ? nd_jump_root+0xe5/0x160 [ 3980.278535] ? path_init+0x437/0x520 [ 3980.279635] path_openat+0xf7/0x500 [ 3980.281106] ? do_raw_spin_unlock+0x75/0x190 [ 3980.282730] do_filp_open+0x99/0x140 [ 3980.284202] ? getname_flags+0x6e/0x330 [ 3980.285747] ? __check_object_size+0xff/0x256 [ 3980.287321] ? do_raw_spin_unlock+0x75/0x190 [ 3980.289038] ? _raw_spin_unlock+0x12/0x30 [ 3980.290802] do_sys_openat2+0x2b4/0x410 [ 3980.292237] do_sys_open+0x73/0xa0 [ 3980.293547] __x64_sys_openat+0x24/0x30 [ 3980.295225] do_syscall_64+0xc1/0x440 [ 3980.296537] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3980.298567] RIP: 0033:0x7ff54af45332 [ 3980.301237] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3980.308328] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3980.311943] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 3980.314570] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 3980.317864] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 3980.320830] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 3980.324084] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 3992.060618] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 3992.069812] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3992.118312] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3992.127565] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3992.142946] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3992.147043] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3992.177031] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3992.179987] RAX: 0000000000000486 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 3992.185500] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 3992.192472] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3992.198973] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 3992.203186] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 3992.207967] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 3992.212225] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3992.218377] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 3992.225486] Call Trace: [ 3992.227503] [ 3992.231093] ? show_regs.cold.9+0x22/0x2f [ 3992.238820] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3992.248521] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3992.261259] ? watchdog+0x40/0x40 [ 3992.265544] ? hrtimer_interrupt+0x128/0x2c0 [ 3992.269893] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3992.272493] ? apic_timer_interrupt+0xf/0x20 [ 3992.276416] [ 3992.279848] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3992.286120] do_raw_spin_lock+0x177/0x190 [ 3992.290166] _raw_spin_lock+0x21/0x30 [ 3992.292132] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3992.296437] ? do_raw_spin_unlock+0x75/0x190 [ 3992.304149] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3992.310126] ? do_wait_intr+0xf0/0xf0 [ 3992.315087] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3992.319299] mdc_close+0x53b/0xd70 [mdc] [ 3992.322429] lmv_close+0x18d/0x490 [lmv] [ 3992.325934] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 3992.331079] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3992.335486] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3992.339251] ll_md_close+0x274/0x800 [lustre] [ 3992.345421] ll_file_release+0x128/0xc50 [lustre] [ 3992.350288] __fput+0x10d/0x440 [ 3992.354074] ____fput+0x12/0x20 [ 3992.359267] task_work_run+0xc6/0x110 [ 3992.363081] exit_to_usermode_loop+0x1dd/0x1f0 [ 3992.369162] do_syscall_64+0x3d6/0x440 [ 3992.376086] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3992.379145] RIP: 0033:0x7ff54af45cd8 [ 3992.383181] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3992.396878] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3992.401679] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 3992.411663] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 3992.416813] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 3992.420522] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 3992.425806] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3993.892969] rcu: INFO: rcu_sched self-detected stall on CPU [ 3993.900379] rcu: 0-...!: (1084979 ticks this GP) idle=98a/1/0x4000000000000002 softirq=125154/125154 fqs=959 [ 3993.911233] (t=2040052 jiffies g=253585 q=3378) [ 3993.918243] rcu: rcu_sched kthread starved for 2035012 jiffies! g253585 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=3 [ 3993.926274] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 3993.934361] rcu: RCU grace-period kthread stack dump: [ 3993.937163] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 3993.952453] Call Trace: [ 3993.955135] __schedule+0x351/0xcb0 [ 3993.958421] schedule+0xc0/0x180 [ 3993.963345] schedule_timeout+0xb4/0x190 [ 3993.965036] ? do_raw_spin_unlock+0x75/0x190 [ 3993.970325] ? __next_timer_interrupt+0x160/0x160 [ 3993.972909] ? prepare_to_swait_event+0xf3/0x140 [ 3993.984441] rcu_gp_kthread+0x8dd/0x15b0 [ 3993.990032] ? force_qs_rnp+0x2c0/0x2c0 [ 3993.994288] kthread+0x1d1/0x200 [ 3993.998362] ? set_kthread_struct+0x70/0x70 [ 3994.000368] ret_from_fork+0x1f/0x30 [ 3994.004808] rcu: Stack dump where RCU GP kthread last ran: [ 3994.011957] Sending NMI from CPU 0 to CPUs 3: [ 3994.018919] NMI backtrace for cpu 3 [ 3994.018926] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3994.018928] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3994.018929] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3994.018931] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3994.018933] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 [ 3994.018935] RAX: 0000000000006b70 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 3994.018937] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 3994.018938] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3994.018940] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 3994.018941] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 3994.018942] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 3994.018944] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3994.018945] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 3994.018946] Call Trace: [ 3994.018947] [ 3994.018948] ? show_regs.cold.9+0x22/0x2f [ 3994.018949] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 3994.018950] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 3994.018951] ? nmi_handle+0xbd/0x210 [ 3994.018953] ? default_do_nmi+0x4b/0x1b0 [ 3994.018954] ? do_nmi+0x280/0x380 [ 3994.018955] ? end_repeat_nmi+0x16/0x1f [ 3994.018956] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3994.018957] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3994.018958] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3994.018959] [ 3994.018960] do_raw_spin_lock+0x177/0x190 [ 3994.018961] _raw_spin_lock+0x21/0x30 [ 3994.018962] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3994.018963] ? do_raw_spin_unlock+0x75/0x190 [ 3994.018964] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3994.018965] ? do_wait_intr+0xf0/0xf0 [ 3994.018966] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3994.018967] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3994.018968] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3994.018969] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3994.018970] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3994.018971] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3994.018972] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3994.018973] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3994.018974] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3994.018975] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3994.018976] lmv_intent_open+0x338/0x1150 [lmv] [ 3994.018977] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3994.018978] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3994.018980] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3994.018981] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3994.018982] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3994.018983] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3994.018984] ? do_raw_spin_unlock+0x75/0x190 [ 3994.018985] ? _raw_spin_unlock+0x12/0x30 [ 3994.018986] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3994.018987] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3994.018988] ll_atomic_open+0x36a/0xd70 [lustre] [ 3994.018989] lookup_open+0x356/0xb40 [ 3994.018990] do_last+0x313/0xfc0 [ 3994.018991] ? nd_jump_root+0xe5/0x160 [ 3994.018992] ? path_init+0x437/0x520 [ 3994.018993] path_openat+0xf7/0x500 [ 3994.018994] ? do_raw_spin_unlock+0x75/0x190 [ 3994.018995] do_filp_open+0x99/0x140 [ 3994.018996] ? getname_flags+0x6e/0x330 [ 3994.018997] ? __check_object_size+0xff/0x256 [ 3994.018998] ? do_raw_spin_unlock+0x75/0x190 [ 3994.018999] ? _raw_spin_unlock+0x12/0x30 [ 3994.019000] do_sys_openat2+0x2b4/0x410 [ 3994.019001] do_sys_open+0x73/0xa0 [ 3994.019002] __x64_sys_openat+0x24/0x30 [ 3994.019003] do_syscall_64+0xc1/0x440 [ 3994.019004] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3994.019005] RIP: 0033:0x7ff54af45332 [ 3994.019007] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3994.019010] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3994.019014] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 3994.019015] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 3994.019017] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 3994.019019] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 3994.019020] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 3994.021186] NMI backtrace for cpu 0 [ 3994.416649] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3994.428391] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3994.438341] Call Trace: [ 3994.439784] [ 3994.443228] ? dump_stack+0xbb/0x10e [ 3994.448584] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 3994.452418] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 3994.455972] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 3994.458733] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 3994.461067] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 3994.463296] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 3994.470041] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 3994.473859] ? update_process_times+0x6d/0xc0 [ 3994.478131] ? tick_sched_handle.isra.18+0x27/0x90 [ 3994.482755] ? tick_sched_timer+0x53/0xb0 [ 3994.486694] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3994.489258] ? tick_sched_handle.isra.18+0x90/0x90 [ 3994.492888] ? hrtimer_interrupt+0x128/0x2c0 [ 3994.498359] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3994.501750] ? apic_timer_interrupt+0xf/0x20 [ 3994.504014] [ 3994.505867] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3994.508245] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3994.511145] ? do_raw_spin_lock+0x177/0x190 [ 3994.517653] ? _raw_spin_lock+0x21/0x30 [ 3994.520170] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3994.524471] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 3994.528314] ? mdc_commit_open+0x91/0xb0 [mdc] [ 3994.531031] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3994.533932] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3994.543282] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 3994.547934] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3994.551261] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3994.555973] ? do_wait_intr+0xf0/0xf0 [ 3994.557993] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3994.562334] ? kthread+0x1d1/0x200 [ 3994.564642] ? set_kthread_struct+0x70/0x70 [ 3994.568047] ? ret_from_fork+0x1f/0x30 [ 3996.062726] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 3996.072920] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3996.141599] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3996.150833] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3996.160230] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3996.164870] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3996.166579] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3996.188659] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3996.202904] RAX: 0000000000003538 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 3996.213173] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 3996.223407] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 3996.230410] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 3996.234317] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 3996.243215] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 3996.258199] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3996.271939] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 3996.282056] Call Trace: [ 3996.289560] [ 3996.290809] ? show_regs.cold.9+0x22/0x2f [ 3996.299561] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3996.310811] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3996.322720] ? watchdog+0x40/0x40 [ 3996.323946] ? hrtimer_interrupt+0x128/0x2c0 [ 3996.339592] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3996.341136] ? apic_timer_interrupt+0xf/0x20 [ 3996.344106] [ 3996.344865] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3996.349142] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3996.353249] do_raw_spin_lock+0x177/0x190 [ 3996.359223] _raw_spin_lock+0x21/0x30 [ 3996.362840] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3996.367798] process_one_work+0x2c8/0x700 [ 3996.369242] worker_thread+0x296/0x6e0 [ 3996.376030] ? rescuer_thread+0x570/0x570 [ 3996.380984] kthread+0x1d1/0x200 [ 3996.382805] ? set_kthread_struct+0x70/0x70 [ 3996.385683] ret_from_fork+0x1f/0x30 [ 4008.931800] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2050s! [ 4008.949941] Showing busy workqueues and worker pools: [ 4008.961898] workqueue events: flags=0x0 [ 4008.969325] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4008.981955] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 4008.992925] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4009.004628] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 4009.030026] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4009.042702] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 4009.059528] workqueue events_long: flags=0x0 [ 4009.072104] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4009.087520] pending: lu_objects_flush [obdclass] [ 4009.092836] workqueue events_unbound: flags=0x2 [ 4009.102906] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4009.105294] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4009.139395] workqueue events_power_efficient: flags=0x80 [ 4009.150280] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4009.166589] pending: neigh_periodic_work [ 4009.182216] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4009.195050] pending: do_cache_clean, check_lifetime [ 4009.206688] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4009.214653] pending: neigh_periodic_work [ 4009.218287] workqueue writeback: flags=0x4a [ 4009.233847] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4009.243583] pending: wb_workfn, wb_workfn [ 4009.253450] workqueue xprtiod: flags=0xa [ 4009.259852] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4009.272346] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 4009.281524] workqueue ipv6_addrconf: flags=0x40008 [ 4009.292721] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4009.303749] pending: addrconf_verify_work [ 4009.314319] workqueue knbd0-recv: flags=0x1a [ 4009.320659] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4009.329327] in-flight: 657:recv_work [ 4009.335323] workqueue ptlrpc_pinger: flags=0x2 [ 4009.342420] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4009.355916] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 4009.369739] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2050s workers=4 idle: 20831 12642 28859 [ 4009.377019] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3999s workers=2 idle: 659 [ 4020.061745] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 4020.066420] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4020.068018] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 4020.082983] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4020.082998] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4020.083001] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4020.083017] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4020.083021] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4020.083027] RAX: 0000000000007f86 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 4020.083029] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 4020.083031] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4020.083033] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 4020.083036] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 4020.083039] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 4020.083041] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4020.083043] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 4020.083050] Call Trace: [ 4020.083055] [ 4020.083062] ? show_regs.cold.9+0x22/0x2f [ 4020.083070] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4020.083136] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4020.083142] ? watchdog+0x40/0x40 [ 4020.083146] ? hrtimer_interrupt+0x128/0x2c0 [ 4020.083150] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4020.083157] ? apic_timer_interrupt+0xf/0x20 [ 4020.083162] [ 4020.083165] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4020.083172] do_raw_spin_lock+0x177/0x190 [ 4020.083177] _raw_spin_lock+0x21/0x30 [ 4020.083181] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4020.083505] ? do_raw_spin_unlock+0x75/0x190 [ 4020.083512] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4020.083648] ? do_wait_intr+0xf0/0xf0 [ 4020.083654] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4020.083832] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4020.084047] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4020.084347] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4020.084476] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4020.084520] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4020.084550] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4020.084575] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4020.084660] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4020.084871] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4020.084901] lmv_intent_open+0x338/0x1150 [lmv] [ 4020.084931] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4020.085018] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4020.085382] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4020.085408] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4020.085503] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4020.085585] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4020.085667] ? do_raw_spin_unlock+0x75/0x190 [ 4020.085674] ? _raw_spin_unlock+0x12/0x30 [ 4020.085680] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4020.085707] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4020.085727] ll_atomic_open+0x36a/0xd70 [lustre] [ 4020.085795] lookup_open+0x356/0xb40 [ 4020.085803] do_last+0x313/0xfc0 [ 4020.085806] ? nd_jump_root+0xe5/0x160 [ 4020.085809] ? path_init+0x437/0x520 [ 4020.085812] path_openat+0xf7/0x500 [ 4020.085816] ? do_raw_spin_unlock+0x75/0x190 [ 4020.085819] do_filp_open+0x99/0x140 [ 4020.085824] ? getname_flags+0x6e/0x330 [ 4020.085827] ? __check_object_size+0xff/0x256 [ 4020.085833] ? do_raw_spin_unlock+0x75/0x190 [ 4020.085836] ? _raw_spin_unlock+0x12/0x30 [ 4020.085841] do_sys_openat2+0x2b4/0x410 [ 4020.085845] do_sys_open+0x73/0xa0 [ 4020.085850] __x64_sys_openat+0x24/0x30 [ 4020.085854] do_syscall_64+0xc1/0x440 [ 4020.085860] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4020.085867] RIP: 0033:0x7ff54af45332 [ 4020.085874] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4020.085878] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4020.085883] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 4020.085886] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 4020.085889] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 4020.085891] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 4020.085893] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 4020.383395] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4020.411983] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4020.421915] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4020.429024] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4020.433965] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4020.446086] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4020.452322] RAX: 0000000000006fa1 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 4020.457532] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 4020.462575] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4020.467506] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 4020.472596] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 4020.478384] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 4020.484230] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4020.489053] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 4020.501010] Call Trace: [ 4020.502851] [ 4020.504659] ? show_regs.cold.9+0x22/0x2f [ 4020.511113] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4020.516181] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4020.521130] ? watchdog+0x40/0x40 [ 4020.523813] ? hrtimer_interrupt+0x128/0x2c0 [ 4020.526895] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4020.529871] ? apic_timer_interrupt+0xf/0x20 [ 4020.533242] [ 4020.534503] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4020.538200] do_raw_spin_lock+0x177/0x190 [ 4020.540521] _raw_spin_lock+0x21/0x30 [ 4020.542771] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4020.546112] ? do_raw_spin_unlock+0x75/0x190 [ 4020.548781] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4020.552225] ? do_wait_intr+0xf0/0xf0 [ 4020.554963] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4020.558501] mdc_close+0x53b/0xd70 [mdc] [ 4020.561284] lmv_close+0x18d/0x490 [lmv] [ 4020.564969] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 4020.571156] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4020.577679] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4020.582464] ll_md_close+0x274/0x800 [lustre] [ 4020.587040] ll_file_release+0x128/0xc50 [lustre] [ 4020.592353] __fput+0x10d/0x440 [ 4020.594392] ____fput+0x12/0x20 [ 4020.596736] task_work_run+0xc6/0x110 [ 4020.599485] exit_to_usermode_loop+0x1dd/0x1f0 [ 4020.603215] do_syscall_64+0x3d6/0x440 [ 4020.606010] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4020.610454] RIP: 0033:0x7ff54af45cd8 [ 4020.612867] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4020.625959] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4020.631204] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 4020.636198] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 4020.640553] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 4020.645075] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 4020.648849] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4024.060784] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 4024.075692] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4024.106990] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4024.111951] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4024.120610] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4024.125942] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4024.128932] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4024.148438] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4024.154317] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 4024.167773] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 4024.179932] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 4024.189898] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 4024.199425] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 4024.210328] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 4024.224909] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4024.228648] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 4024.236170] Call Trace: [ 4024.237679] [ 4024.238643] ? show_regs.cold.9+0x22/0x2f [ 4024.246100] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4024.254415] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4024.262394] ? watchdog+0x40/0x40 [ 4024.270760] ? hrtimer_interrupt+0x128/0x2c0 [ 4024.276159] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4024.280176] ? apic_timer_interrupt+0xf/0x20 [ 4024.285799] [ 4024.287556] ? native_safe_halt+0x2e/0x50 [ 4024.291620] kvm_wait+0xcd/0xe0 [ 4024.296158] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4024.305190] do_raw_spin_lock+0x177/0x190 [ 4024.315203] _raw_spin_lock+0x21/0x30 [ 4024.319226] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4024.327474] process_one_work+0x2c8/0x700 [ 4024.333157] worker_thread+0x296/0x6e0 [ 4024.344812] ? rescuer_thread+0x570/0x570 [ 4024.354148] kthread+0x1d1/0x200 [ 4024.358523] ? set_kthread_struct+0x70/0x70 [ 4024.367172] ret_from_fork+0x1f/0x30 [ 4036.058355] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 4036.072668] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4036.120278] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4036.131985] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4036.142842] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4036.148834] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4036.166498] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4036.172431] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 4036.174757] RDX: 0000000000007aa4 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 4036.179835] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 4036.183561] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 4036.186757] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4036.189501] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 4036.193407] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4036.195880] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 4036.199363] Call Trace: [ 4036.200475] [ 4036.201441] ? show_regs.cold.9+0x22/0x2f [ 4036.203240] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4036.205620] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4036.207859] ? watchdog+0x40/0x40 [ 4036.209512] ? hrtimer_interrupt+0x128/0x2c0 [ 4036.211887] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4036.214077] ? apic_timer_interrupt+0xf/0x20 [ 4036.216042] [ 4036.217061] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4036.220019] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4036.222571] do_raw_spin_lock+0x177/0x190 [ 4036.224525] _raw_spin_lock+0x21/0x30 [ 4036.226188] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4036.228778] obd_mod_free+0x46/0x140 [ptlrpc] [ 4036.231098] mdc_commit_open+0x91/0xb0 [mdc] [ 4036.233346] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4036.235505] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4036.237767] after_reply+0xe2c/0x1780 [ptlrpc] [ 4036.239558] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4036.241889] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4036.243723] ? do_wait_intr+0xf0/0xf0 [ 4036.245354] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4036.247376] kthread+0x1d1/0x200 [ 4036.248708] ? set_kthread_struct+0x70/0x70 [ 4036.250216] ret_from_fork+0x1f/0x30 [ 4039.649186] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2080s! [ 4039.661265] Showing busy workqueues and worker pools: [ 4039.667856] workqueue events: flags=0x0 [ 4039.671122] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4039.675516] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 4039.686746] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4039.696457] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 4039.703234] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4039.706470] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 4039.710511] workqueue events_long: flags=0x0 [ 4039.713327] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4039.717501] pending: lu_objects_flush [obdclass] [ 4039.720994] workqueue events_unbound: flags=0x2 [ 4039.723849] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4039.728532] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4039.735640] workqueue events_power_efficient: flags=0x80 [ 4039.740200] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4039.749289] pending: neigh_periodic_work [ 4039.750777] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4039.765750] pending: do_cache_clean, check_lifetime [ 4039.775648] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4039.787615] pending: neigh_periodic_work [ 4039.792760] workqueue writeback: flags=0x4a [ 4039.794982] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4039.806677] pending: wb_workfn, wb_workfn [ 4039.814788] workqueue xprtiod: flags=0xa [ 4039.816324] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4039.823823] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 4039.828389] workqueue ipv6_addrconf: flags=0x40008 [ 4039.831182] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4039.834558] pending: addrconf_verify_work [ 4039.842891] workqueue knbd0-recv: flags=0x1a [ 4039.854252] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4039.862882] in-flight: 657:recv_work [ 4039.865931] workqueue ptlrpc_pinger: flags=0x2 [ 4039.870386] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4039.874123] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 4039.878349] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2080s workers=4 idle: 20831 12642 28859 [ 4039.882967] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4030s workers=2 idle: 659 [ 4048.060075] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [ 4048.061372] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [ 4048.063597] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4048.063675] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4048.063683] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4048.063686] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4048.063699] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4048.063704] RSP: 0000:ffffaf9fc15234f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4048.063709] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 4048.063711] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 4048.063714] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4048.063716] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 4048.063718] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 4048.063721] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 4048.063724] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4048.063726] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 4048.063734] Call Trace: [ 4048.063740] [ 4048.063745] ? show_regs.cold.9+0x22/0x2f [ 4048.063755] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4048.063762] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4048.063767] ? watchdog+0x40/0x40 [ 4048.063771] ? hrtimer_interrupt+0x128/0x2c0 [ 4048.063776] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4048.064049] ? apic_timer_interrupt+0xf/0x20 [ 4048.064054] [ 4048.064057] ? native_safe_halt+0x2e/0x50 [ 4048.064061] kvm_wait+0xcd/0xe0 [ 4048.064069] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4048.064077] do_raw_spin_lock+0x177/0x190 [ 4048.064082] _raw_spin_lock+0x21/0x30 [ 4048.064087] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4048.064394] ? do_raw_spin_unlock+0x75/0x190 [ 4048.064400] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4048.064595] ? do_wait_intr+0xf0/0xf0 [ 4048.064603] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4048.069462] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) [ 4048.091250] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4048.091561] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4048.091860] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4048.092243] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4048.092282] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4048.092313] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4048.092342] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4048.092430] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4048.092667] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4048.092699] lmv_intent_open+0x338/0x1150 [lmv] [ 4048.092731] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4048.092906] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4048.093118] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4048.093142] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4048.093237] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4048.093321] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4048.093405] ? do_raw_spin_unlock+0x75/0x190 [ 4048.093413] ? _raw_spin_unlock+0x12/0x30 [ 4048.093420] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4048.093450] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4048.093471] ll_atomic_open+0x36a/0xd70 [lustre] [ 4048.093556] lookup_open+0x356/0xb40 [ 4048.093566] do_last+0x313/0xfc0 [ 4048.093572] ? nd_jump_root+0xe5/0x160 [ 4048.093576] ? path_init+0x437/0x520 [ 4048.093581] path_openat+0xf7/0x500 [ 4048.093586] ? do_raw_spin_unlock+0x75/0x190 [ 4048.093591] do_filp_open+0x99/0x140 [ 4048.093598] ? getname_flags+0x6e/0x330 [ 4048.093602] ? __check_object_size+0xff/0x256 [ 4048.093611] ? do_raw_spin_unlock+0x75/0x190 [ 4048.093615] ? _raw_spin_unlock+0x12/0x30 [ 4048.093619] do_sys_openat2+0x2b4/0x410 [ 4048.093624] do_sys_open+0x73/0xa0 [ 4048.093629] __x64_sys_openat+0x24/0x30 [ 4048.093633] do_syscall_64+0xc1/0x440 [ 4048.093640] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4048.093646] RIP: 0033:0x7ff54af45332 [ 4048.093654] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4048.093658] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4048.093663] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 4048.093667] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 4048.093669] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 4048.093671] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 4048.093674] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 4048.348375] obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4048.365082] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4048.369674] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4048.373437] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4048.376131] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4048.384900] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4048.388455] RAX: 0000000000003e1f RBX: ffff96848811cae0 RCX: 0000000000000001 [ 4048.391809] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 4048.393910] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4048.396935] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 4048.400170] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 4048.403373] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 4048.406969] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4048.409754] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 4048.412946] Call Trace: [ 4048.414231] [ 4048.415226] ? show_regs.cold.9+0x22/0x2f [ 4048.416961] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4048.419645] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4048.425530] ? watchdog+0x40/0x40 [ 4048.428249] ? hrtimer_interrupt+0x128/0x2c0 [ 4048.432077] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4048.435463] ? apic_timer_interrupt+0xf/0x20 [ 4048.438188] [ 4048.439554] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4048.442731] do_raw_spin_lock+0x177/0x190 [ 4048.445684] _raw_spin_lock+0x21/0x30 [ 4048.451449] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4048.459351] ? do_raw_spin_unlock+0x75/0x190 [ 4048.462653] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4048.469897] ? do_wait_intr+0xf0/0xf0 [ 4048.478716] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4048.481489] mdc_close+0x53b/0xd70 [mdc] [ 4048.486923] lmv_close+0x18d/0x490 [lmv] [ 4048.491841] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 4048.500599] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4048.505377] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4048.512498] ll_md_close+0x274/0x800 [lustre] [ 4048.517550] ll_file_release+0x128/0xc50 [lustre] [ 4048.525504] __fput+0x10d/0x440 [ 4048.527803] ____fput+0x12/0x20 [ 4048.532228] task_work_run+0xc6/0x110 [ 4048.536949] exit_to_usermode_loop+0x1dd/0x1f0 [ 4048.539313] do_syscall_64+0x3d6/0x440 [ 4048.541383] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4048.544645] RIP: 0033:0x7ff54af45cd8 [ 4048.546673] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4048.555484] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4048.558902] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 4048.562750] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 4048.565915] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 4048.569303] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 4048.572481] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4052.059142] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 4052.063970] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4052.112990] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4052.124317] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4052.134219] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4052.139071] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4052.145081] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4052.159519] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4052.165467] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 4052.168992] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 4052.170813] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 4052.179465] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 4052.186597] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 4052.193748] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 4052.198688] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4052.208099] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 4052.221762] Call Trace: [ 4052.223627] [ 4052.226745] ? show_regs.cold.9+0x22/0x2f [ 4052.234725] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4052.244407] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4052.249207] ? watchdog+0x40/0x40 [ 4052.253183] ? hrtimer_interrupt+0x128/0x2c0 [ 4052.257940] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4052.262510] ? apic_timer_interrupt+0xf/0x20 [ 4052.265104] [ 4052.270459] ? native_safe_halt+0x2e/0x50 [ 4052.272656] kvm_wait+0xcd/0xe0 [ 4052.274396] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4052.276302] do_raw_spin_lock+0x177/0x190 [ 4052.278056] _raw_spin_lock+0x21/0x30 [ 4052.279706] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4052.282391] process_one_work+0x2c8/0x700 [ 4052.284395] worker_thread+0x296/0x6e0 [ 4052.286119] ? rescuer_thread+0x570/0x570 [ 4052.287898] kthread+0x1d1/0x200 [ 4052.289385] ? set_kthread_struct+0x70/0x70 [ 4052.291028] ret_from_fork+0x1f/0x30 [ 4064.055143] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 4064.064332] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4064.122586] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4064.130525] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4064.146378] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4064.150809] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4064.161160] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4064.172321] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 4064.191337] RDX: 0000000000003775 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 4064.193699] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 4064.211877] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 4064.225387] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4064.231620] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 4064.235768] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4064.238888] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 4064.246232] Call Trace: [ 4064.248925] [ 4064.249920] ? show_regs.cold.9+0x22/0x2f [ 4064.254379] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4064.260260] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4064.267203] ? watchdog+0x40/0x40 [ 4064.272216] ? hrtimer_interrupt+0x128/0x2c0 [ 4064.280153] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4064.283443] ? apic_timer_interrupt+0xf/0x20 [ 4064.288717] [ 4064.289538] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4064.295394] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4064.301524] do_raw_spin_lock+0x177/0x190 [ 4064.303975] _raw_spin_lock+0x21/0x30 [ 4064.306642] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4064.309116] obd_mod_free+0x46/0x140 [ptlrpc] [ 4064.311713] mdc_commit_open+0x91/0xb0 [mdc] [ 4064.313653] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4064.318101] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4064.325052] after_reply+0xe2c/0x1780 [ptlrpc] [ 4064.329860] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4064.332852] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4064.335441] ? do_wait_intr+0xf0/0xf0 [ 4064.337260] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4064.340062] kthread+0x1d1/0x200 [ 4064.341561] ? set_kthread_struct+0x70/0x70 [ 4064.343648] ret_from_fork+0x1f/0x30 [ 4070.367067] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2111s! [ 4070.381809] Showing busy workqueues and worker pools: [ 4070.388337] workqueue events: flags=0x0 [ 4070.394499] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4070.399373] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 4070.409932] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4070.416883] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 4070.428450] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4070.432874] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 4070.442727] workqueue events_long: flags=0x0 [ 4070.449630] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4070.461919] pending: lu_objects_flush [obdclass] [ 4070.467664] workqueue events_unbound: flags=0x2 [ 4070.474680] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4070.478651] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4070.486633] workqueue events_power_efficient: flags=0x80 [ 4070.488751] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4070.495220] pending: neigh_periodic_work [ 4070.502742] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4070.510490] pending: do_cache_clean, check_lifetime [ 4070.517726] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4070.529030] pending: neigh_periodic_work [ 4070.534174] workqueue writeback: flags=0x4a [ 4070.537496] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4070.540764] pending: wb_workfn, wb_workfn [ 4070.544384] workqueue xprtiod: flags=0xa [ 4070.545987] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4070.551245] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 4070.562695] workqueue ipv6_addrconf: flags=0x40008 [ 4070.568893] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4070.578768] pending: addrconf_verify_work [ 4070.583639] workqueue knbd0-recv: flags=0x1a [ 4070.586135] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4070.588674] in-flight: 657:recv_work [ 4070.590538] workqueue ptlrpc_pinger: flags=0x2 [ 4070.592427] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4070.599024] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 4070.606475] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2111s workers=4 idle: 20831 12642 28859 [ 4070.616011] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4060s workers=2 idle: 659 [ 4076.061094] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [ 4076.062146] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [ 4076.070982] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4076.071070] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4076.071078] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4076.071081] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4076.071096] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4076.071100] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4076.071104] RAX: 0000000000002a0c RBX: ffff96848811cae0 RCX: 0000000000000001 [ 4076.071107] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 4076.071109] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4076.071111] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 4076.071114] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 4076.071117] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 4076.071120] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4076.071122] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 4076.071130] Call Trace: [ 4076.071135] [ 4076.071141] ? show_regs.cold.9+0x22/0x2f [ 4076.071151] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4076.071157] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4076.071161] ? watchdog+0x40/0x40 [ 4076.071165] ? hrtimer_interrupt+0x128/0x2c0 [ 4076.071169] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4076.071177] ? apic_timer_interrupt+0xf/0x20 [ 4076.071182] [ 4076.071185] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4076.071191] do_raw_spin_lock+0x177/0x190 [ 4076.071195] _raw_spin_lock+0x21/0x30 [ 4076.071199] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4076.071708] ? do_raw_spin_unlock+0x75/0x190 [ 4076.071766] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4076.071977] ? do_wait_intr+0xf0/0xf0 [ 4076.071985] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4076.072186] mdc_close+0x53b/0xd70 [mdc] [ 4076.072226] lmv_close+0x18d/0x490 [lmv] [ 4076.072259] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 4076.072435] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4076.072818] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4076.072907] ll_md_close+0x274/0x800 [lustre] [ 4076.072990] ll_file_release+0x128/0xc50 [lustre] [ 4076.073071] __fput+0x10d/0x440 [ 4076.073080] ____fput+0x12/0x20 [ 4076.073085] task_work_run+0xc6/0x110 [ 4076.073091] exit_to_usermode_loop+0x1dd/0x1f0 [ 4076.073098] do_syscall_64+0x3d6/0x440 [ 4076.073103] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4076.073111] RIP: 0033:0x7ff54af45cd8 [ 4076.073118] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4076.073122] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4076.073128] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 4076.073130] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 4076.073132] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 4076.073134] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 4076.073136] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4076.274798] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4076.296864] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4076.301722] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4076.305480] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4076.308467] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4076.316571] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4076.319753] RAX: 000000000000598a RBX: ffff96848811cae0 RCX: 0000000000000001 [ 4076.322584] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 4076.325927] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4076.328458] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 4076.331372] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 4076.334400] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 4076.337869] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4076.340490] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 4076.344034] Call Trace: [ 4076.344995] [ 4076.345917] ? show_regs.cold.9+0x22/0x2f [ 4076.347703] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4076.350062] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4076.351645] ? watchdog+0x40/0x40 [ 4076.352953] ? hrtimer_interrupt+0x128/0x2c0 [ 4076.354988] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4076.357030] ? apic_timer_interrupt+0xf/0x20 [ 4076.358666] [ 4076.359680] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4076.362253] do_raw_spin_lock+0x177/0x190 [ 4076.363836] _raw_spin_lock+0x21/0x30 [ 4076.365183] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4076.368609] ? do_raw_spin_unlock+0x75/0x190 [ 4076.370944] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4076.373603] ? do_wait_intr+0xf0/0xf0 [ 4076.376810] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4076.379940] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4076.382641] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4076.386067] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4076.388874] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4076.393498] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4076.395664] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4076.397797] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4076.400108] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4076.402063] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4076.403934] lmv_intent_open+0x338/0x1150 [lmv] [ 4076.405484] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4076.407483] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4076.409426] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4076.411110] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4076.412967] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4076.415189] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4076.417104] ? do_raw_spin_unlock+0x75/0x190 [ 4076.418501] ? _raw_spin_unlock+0x12/0x30 [ 4076.420030] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4076.421875] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4076.423662] ll_atomic_open+0x36a/0xd70 [lustre] [ 4076.425394] lookup_open+0x356/0xb40 [ 4076.426816] do_last+0x313/0xfc0 [ 4076.427974] ? nd_jump_root+0xe5/0x160 [ 4076.429415] ? path_init+0x437/0x520 [ 4076.430686] path_openat+0xf7/0x500 [ 4076.432116] ? do_raw_spin_unlock+0x75/0x190 [ 4076.435852] do_filp_open+0x99/0x140 [ 4076.437002] ? getname_flags+0x6e/0x330 [ 4076.438237] ? __check_object_size+0xff/0x256 [ 4076.447002] ? do_raw_spin_unlock+0x75/0x190 [ 4076.452540] ? _raw_spin_unlock+0x12/0x30 [ 4076.460687] do_sys_openat2+0x2b4/0x410 [ 4076.462034] do_sys_open+0x73/0xa0 [ 4076.467508] __x64_sys_openat+0x24/0x30 [ 4076.468806] do_syscall_64+0xc1/0x440 [ 4076.469980] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4076.482166] RIP: 0033:0x7ff54af45332 [ 4076.488447] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4076.500424] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4076.506004] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 4076.509077] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 4076.512265] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 4076.515491] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 4076.518929] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 4080.062729] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 4080.076107] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4080.121580] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4080.143868] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4080.149422] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4080.152991] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4080.161069] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4080.183375] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 4080.190075] RAX: 0000000000007212 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 4080.193016] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 4080.197481] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 4080.205420] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 4080.209742] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 4080.215876] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 4080.221378] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4080.223774] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 4080.228326] Call Trace: [ 4080.230558] [ 4080.232670] ? show_regs.cold.9+0x22/0x2f [ 4080.235239] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4080.237057] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4080.242891] ? watchdog+0x40/0x40 [ 4080.246469] ? hrtimer_interrupt+0x128/0x2c0 [ 4080.249274] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4080.251638] ? apic_timer_interrupt+0xf/0x20 [ 4080.255381] [ 4080.257432] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4080.262016] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4080.265847] do_raw_spin_lock+0x177/0x190 [ 4080.267585] _raw_spin_lock+0x21/0x30 [ 4080.270355] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4080.272881] process_one_work+0x2c8/0x700 [ 4080.276424] worker_thread+0x296/0x6e0 [ 4080.277989] ? rescuer_thread+0x570/0x570 [ 4080.281580] kthread+0x1d1/0x200 [ 4080.282644] ? set_kthread_struct+0x70/0x70 [ 4080.283977] ret_from_fork+0x1f/0x30 [ 4096.055494] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 4096.063614] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4096.090056] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4096.096055] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4096.100962] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4096.103795] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4096.113376] RSP: 0000:ffffaf9fc1b97bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4096.117171] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 4096.120507] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 4096.124282] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 4096.127750] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 4096.131676] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4096.135812] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 4096.144379] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4096.147805] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 4096.152064] Call Trace: [ 4096.153953] [ 4096.154876] ? show_regs.cold.9+0x22/0x2f [ 4096.157211] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4096.160170] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4096.162645] ? watchdog+0x40/0x40 [ 4096.164622] ? hrtimer_interrupt+0x128/0x2c0 [ 4096.166827] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4096.169595] ? apic_timer_interrupt+0xf/0x20 [ 4096.171976] [ 4096.173362] ? native_safe_halt+0x2e/0x50 [ 4096.175976] kvm_wait+0xcd/0xe0 [ 4096.177827] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4096.180664] do_raw_spin_lock+0x177/0x190 [ 4096.182969] _raw_spin_lock+0x21/0x30 [ 4096.184991] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4096.189142] obd_mod_free+0x46/0x140 [ptlrpc] [ 4096.196717] mdc_commit_open+0x91/0xb0 [mdc] [ 4096.203347] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4096.212400] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4096.228369] after_reply+0xe2c/0x1780 [ptlrpc] [ 4096.234794] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4096.240295] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4096.243456] ? do_wait_intr+0xf0/0xf0 [ 4096.246021] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4096.248124] kthread+0x1d1/0x200 [ 4096.249263] ? set_kthread_struct+0x70/0x70 [ 4096.251437] ret_from_fork+0x1f/0x30 [ 4101.094423] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2142s! [ 4101.112018] Showing busy workqueues and worker pools: [ 4101.113495] workqueue events: flags=0x0 [ 4101.121960] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4101.129105] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 4101.134114] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4101.143818] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 4101.154231] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4101.157770] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 4101.161350] workqueue events_long: flags=0x0 [ 4101.165958] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4101.168733] pending: lu_objects_flush [obdclass] [ 4101.176141] workqueue events_unbound: flags=0x2 [ 4101.179827] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4101.185866] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4101.195737] workqueue events_power_efficient: flags=0x80 [ 4101.197022] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4101.204878] pending: neigh_periodic_work [ 4101.208321] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4101.211874] pending: do_cache_clean, check_lifetime [ 4101.214565] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4101.217428] pending: neigh_periodic_work [ 4101.219839] workqueue writeback: flags=0x4a [ 4101.221890] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4101.227291] pending: wb_workfn, wb_workfn [ 4101.229495] workqueue xprtiod: flags=0xa [ 4101.231113] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4101.233840] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 4101.236901] workqueue ipv6_addrconf: flags=0x40008 [ 4101.241339] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4101.243557] pending: addrconf_verify_work [ 4101.245030] workqueue knbd0-recv: flags=0x1a [ 4101.246473] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4101.248677] in-flight: 657:recv_work [ 4101.250429] workqueue ptlrpc_pinger: flags=0x2 [ 4101.252361] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4101.254777] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 4101.257291] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2142s workers=4 idle: 20831 12642 28859 [ 4101.260873] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4091s workers=2 idle: 659 [ 4104.063758] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [ 4104.066602] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [ 4104.069731] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4104.069813] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4104.069821] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4104.069823] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4104.069836] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4104.069841] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4104.069845] RAX: 0000000000001733 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 4104.069848] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 4104.069850] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4104.069853] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 4104.069855] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 4104.069858] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 4104.069862] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4104.069864] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 4104.069872] Call Trace: [ 4104.069878] [ 4104.069884] ? show_regs.cold.9+0x22/0x2f [ 4104.069893] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4104.069899] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4104.069903] ? watchdog+0x40/0x40 [ 4104.069907] ? hrtimer_interrupt+0x128/0x2c0 [ 4104.069911] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4104.069919] ? apic_timer_interrupt+0xf/0x20 [ 4104.069923] [ 4104.069926] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4104.069933] do_raw_spin_lock+0x177/0x190 [ 4104.069937] _raw_spin_lock+0x21/0x30 [ 4104.069942] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4104.070229] ? do_raw_spin_unlock+0x75/0x190 [ 4104.070235] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4104.070435] ? do_wait_intr+0xf0/0xf0 [ 4104.070444] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4104.070782] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4104.070995] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4104.071210] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4104.071329] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4104.071367] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4104.071398] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4104.071426] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4104.071651] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4104.071869] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4104.071898] lmv_intent_open+0x338/0x1150 [lmv] [ 4104.071934] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4104.072023] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4104.081345] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4104.081425] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4104.081432] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4104.081434] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4104.081447] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4104.081450] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4104.081453] RAX: 0000000000003d03 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 4104.081455] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 4104.081457] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4104.081458] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 4104.081460] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 4104.081462] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 4104.081464] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4104.081465] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 4104.081472] Call Trace: [ 4104.081477] [ 4104.081852] ? show_regs.cold.9+0x22/0x2f [ 4104.121139] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4104.121190] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4104.121322] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4104.121406] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4104.121491] ? do_raw_spin_unlock+0x75/0x190 [ 4104.121499] ? _raw_spin_unlock+0x12/0x30 [ 4104.121507] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4104.121535] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4104.121551] ll_atomic_open+0x36a/0xd70 [lustre] [ 4104.121649] lookup_open+0x356/0xb40 [ 4104.121658] do_last+0x313/0xfc0 [ 4104.121663] ? nd_jump_root+0xe5/0x160 [ 4104.121668] ? path_init+0x437/0x520 [ 4104.121673] path_openat+0xf7/0x500 [ 4104.121678] ? do_raw_spin_unlock+0x75/0x190 [ 4104.121682] do_filp_open+0x99/0x140 [ 4104.121688] ? getname_flags+0x6e/0x330 [ 4104.121692] ? __check_object_size+0xff/0x256 [ 4104.121700] ? do_raw_spin_unlock+0x75/0x190 [ 4104.121704] ? _raw_spin_unlock+0x12/0x30 [ 4104.121708] do_sys_openat2+0x2b4/0x410 [ 4104.121713] do_sys_open+0x73/0xa0 [ 4104.121718] __x64_sys_openat+0x24/0x30 [ 4104.121722] do_syscall_64+0xc1/0x440 [ 4104.121728] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4104.121736] RIP: 0033:0x7ff54af45332 [ 4104.121744] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4104.121748] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4104.121753] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 4104.121755] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 4104.121757] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 4104.121759] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 4104.121761] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 4104.653521] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4104.655933] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4104.658042] ? watchdog+0x40/0x40 [ 4104.660327] ? hrtimer_interrupt+0x128/0x2c0 [ 4104.662948] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4104.664714] ? apic_timer_interrupt+0xf/0x20 [ 4104.666854] [ 4104.667852] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4104.671168] do_raw_spin_lock+0x177/0x190 [ 4104.673302] _raw_spin_lock+0x21/0x30 [ 4104.675318] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4104.678171] ? do_raw_spin_unlock+0x75/0x190 [ 4104.680297] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4104.685088] ? do_wait_intr+0xf0/0xf0 [ 4104.687430] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4104.691131] mdc_close+0x53b/0xd70 [mdc] [ 4104.693178] lmv_close+0x18d/0x490 [lmv] [ 4104.696996] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 4104.700733] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4104.707096] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4104.712175] ll_md_close+0x274/0x800 [lustre] [ 4104.718031] ll_file_release+0x128/0xc50 [lustre] [ 4104.720201] __fput+0x10d/0x440 [ 4104.721177] ____fput+0x12/0x20 [ 4104.721964] task_work_run+0xc6/0x110 [ 4104.722768] exit_to_usermode_loop+0x1dd/0x1f0 [ 4104.728185] do_syscall_64+0x3d6/0x440 [ 4104.729495] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4104.730579] RIP: 0033:0x7ff54af45cd8 [ 4104.731495] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4104.738998] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4104.744668] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 4104.747731] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 4104.750533] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 4104.753080] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 4104.758956] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4108.060899] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 4108.070388] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4108.122112] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4108.130303] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4108.136090] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4108.139217] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4108.141804] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4108.152228] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4108.158655] RAX: 000000000000760a RBX: ffff96848811cae0 RCX: 0000000000000001 [ 4108.164825] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 4108.169296] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 4108.174368] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 4108.181380] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 4108.186303] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 4108.194998] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4108.198163] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 4108.202227] Call Trace: [ 4108.203274] [ 4108.204257] ? show_regs.cold.9+0x22/0x2f [ 4108.205683] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4108.209254] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4108.212723] ? watchdog+0x40/0x40 [ 4108.215375] ? hrtimer_interrupt+0x128/0x2c0 [ 4108.218078] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4108.220042] ? apic_timer_interrupt+0xf/0x20 [ 4108.222336] [ 4108.223393] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4108.229730] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4108.236542] do_raw_spin_lock+0x177/0x190 [ 4108.239351] _raw_spin_lock+0x21/0x30 [ 4108.246539] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4108.252322] process_one_work+0x2c8/0x700 [ 4108.256083] worker_thread+0x296/0x6e0 [ 4108.259297] ? rescuer_thread+0x570/0x570 [ 4108.262887] kthread+0x1d1/0x200 [ 4108.265908] ? set_kthread_struct+0x70/0x70 [ 4108.270612] ret_from_fork+0x1f/0x30 [ 4128.055420] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 4128.063945] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4128.094789] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4128.105071] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4128.111385] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4128.116417] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4128.128161] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4128.132548] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 4128.136327] RDX: 0000000000007bee RSI: 0000000000000003 RDI: ffff96848811cae0 [ 4128.140367] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 4128.143408] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 4128.148485] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4128.153078] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 4128.157556] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4128.160896] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 4128.165129] Call Trace: [ 4128.166557] [ 4128.167672] ? show_regs.cold.9+0x22/0x2f [ 4128.170152] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4128.172760] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4128.175061] ? watchdog+0x40/0x40 [ 4128.176716] ? hrtimer_interrupt+0x128/0x2c0 [ 4128.179455] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4128.181792] ? apic_timer_interrupt+0xf/0x20 [ 4128.183942] [ 4128.185102] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4128.188744] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4128.193504] do_raw_spin_lock+0x177/0x190 [ 4128.197372] _raw_spin_lock+0x21/0x30 [ 4128.199632] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4128.204598] obd_mod_free+0x46/0x140 [ptlrpc] [ 4128.208627] mdc_commit_open+0x91/0xb0 [mdc] [ 4128.210770] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4128.213902] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4128.218034] after_reply+0xe2c/0x1780 [ptlrpc] [ 4128.222671] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4128.227265] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4128.229434] ? do_wait_intr+0xf0/0xf0 [ 4128.231284] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4128.235630] kthread+0x1d1/0x200 [ 4128.237622] ? set_kthread_struct+0x70/0x70 [ 4128.239761] ret_from_fork+0x1f/0x30 [ 4131.297316] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2172s! [ 4131.306022] Showing busy workqueues and worker pools: [ 4131.309924] workqueue events: flags=0x0 [ 4131.315749] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4131.327027] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 4131.341945] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4131.352598] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 4131.367511] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4131.380263] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 4131.385051] workqueue events_long: flags=0x0 [ 4131.389279] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4131.394761] pending: lu_objects_flush [obdclass] [ 4131.400261] workqueue events_unbound: flags=0x2 [ 4131.404296] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4131.412734] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4131.438166] workqueue events_power_efficient: flags=0x80 [ 4131.445314] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4131.454320] pending: neigh_periodic_work [ 4131.458273] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4131.468976] pending: do_cache_clean, check_lifetime [ 4131.485044] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4131.497589] pending: neigh_periodic_work [ 4131.504039] workqueue writeback: flags=0x4a [ 4131.511155] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4131.518531] pending: wb_workfn, wb_workfn [ 4131.522117] workqueue xprtiod: flags=0xa [ 4131.524994] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4131.529168] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 4131.535072] workqueue ipv6_addrconf: flags=0x40008 [ 4131.538681] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4131.544023] pending: addrconf_verify_work [ 4131.547695] workqueue knbd0-recv: flags=0x1a [ 4131.550739] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4131.555307] in-flight: 657:recv_work [ 4131.558076] workqueue ptlrpc_pinger: flags=0x2 [ 4131.561392] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4131.569746] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 4131.573860] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2172s workers=4 idle: 20831 12642 28859 [ 4131.585506] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4121s workers=2 idle: 659 [ 4132.060130] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [ 4132.060992] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [ 4132.072808] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4132.072894] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4132.072903] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4132.072905] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4132.072920] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4132.072924] RSP: 0000:ffffaf9fc15234f0 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 4132.072928] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 4132.072931] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 4132.072933] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4132.072934] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 4132.072937] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 4132.072941] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 4132.072944] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4132.072946] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 4132.072954] Call Trace: [ 4132.072961] [ 4132.072968] ? show_regs.cold.9+0x22/0x2f [ 4132.072978] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4132.072986] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4132.072990] ? watchdog+0x40/0x40 [ 4132.072995] ? hrtimer_interrupt+0x128/0x2c0 [ 4132.072999] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4132.073399] ? apic_timer_interrupt+0xf/0x20 [ 4132.073405] [ 4132.073408] ? native_safe_halt+0x2e/0x50 [ 4132.073412] kvm_wait+0xcd/0xe0 [ 4132.073419] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4132.073427] do_raw_spin_lock+0x177/0x190 [ 4132.073432] _raw_spin_lock+0x21/0x30 [ 4132.073436] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4132.073718] ? do_raw_spin_unlock+0x75/0x190 [ 4132.075786] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4132.075838] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4132.075843] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4132.075845] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4132.075854] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4132.075857] RSP: 0018:ffffaf9fc1543ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4132.075860] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 4132.075862] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 4132.075863] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4132.075865] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 4132.075867] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 4132.075869] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 4132.075871] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4132.075872] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 4132.075878] Call Trace: [ 4132.075880] [ 4132.075884] ? show_regs.cold.9+0x22/0x2f [ 4132.075890] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4132.075895] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4132.075897] ? watchdog+0x40/0x40 [ 4132.075900] ? hrtimer_interrupt+0x128/0x2c0 [ 4132.075903] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4132.075907] ? apic_timer_interrupt+0xf/0x20 [ 4132.075910] [ 4132.075913] ? native_safe_halt+0x2e/0x50 [ 4132.075915] kvm_wait+0xcd/0xe0 [ 4132.075920] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4132.075926] do_raw_spin_lock+0x177/0x190 [ 4132.075928] _raw_spin_lock+0x21/0x30 [ 4132.075931] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4132.076124] ? do_raw_spin_unlock+0x75/0x190 [ 4132.076127] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4132.076326] ? do_wait_intr+0xf0/0xf0 [ 4132.076332] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4132.076488] mdc_close+0x53b/0xd70 [mdc] [ 4132.076527] lmv_close+0x18d/0x490 [lmv] [ 4132.076736] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 4132.076864] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4132.076951] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4132.077002] ll_md_close+0x274/0x800 [lustre] [ 4132.077055] ll_file_release+0x128/0xc50 [lustre] [ 4132.077107] __fput+0x10d/0x440 [ 4132.077114] ____fput+0x12/0x20 [ 4132.077117] task_work_run+0xc6/0x110 [ 4132.077123] exit_to_usermode_loop+0x1dd/0x1f0 [ 4132.077128] do_syscall_64+0x3d6/0x440 [ 4132.077131] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4132.077136] RIP: 0033:0x7ff54af45cd8 [ 4132.077250] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4132.077255] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4132.077258] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 4132.077260] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 4132.077261] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 4132.077263] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 4132.077264] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4132.566843] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4132.572824] ? do_wait_intr+0xf0/0xf0 [ 4132.574558] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4132.577629] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4132.582038] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4132.584534] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4132.592882] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4132.602087] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4132.604042] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4132.609278] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4132.611821] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4132.614526] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4132.621931] lmv_intent_open+0x338/0x1150 [lmv] [ 4132.629949] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4132.639726] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4132.657594] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4132.662417] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4132.666548] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4132.680598] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4132.683836] ? do_raw_spin_unlock+0x75/0x190 [ 4132.685779] ? _raw_spin_unlock+0x12/0x30 [ 4132.688814] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4132.692645] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4132.694801] ll_atomic_open+0x36a/0xd70 [lustre] [ 4132.700639] lookup_open+0x356/0xb40 [ 4132.705661] do_last+0x313/0xfc0 [ 4132.712515] ? nd_jump_root+0xe5/0x160 [ 4132.721460] ? path_init+0x437/0x520 [ 4132.725655] path_openat+0xf7/0x500 [ 4132.732226] ? do_raw_spin_unlock+0x75/0x190 [ 4132.742883] do_filp_open+0x99/0x140 [ 4132.750850] ? getname_flags+0x6e/0x330 [ 4132.753743] ? __check_object_size+0xff/0x256 [ 4132.760513] ? do_raw_spin_unlock+0x75/0x190 [ 4132.768539] ? _raw_spin_unlock+0x12/0x30 [ 4132.782289] do_sys_openat2+0x2b4/0x410 [ 4132.789562] do_sys_open+0x73/0xa0 [ 4132.795806] __x64_sys_openat+0x24/0x30 [ 4132.798303] do_syscall_64+0xc1/0x440 [ 4132.800787] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4132.805216] RIP: 0033:0x7ff54af45332 [ 4132.810253] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4132.838684] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4132.852918] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 4132.869105] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 4132.880974] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 4132.889021] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 4132.893676] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 4136.059023] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 4136.061540] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4136.106134] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4136.115392] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4136.120939] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4136.128575] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4136.132308] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4136.143244] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4136.147790] RAX: 000000000000690c RBX: ffff96848811cae0 RCX: 0000000000000001 [ 4136.150894] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 4136.153795] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 4136.164814] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 4136.170195] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 4136.174098] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 4136.177672] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4136.180979] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 4136.184921] Call Trace: [ 4136.186063] [ 4136.186980] ? show_regs.cold.9+0x22/0x2f [ 4136.191134] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4136.195427] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4136.197194] ? watchdog+0x40/0x40 [ 4136.198417] ? hrtimer_interrupt+0x128/0x2c0 [ 4136.203413] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4136.209808] ? apic_timer_interrupt+0xf/0x20 [ 4136.212296] [ 4136.215321] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4136.218352] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4136.220638] do_raw_spin_lock+0x177/0x190 [ 4136.222538] _raw_spin_lock+0x21/0x30 [ 4136.224444] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4136.229030] process_one_work+0x2c8/0x700 [ 4136.231750] worker_thread+0x296/0x6e0 [ 4136.234714] ? rescuer_thread+0x570/0x570 [ 4136.241320] kthread+0x1d1/0x200 [ 4136.246591] ? set_kthread_struct+0x70/0x70 [ 4136.249232] ret_from_fork+0x1f/0x30 [ 4156.055494] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 4156.062653] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4156.127921] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4156.141133] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4156.156326] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4156.163505] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4156.182525] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4156.191302] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 4156.199957] RDX: 0000000000001eeb RSI: 0000000000000003 RDI: ffff96848811cae0 [ 4156.204617] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 4156.215279] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 4156.225211] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4156.227063] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 4156.239681] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4156.249268] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 4156.258635] Call Trace: [ 4156.262478] [ 4156.269172] ? show_regs.cold.9+0x22/0x2f [ 4156.272815] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4156.284092] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4156.288513] ? watchdog+0x40/0x40 [ 4156.294082] ? hrtimer_interrupt+0x128/0x2c0 [ 4156.300862] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4156.308920] ? apic_timer_interrupt+0xf/0x20 [ 4156.320614] [ 4156.324794] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4156.337411] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4156.345805] do_raw_spin_lock+0x177/0x190 [ 4156.347203] _raw_spin_lock+0x21/0x30 [ 4156.361370] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4156.370811] obd_mod_free+0x46/0x140 [ptlrpc] [ 4156.376140] mdc_commit_open+0x91/0xb0 [mdc] [ 4156.380538] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4156.382616] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4156.386615] after_reply+0xe2c/0x1780 [ptlrpc] [ 4156.390741] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4156.393596] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4156.395763] ? do_wait_intr+0xf0/0xf0 [ 4156.397862] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4156.403377] kthread+0x1d1/0x200 [ 4156.404776] ? set_kthread_struct+0x70/0x70 [ 4156.406714] ret_from_fork+0x1f/0x30 [ 4160.058166] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 4160.060069] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 4160.061246] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr [ 4160.064237] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4160.064337] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4160.064347] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4160.064351] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4160.064371] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4160.064375] RSP: 0000:ffffaf9fc15234f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4160.064381] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 4160.064383] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 4160.064385] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4160.064387] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 4160.064390] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 4160.064393] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 4160.064397] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4160.064400] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 4160.064410] Call Trace: [ 4160.064420] [ 4160.064431] ? show_regs.cold.9+0x22/0x2f [ 4160.082486] intel_rapl_common [ 4160.124234] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4160.128116] sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4160.128161] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4160.128170] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4160.128173] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4160.128390] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4160.128395] RSP: 0018:ffffaf9fc1543ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4160.128400] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 4160.128402] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 4160.128404] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4160.128406] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 4160.128408] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 4160.128411] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 4160.128415] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4160.128417] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 4160.128425] Call Trace: [ 4160.128430] [ 4160.128436] ? show_regs.cold.9+0x22/0x2f [ 4160.128447] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4160.128454] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4160.128459] ? watchdog+0x40/0x40 [ 4160.128463] ? hrtimer_interrupt+0x128/0x2c0 [ 4160.128467] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4160.128474] ? apic_timer_interrupt+0xf/0x20 [ 4160.128478] [ 4160.128481] ? native_safe_halt+0x2e/0x50 [ 4160.128484] kvm_wait+0xcd/0xe0 [ 4160.128493] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4160.128502] do_raw_spin_lock+0x177/0x190 [ 4160.128507] _raw_spin_lock+0x21/0x30 [ 4160.128511] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4160.135639] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4160.136819] ? do_raw_spin_unlock+0x75/0x190 [ 4160.136835] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4160.147524] ? watchdog+0x40/0x40 [ 4160.150255] ? do_wait_intr+0xf0/0xf0 [ 4160.150278] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4160.150601] mdc_close+0x53b/0xd70 [mdc] [ 4160.150643] lmv_close+0x18d/0x490 [lmv] [ 4160.150676] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 4160.150873] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4160.151087] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4160.151167] ll_md_close+0x274/0x800 [lustre] [ 4160.151249] ll_file_release+0x128/0xc50 [lustre] [ 4160.151337] __fput+0x10d/0x440 [ 4160.151348] ____fput+0x12/0x20 [ 4160.151354] task_work_run+0xc6/0x110 [ 4160.151363] exit_to_usermode_loop+0x1dd/0x1f0 [ 4160.151372] do_syscall_64+0x3d6/0x440 [ 4160.151377] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4160.151388] RIP: 0033:0x7ff54af45cd8 [ 4160.151396] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4160.151401] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4160.151407] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 4160.151409] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 4160.151411] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 4160.151413] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 4160.151416] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4160.507503] ? hrtimer_interrupt+0x128/0x2c0 [ 4160.508804] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4160.511250] ? apic_timer_interrupt+0xf/0x20 [ 4160.515871] [ 4160.518676] ? native_safe_halt+0x2e/0x50 [ 4160.522549] kvm_wait+0xcd/0xe0 [ 4160.524456] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4160.526724] do_raw_spin_lock+0x177/0x190 [ 4160.528674] _raw_spin_lock+0x21/0x30 [ 4160.530580] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4160.533782] ? do_raw_spin_unlock+0x75/0x190 [ 4160.535534] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4160.538466] ? do_wait_intr+0xf0/0xf0 [ 4160.540574] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4160.543261] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4160.545498] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4160.547731] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4160.549787] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4160.551975] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4160.553288] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4160.558597] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4160.563602] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4160.569846] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4160.573733] lmv_intent_open+0x338/0x1150 [lmv] [ 4160.577869] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4160.583556] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4160.588679] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4160.592757] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4160.598362] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4160.603729] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4160.605805] ? do_raw_spin_unlock+0x75/0x190 [ 4160.607583] ? _raw_spin_unlock+0x12/0x30 [ 4160.608870] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4160.611501] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4160.613677] ll_atomic_open+0x36a/0xd70 [lustre] [ 4160.615420] lookup_open+0x356/0xb40 [ 4160.616787] do_last+0x313/0xfc0 [ 4160.617832] ? nd_jump_root+0xe5/0x160 [ 4160.619541] ? path_init+0x437/0x520 [ 4160.620859] path_openat+0xf7/0x500 [ 4160.623367] ? do_raw_spin_unlock+0x75/0x190 [ 4160.624795] do_filp_open+0x99/0x140 [ 4160.627613] ? getname_flags+0x6e/0x330 [ 4160.630885] ? __check_object_size+0xff/0x256 [ 4160.636440] ? do_raw_spin_unlock+0x75/0x190 [ 4160.638825] ? _raw_spin_unlock+0x12/0x30 [ 4160.643576] do_sys_openat2+0x2b4/0x410 [ 4160.645820] do_sys_open+0x73/0xa0 [ 4160.649776] __x64_sys_openat+0x24/0x30 [ 4160.653824] do_syscall_64+0xc1/0x440 [ 4160.656060] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4160.661749] RIP: 0033:0x7ff54af45332 [ 4160.666477] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4160.684743] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4160.689162] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 4160.697680] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 4160.702666] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 4160.707275] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 4160.711521] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 4162.015184] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2203s! [ 4162.018729] Showing busy workqueues and worker pools: [ 4162.020833] workqueue events: flags=0x0 [ 4162.022723] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4162.026236] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 4162.030787] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4162.034584] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 4162.039697] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4162.042688] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 4162.046573] workqueue events_long: flags=0x0 [ 4162.049026] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4162.052666] pending: lu_objects_flush [obdclass] [ 4162.055638] workqueue events_unbound: flags=0x2 [ 4162.057774] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4162.060137] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4162.065186] workqueue events_power_efficient: flags=0x80 [ 4162.067781] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4162.070745] pending: neigh_periodic_work [ 4162.072673] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4162.075557] pending: do_cache_clean, check_lifetime [ 4162.078010] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4162.081110] pending: neigh_periodic_work [ 4162.083388] workqueue writeback: flags=0x4a [ 4162.085197] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4162.088090] pending: wb_workfn, wb_workfn [ 4162.090330] workqueue xprtiod: flags=0xa [ 4162.092188] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4162.095039] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 4162.099421] workqueue ipv6_addrconf: flags=0x40008 [ 4162.101844] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4162.105071] pending: addrconf_verify_work [ 4162.107134] workqueue knbd0-recv: flags=0x1a [ 4162.109171] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4162.112805] in-flight: 657:recv_work [ 4162.114774] workqueue ptlrpc_pinger: flags=0x2 [ 4162.117043] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4162.119999] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 4162.123085] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2203s workers=4 idle: 20831 12642 28859 [ 4162.126672] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4152s workers=2 idle: 659 [ 4164.059050] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 4164.061677] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4164.092348] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4164.099975] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4164.104452] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4164.112045] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4164.117266] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4164.131127] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4164.134297] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 4164.138310] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 4164.150614] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 4164.154838] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 4164.158274] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 4164.161996] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 4164.165430] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4164.168763] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 4164.172419] Call Trace: [ 4164.173493] [ 4164.174753] ? show_regs.cold.9+0x22/0x2f [ 4164.178919] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4164.186360] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4164.189387] ? watchdog+0x40/0x40 [ 4164.191058] ? hrtimer_interrupt+0x128/0x2c0 [ 4164.196469] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4164.202159] ? apic_timer_interrupt+0xf/0x20 [ 4164.204556] [ 4164.209086] ? native_safe_halt+0x2e/0x50 [ 4164.211946] kvm_wait+0xcd/0xe0 [ 4164.218992] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4164.223425] do_raw_spin_lock+0x177/0x190 [ 4164.226494] _raw_spin_lock+0x21/0x30 [ 4164.228629] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4164.232736] process_one_work+0x2c8/0x700 [ 4164.236045] worker_thread+0x296/0x6e0 [ 4164.237522] ? rescuer_thread+0x570/0x570 [ 4164.239184] kthread+0x1d1/0x200 [ 4164.240710] ? set_kthread_struct+0x70/0x70 [ 4164.243380] ret_from_fork+0x1f/0x30 [ 4173.896166] rcu: INFO: rcu_sched self-detected stall on CPU [ 4173.901192] rcu: 0-...!: (1179787 ticks this GP) idle=98a/1/0x4000000000000002 softirq=125154/125154 fqs=959 [ 4173.912666] (t=2220056 jiffies g=253585 q=3439) [ 4173.914120] rcu: rcu_sched kthread starved for 2215016 jiffies! g253585 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [ 4173.929625] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 4173.937505] rcu: RCU grace-period kthread stack dump: [ 4173.945843] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 4173.951415] Call Trace: [ 4173.952558] __schedule+0x351/0xcb0 [ 4173.954630] schedule+0xc0/0x180 [ 4173.956065] schedule_timeout+0xb4/0x190 [ 4173.957234] ? do_raw_spin_unlock+0x75/0x190 [ 4173.958320] ? __next_timer_interrupt+0x160/0x160 [ 4173.959697] ? prepare_to_swait_event+0xf3/0x140 [ 4173.961704] rcu_gp_kthread+0x8dd/0x15b0 [ 4173.963362] ? force_qs_rnp+0x2c0/0x2c0 [ 4173.965080] kthread+0x1d1/0x200 [ 4173.966079] ? set_kthread_struct+0x70/0x70 [ 4173.967313] ret_from_fork+0x1f/0x30 [ 4173.968615] rcu: Stack dump where RCU GP kthread last ran: [ 4173.970847] Sending NMI from CPU 0 to CPUs 1: [ 4173.974709] NMI backtrace for cpu 1 [ 4173.974718] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4173.974721] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4173.974723] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4173.974725] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4173.974727] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 [ 4173.974730] RAX: 000000000000280b RBX: ffff96848811cae0 RCX: 0000000000000001 [ 4173.974731] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 4173.974733] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4173.974734] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 4173.974736] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 4173.974737] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 4173.974738] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4173.974739] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 4173.974740] Call Trace: [ 4173.974741] [ 4173.974742] ? show_regs.cold.9+0x22/0x2f [ 4173.974742] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 4173.974743] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 4173.974932] ? nmi_handle+0xbd/0x210 [ 4173.974934] ? default_do_nmi+0x4b/0x1b0 [ 4173.974935] ? do_nmi+0x280/0x380 [ 4173.974936] ? end_repeat_nmi+0x16/0x1f [ 4173.974937] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4173.974938] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4173.974939] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4173.974940] [ 4173.974940] do_raw_spin_lock+0x177/0x190 [ 4173.974941] _raw_spin_lock+0x21/0x30 [ 4173.974949] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4173.974950] ? do_raw_spin_unlock+0x75/0x190 [ 4173.974951] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4173.974951] ? do_wait_intr+0xf0/0xf0 [ 4173.974952] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4173.974953] mdc_close+0x53b/0xd70 [mdc] [ 4173.974954] lmv_close+0x18d/0x490 [lmv] [ 4173.974955] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 4173.974956] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4173.974957] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4173.974958] ll_md_close+0x274/0x800 [lustre] [ 4173.974959] ll_file_release+0x128/0xc50 [lustre] [ 4173.974960] __fput+0x10d/0x440 [ 4173.974961] ____fput+0x12/0x20 [ 4173.974962] task_work_run+0xc6/0x110 [ 4173.974963] exit_to_usermode_loop+0x1dd/0x1f0 [ 4173.974964] do_syscall_64+0x3d6/0x440 [ 4173.974965] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4173.974966] RIP: 0033:0x7ff54af45cd8 [ 4173.974967] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4173.974969] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4173.974972] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 4173.974973] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 4173.974975] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 4173.974976] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 4173.974977] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4173.975174] NMI backtrace for cpu 0 [ 4174.240730] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4174.251102] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4174.266142] Call Trace: [ 4174.267412] [ 4174.268846] ? dump_stack+0xbb/0x10e [ 4174.273170] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 4174.277720] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 4174.283324] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 4174.289360] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 4174.294314] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 4174.299953] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 4174.307175] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 4174.311982] ? update_process_times+0x6d/0xc0 [ 4174.315654] ? tick_sched_handle.isra.18+0x27/0x90 [ 4174.321209] ? tick_sched_timer+0x53/0xb0 [ 4174.323544] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4174.327428] ? tick_sched_handle.isra.18+0x90/0x90 [ 4174.337481] ? hrtimer_interrupt+0x128/0x2c0 [ 4174.342210] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4174.344681] ? apic_timer_interrupt+0xf/0x20 [ 4174.349743] [ 4174.351961] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4174.361395] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4174.363979] ? do_raw_spin_lock+0x177/0x190 [ 4174.369638] ? _raw_spin_lock+0x21/0x30 [ 4174.375037] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4174.389066] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 4174.404509] ? mdc_commit_open+0x91/0xb0 [mdc] [ 4174.407177] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4174.413530] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4174.418315] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 4174.426136] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4174.432536] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4174.437856] ? do_wait_intr+0xf0/0xf0 [ 4174.442629] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4174.451962] ? kthread+0x1d1/0x200 [ 4174.456355] ? set_kthread_struct+0x70/0x70 [ 4174.463307] ? ret_from_fork+0x1f/0x30 [ 4188.060014] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 4188.063629] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4188.097345] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4188.116585] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4188.132077] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4188.141602] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4188.177454] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4188.190980] RAX: 0000000000000644 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 4188.195018] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 4188.203953] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4188.209607] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 4188.217947] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 4188.220400] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 4188.235437] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4188.245971] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 4188.254988] Call Trace: [ 4188.257429] [ 4188.258652] ? show_regs.cold.9+0x22/0x2f [ 4188.263907] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4188.269446] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4188.276543] ? watchdog+0x40/0x40 [ 4188.279596] ? hrtimer_interrupt+0x128/0x2c0 [ 4188.286783] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4188.291091] ? apic_timer_interrupt+0xf/0x20 [ 4188.299155] [ 4188.300106] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4188.305282] do_raw_spin_lock+0x177/0x190 [ 4188.307496] _raw_spin_lock+0x21/0x30 [ 4188.312585] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4188.315529] ? do_raw_spin_unlock+0x75/0x190 [ 4188.320335] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4188.323465] ? do_wait_intr+0xf0/0xf0 [ 4188.325517] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4188.329323] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4188.335977] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4188.343393] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4188.348166] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4188.352700] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4188.354289] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4188.361903] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4188.367197] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4188.372566] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4188.374619] lmv_intent_open+0x338/0x1150 [lmv] [ 4188.377443] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4188.380246] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4188.383194] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4188.385115] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4188.387226] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4188.389683] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4188.392437] ? do_raw_spin_unlock+0x75/0x190 [ 4188.394581] ? _raw_spin_unlock+0x12/0x30 [ 4188.396617] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4188.399417] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4188.401197] ll_atomic_open+0x36a/0xd70 [lustre] [ 4188.403938] lookup_open+0x356/0xb40 [ 4188.405302] do_last+0x313/0xfc0 [ 4188.406942] ? nd_jump_root+0xe5/0x160 [ 4188.408043] ? path_init+0x437/0x520 [ 4188.409099] path_openat+0xf7/0x500 [ 4188.410310] ? do_raw_spin_unlock+0x75/0x190 [ 4188.412538] do_filp_open+0x99/0x140 [ 4188.414296] ? getname_flags+0x6e/0x330 [ 4188.415617] ? __check_object_size+0xff/0x256 [ 4188.417528] ? do_raw_spin_unlock+0x75/0x190 [ 4188.419251] ? _raw_spin_unlock+0x12/0x30 [ 4188.420721] do_sys_openat2+0x2b4/0x410 [ 4188.422858] do_sys_open+0x73/0xa0 [ 4188.424031] __x64_sys_openat+0x24/0x30 [ 4188.425934] do_syscall_64+0xc1/0x440 [ 4188.427538] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4188.430022] RIP: 0033:0x7ff54af45332 [ 4188.431958] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4188.442320] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4188.445338] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 4188.448475] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 4188.451529] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 4188.455130] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 4188.458612] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 4192.060896] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 4192.074279] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4192.141269] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4192.152565] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4192.158230] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4192.170635] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4192.172500] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4192.193193] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4192.196325] RAX: 0000000000007f14 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 4192.204903] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 4192.211246] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 4192.216590] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 4192.222050] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 4192.227123] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 4192.227685] BUG: workqueue lockup - pool [ 4192.238734] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4192.238748] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 4192.238757] Call Trace: [ 4192.240671] [ 4192.240677] ? show_regs.cold.9+0x22/0x2f [ 4192.240692] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4192.240699] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4192.240703] ? watchdog+0x40/0x40 [ 4192.240708] ? hrtimer_interrupt+0x128/0x2c0 [ 4192.240712] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4192.240721] ? apic_timer_interrupt+0xf/0x20 [ 4192.240726] [ 4192.240730] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4192.240737] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4192.240744] do_raw_spin_lock+0x177/0x190 [ 4192.240749] _raw_spin_lock+0x21/0x30 [ 4192.240754] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4192.242296] process_one_work+0x2c8/0x700 [ 4192.242306] worker_thread+0x296/0x6e0 [ 4192.242310] ? rescuer_thread+0x570/0x570 [ 4192.242314] kthread+0x1d1/0x200 [ 4192.242318] ? set_kthread_struct+0x70/0x70 [ 4192.242321] ret_from_fork+0x1f/0x30 [ 4192.483978] cpus=0-3 flags=0x4 nice=0 stuck for 2233s! [ 4192.491974] Showing busy workqueues and worker pools: [ 4192.502695] workqueue events: flags=0x0 [ 4192.518068] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4192.523132] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 4192.540733] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4192.549031] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 4192.567502] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4192.586056] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 4192.603136] workqueue events_long: flags=0x0 [ 4192.614386] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4192.620448] pending: lu_objects_flush [obdclass] [ 4192.622622] workqueue events_unbound: flags=0x2 [ 4192.629703] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4192.634206] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4192.647076] workqueue events_power_efficient: flags=0x80 [ 4192.653045] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4192.657301] pending: neigh_periodic_work [ 4192.661136] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4192.665349] pending: do_cache_clean, check_lifetime [ 4192.669340] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4192.676697] pending: neigh_periodic_work [ 4192.680720] workqueue writeback: flags=0x4a [ 4192.683201] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4192.686669] pending: wb_workfn, wb_workfn [ 4192.688910] workqueue xprtiod: flags=0xa [ 4192.691175] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4192.694345] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 4192.696890] workqueue ipv6_addrconf: flags=0x40008 [ 4192.698361] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4192.700374] pending: addrconf_verify_work [ 4192.702351] workqueue knbd0-recv: flags=0x1a [ 4192.704188] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4192.708712] in-flight: 657:recv_work [ 4192.710896] workqueue ptlrpc_pinger: flags=0x2 [ 4192.713072] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4192.716144] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 4192.719882] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2233s workers=4 idle: 20831 12642 28859 [ 4192.728517] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4182s workers=2 idle: 659 [ 4200.058220] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 4200.066210] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4200.109364] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4200.126280] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4200.135263] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4200.137287] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4200.162111] RSP: 0018:ffffaf9fc1543ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4200.180467] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 4200.197113] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 4200.205867] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4200.227684] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 4200.237167] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 4200.247398] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 4200.259095] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4200.266329] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 4200.271131] Call Trace: [ 4200.284125] [ 4200.287768] ? show_regs.cold.9+0x22/0x2f [ 4200.292077] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4200.295187] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4200.301102] ? watchdog+0x40/0x40 [ 4200.302344] ? hrtimer_interrupt+0x128/0x2c0 [ 4200.307316] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4200.315132] ? apic_timer_interrupt+0xf/0x20 [ 4200.320507] [ 4200.322123] ? native_safe_halt+0x2e/0x50 [ 4200.333851] kvm_wait+0xcd/0xe0 [ 4200.336293] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4200.353161] do_raw_spin_lock+0x177/0x190 [ 4200.358779] _raw_spin_lock+0x21/0x30 [ 4200.363885] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4200.373122] ? do_raw_spin_unlock+0x75/0x190 [ 4200.376386] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4200.380412] ? do_wait_intr+0xf0/0xf0 [ 4200.390214] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4200.392143] mdc_close+0x53b/0xd70 [mdc] [ 4200.399117] lmv_close+0x18d/0x490 [lmv] [ 4200.400334] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 4200.402345] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4200.404187] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4200.405857] ll_md_close+0x274/0x800 [lustre] [ 4200.408975] ll_file_release+0x128/0xc50 [lustre] [ 4200.410171] __fput+0x10d/0x440 [ 4200.411115] ____fput+0x12/0x20 [ 4200.412029] task_work_run+0xc6/0x110 [ 4200.416942] exit_to_usermode_loop+0x1dd/0x1f0 [ 4200.418033] do_syscall_64+0x3d6/0x440 [ 4200.421189] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4200.424932] RIP: 0033:0x7ff54af45cd8 [ 4200.426241] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4200.435404] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4200.439117] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 4200.442791] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 4200.448830] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 4200.455025] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 4200.459357] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4216.060770] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 4216.078754] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4216.149513] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4216.167009] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4216.187012] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4216.196104] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4216.211871] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4216.215159] RAX: 0000000000007f85 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 4216.220066] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 4216.227669] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4216.231117] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 4216.234673] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 4216.237355] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 4216.240161] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4216.244909] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 4216.250663] Call Trace: [ 4216.252612] [ 4216.253560] ? show_regs.cold.9+0x22/0x2f [ 4216.255678] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4216.260148] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4216.263605] ? watchdog+0x40/0x40 [ 4216.266018] ? hrtimer_interrupt+0x128/0x2c0 [ 4216.269629] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4216.272606] ? apic_timer_interrupt+0xf/0x20 [ 4216.278497] [ 4216.280255] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4216.286116] do_raw_spin_lock+0x177/0x190 [ 4216.293867] _raw_spin_lock+0x21/0x30 [ 4216.310103] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4216.313481] ? do_raw_spin_unlock+0x75/0x190 [ 4216.316452] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4216.319708] ? do_wait_intr+0xf0/0xf0 [ 4216.321821] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4216.325094] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4216.328323] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4216.332585] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4216.335796] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4216.338907] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4216.343171] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4216.347816] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4216.352143] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4216.356900] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4216.361885] lmv_intent_open+0x338/0x1150 [lmv] [ 4216.365096] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4216.369820] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4216.372171] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4216.378543] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4216.382422] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4216.386838] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4216.390882] ? do_raw_spin_unlock+0x75/0x190 [ 4216.393821] ? _raw_spin_unlock+0x12/0x30 [ 4216.396961] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4216.404335] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4216.415955] ll_atomic_open+0x36a/0xd70 [lustre] [ 4216.423292] lookup_open+0x356/0xb40 [ 4216.430138] do_last+0x313/0xfc0 [ 4216.435085] ? nd_jump_root+0xe5/0x160 [ 4216.441804] ? path_init+0x437/0x520 [ 4216.444124] path_openat+0xf7/0x500 [ 4216.447354] ? do_raw_spin_unlock+0x75/0x190 [ 4216.452987] do_filp_open+0x99/0x140 [ 4216.456915] ? getname_flags+0x6e/0x330 [ 4216.460948] ? __check_object_size+0xff/0x256 [ 4216.465920] ? do_raw_spin_unlock+0x75/0x190 [ 4216.469404] ? _raw_spin_unlock+0x12/0x30 [ 4216.471006] do_sys_openat2+0x2b4/0x410 [ 4216.477647] do_sys_open+0x73/0xa0 [ 4216.482699] __x64_sys_openat+0x24/0x30 [ 4216.491198] do_syscall_64+0xc1/0x440 [ 4216.502240] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4216.504450] RIP: 0033:0x7ff54af45332 [ 4216.509423] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4216.521251] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4216.527257] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 4216.531913] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 4216.534106] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 4216.537048] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 4216.542669] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 4220.055862] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [ 4220.059479] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 4220.068677] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4220.069405] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4220.069412] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4220.069415] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4220.069430] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4220.069435] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4220.069438] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 4220.069441] RDX: 0000000000000edb RSI: 0000000000000003 RDI: ffff96848811cae0 [ 4220.069443] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 4220.069445] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 4220.069447] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4220.069450] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 4220.069453] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4220.069638] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 4220.069646] Call Trace: [ 4220.069652] [ 4220.069658] ? show_regs.cold.9+0x22/0x2f [ 4220.069667] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4220.069674] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4220.069679] ? watchdog+0x40/0x40 [ 4220.069683] ? hrtimer_interrupt+0x128/0x2c0 [ 4220.069688] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4220.069697] ? apic_timer_interrupt+0xf/0x20 [ 4220.069702] [ 4220.069705] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4220.069710] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4220.069715] do_raw_spin_lock+0x177/0x190 [ 4220.069719] _raw_spin_lock+0x21/0x30 [ 4220.069726] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4220.077960] Modules linked in: lustre(O) osp(O) [ 4220.123103] obd_mod_free+0x46/0x140 [ptlrpc] [ 4220.131638] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4220.131840] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4220.131847] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4220.131855] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4220.132149] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4220.132162] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4220.132167] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4220.132171] RAX: 0000000000006252 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 4220.132174] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 4220.132176] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 4220.132178] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 4220.132181] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 4220.132184] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 4220.132187] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4220.132190] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 4220.132200] Call Trace: [ 4220.132207] [ 4220.132212] ? show_regs.cold.9+0x22/0x2f [ 4220.132221] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4220.132228] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4220.132233] ? watchdog+0x40/0x40 [ 4220.132238] ? hrtimer_interrupt+0x128/0x2c0 [ 4220.132242] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4220.132251] ? apic_timer_interrupt+0xf/0x20 [ 4220.132256] [ 4220.132259] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4220.132265] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4220.132271] do_raw_spin_lock+0x177/0x190 [ 4220.132276] _raw_spin_lock+0x21/0x30 [ 4220.132281] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4220.132573] process_one_work+0x2c8/0x700 [ 4220.132581] worker_thread+0x296/0x6e0 [ 4220.132586] ? rescuer_thread+0x570/0x570 [ 4220.132590] kthread+0x1d1/0x200 [ 4220.132594] ? set_kthread_struct+0x70/0x70 [ 4220.132598] ret_from_fork+0x1f/0x30 [ 4220.542838] mdc_commit_open+0x91/0xb0 [mdc] [ 4220.546542] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4220.552268] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4220.557571] after_reply+0xe2c/0x1780 [ptlrpc] [ 4220.561356] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4220.564353] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4220.568422] ? do_wait_intr+0xf0/0xf0 [ 4220.571393] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4220.578753] kthread+0x1d1/0x200 [ 4220.580643] ? set_kthread_struct+0x70/0x70 [ 4220.583012] ret_from_fork+0x1f/0x30 [ 4222.943027] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2264s! [ 4222.949646] Showing busy workqueues and worker pools: [ 4222.955827] workqueue events: flags=0x0 [ 4222.965855] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4222.975103] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 4222.986733] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4222.989973] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 4223.000289] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4223.006151] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 4223.015852] workqueue events_long: flags=0x0 [ 4223.020984] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4223.026070] pending: lu_objects_flush [obdclass] [ 4223.027923] workqueue events_unbound: flags=0x2 [ 4223.039062] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4223.044643] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4223.055128] workqueue events_power_efficient: flags=0x80 [ 4223.065218] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4223.077230] pending: neigh_periodic_work [ 4223.090360] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4223.099399] pending: do_cache_clean, check_lifetime [ 4223.103731] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4223.108964] pending: neigh_periodic_work [ 4223.111724] workqueue writeback: flags=0x4a [ 4223.115228] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4223.119231] pending: wb_workfn, wb_workfn [ 4223.121640] workqueue xprtiod: flags=0xa [ 4223.124249] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4223.127630] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 4223.134570] workqueue ipv6_addrconf: flags=0x40008 [ 4223.137514] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4223.141608] pending: addrconf_verify_work [ 4223.144596] workqueue knbd0-recv: flags=0x1a [ 4223.147071] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4223.150208] in-flight: 657:recv_work [ 4223.153021] workqueue ptlrpc_pinger: flags=0x2 [ 4223.155404] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4223.157945] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 4223.160660] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2264s workers=4 idle: 20831 12642 28859 [ 4223.165597] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4213s workers=2 idle: 659 [ 4228.058292] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 4228.067958] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4228.106229] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4228.119772] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4228.131982] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4228.135910] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4228.166173] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4228.182841] RAX: 0000000000007e8b RBX: ffff96848811cae0 RCX: 0000000000000001 [ 4228.193970] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 4228.204013] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4228.215173] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 4228.217865] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 4228.221070] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 4228.224523] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4228.227933] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 4228.231193] Call Trace: [ 4228.232388] [ 4228.233365] ? show_regs.cold.9+0x22/0x2f [ 4228.235177] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4228.237433] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4228.239494] ? watchdog+0x40/0x40 [ 4228.241035] ? hrtimer_interrupt+0x128/0x2c0 [ 4228.243067] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4228.245149] ? apic_timer_interrupt+0xf/0x20 [ 4228.247177] [ 4228.248187] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4228.251049] do_raw_spin_lock+0x177/0x190 [ 4228.252830] _raw_spin_lock+0x21/0x30 [ 4228.254957] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4228.257818] ? do_raw_spin_unlock+0x75/0x190 [ 4228.259730] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4228.262086] ? do_wait_intr+0xf0/0xf0 [ 4228.263530] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4228.265919] mdc_close+0x53b/0xd70 [mdc] [ 4228.267642] lmv_close+0x18d/0x490 [lmv] [ 4228.270651] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 4228.274952] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4228.278999] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4228.282377] ll_md_close+0x274/0x800 [lustre] [ 4228.284945] ll_file_release+0x128/0xc50 [lustre] [ 4228.287793] __fput+0x10d/0x440 [ 4228.289361] ____fput+0x12/0x20 [ 4228.291123] task_work_run+0xc6/0x110 [ 4228.292862] exit_to_usermode_loop+0x1dd/0x1f0 [ 4228.295068] do_syscall_64+0x3d6/0x440 [ 4228.296854] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4228.300121] RIP: 0033:0x7ff54af45cd8 [ 4228.301801] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4228.314957] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4228.324772] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 4228.335036] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 4228.338681] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 4228.344409] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 4228.355624] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4244.060171] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 4244.063796] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4244.104085] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4244.115038] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4244.121547] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4244.123494] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4244.129875] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4244.140643] RAX: 0000000000000245 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 4244.147915] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 4244.151666] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4244.156795] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 4244.161932] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 4244.171319] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 4244.177575] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4244.181719] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 4244.184939] Call Trace: [ 4244.185910] [ 4244.188108] ? show_regs.cold.9+0x22/0x2f [ 4244.189951] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4244.196068] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4244.199241] ? watchdog+0x40/0x40 [ 4244.201994] ? hrtimer_interrupt+0x128/0x2c0 [ 4244.209144] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4244.211716] ? apic_timer_interrupt+0xf/0x20 [ 4244.213360] [ 4244.214781] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4244.218998] do_raw_spin_lock+0x177/0x190 [ 4244.220661] _raw_spin_lock+0x21/0x30 [ 4244.222321] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4244.231236] ? do_raw_spin_unlock+0x75/0x190 [ 4244.235812] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4244.240373] ? do_wait_intr+0xf0/0xf0 [ 4244.244285] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4244.247593] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4244.253247] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4244.257753] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4244.260334] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4244.265872] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4244.271667] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4244.273583] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4244.278441] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4244.282187] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4244.289901] lmv_intent_open+0x338/0x1150 [lmv] [ 4244.291960] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4244.294907] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4244.300338] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4244.303798] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4244.306354] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4244.308247] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4244.311750] ? do_raw_spin_unlock+0x75/0x190 [ 4244.313826] ? _raw_spin_unlock+0x12/0x30 [ 4244.317666] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4244.320391] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4244.323753] ll_atomic_open+0x36a/0xd70 [lustre] [ 4244.325945] lookup_open+0x356/0xb40 [ 4244.331691] do_last+0x313/0xfc0 [ 4244.333363] ? nd_jump_root+0xe5/0x160 [ 4244.339780] ? path_init+0x437/0x520 [ 4244.340934] path_openat+0xf7/0x500 [ 4244.342093] ? do_raw_spin_unlock+0x75/0x190 [ 4244.351077] do_filp_open+0x99/0x140 [ 4244.352962] ? getname_flags+0x6e/0x330 [ 4244.355996] ? __check_object_size+0xff/0x256 [ 4244.360636] ? do_raw_spin_unlock+0x75/0x190 [ 4244.363030] ? _raw_spin_unlock+0x12/0x30 [ 4244.365333] do_sys_openat2+0x2b4/0x410 [ 4244.367628] do_sys_open+0x73/0xa0 [ 4244.371365] __x64_sys_openat+0x24/0x30 [ 4244.373673] do_syscall_64+0xc1/0x440 [ 4244.374941] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4244.378952] RIP: 0033:0x7ff54af45332 [ 4244.380139] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4244.390935] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4244.397928] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 4244.403911] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 4244.409061] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 4244.416978] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 4244.427062] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 4248.055027] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [ 4248.059033] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 4248.062749] Modules linked in: [ 4248.065936] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4248.066023] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4248.066031] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4248.066039] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4248.105060] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4248.106216] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4248.106234] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4248.106237] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4248.106250] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4248.106255] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4248.106259] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 4248.106261] RDX: 0000000000007f82 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 4248.106263] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 4248.106265] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 4248.106268] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4248.106271] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 4248.106273] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4248.106276] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 4248.106283] Call Trace: [ 4248.106288] [ 4248.106295] ? show_regs.cold.9+0x22/0x2f [ 4248.106304] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4248.106310] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4248.106314] ? watchdog+0x40/0x40 [ 4248.106318] ? hrtimer_interrupt+0x128/0x2c0 [ 4248.106321] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4248.106330] ? apic_timer_interrupt+0xf/0x20 [ 4248.106335] [ 4248.106338] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4248.106342] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4248.106346] do_raw_spin_lock+0x177/0x190 [ 4248.106351] _raw_spin_lock+0x21/0x30 [ 4248.106357] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4248.106732] obd_mod_free+0x46/0x140 [ptlrpc] [ 4248.113692] [ 4248.113701] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4248.113719] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4248.113723] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4248.113729] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 4248.113731] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 4248.113733] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 4248.113735] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 4248.113738] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 4248.113740] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 4248.113742] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4248.113744] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 4248.113751] Call Trace: [ 4248.113756] [ 4248.113761] ? show_regs.cold.9+0x22/0x2f [ 4248.113768] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4248.113772] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4248.113775] ? watchdog+0x40/0x40 [ 4248.113778] ? hrtimer_interrupt+0x128/0x2c0 [ 4248.113780] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4248.113785] ? apic_timer_interrupt+0xf/0x20 [ 4248.113788] [ 4248.113790] ? native_safe_halt+0x2e/0x50 [ 4248.113792] kvm_wait+0xcd/0xe0 [ 4248.113798] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4248.113805] do_raw_spin_lock+0x177/0x190 [ 4248.113808] _raw_spin_lock+0x21/0x30 [ 4248.113811] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4248.114041] process_one_work+0x2c8/0x700 [ 4248.114047] worker_thread+0x296/0x6e0 [ 4248.114478] ? rescuer_thread+0x570/0x570 [ 4248.114487] kthread+0x1d1/0x200 [ 4248.114491] ? set_kthread_struct+0x70/0x70 [ 4248.114495] ret_from_fork+0x1f/0x30 [ 4248.450910] mdc_commit_open+0x91/0xb0 [mdc] [ 4248.453823] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4248.455848] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4248.458505] after_reply+0xe2c/0x1780 [ptlrpc] [ 4248.461228] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4248.463934] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4248.466108] ? do_wait_intr+0xf0/0xf0 [ 4248.467742] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4248.470551] kthread+0x1d1/0x200 [ 4248.471927] ? set_kthread_struct+0x70/0x70 [ 4248.474229] ret_from_fork+0x1f/0x30 [ 4253.669352] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2294s! [ 4253.674388] Showing busy workqueues and worker pools: [ 4253.678686] workqueue events: flags=0x0 [ 4253.686607] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4253.695008] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 4253.701846] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4253.707599] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 4253.726368] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4253.732576] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 4253.738977] workqueue events_long: flags=0x0 [ 4253.743328] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4253.747695] pending: lu_objects_flush [obdclass] [ 4253.750740] workqueue events_unbound: flags=0x2 [ 4253.753193] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4253.755938] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4253.761928] workqueue events_power_efficient: flags=0x80 [ 4253.764798] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4253.767734] pending: neigh_periodic_work [ 4253.769336] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4253.772564] pending: do_cache_clean, check_lifetime [ 4253.774762] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4253.778219] pending: neigh_periodic_work [ 4253.781573] workqueue writeback: flags=0x4a [ 4253.783678] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4253.786432] pending: wb_workfn, wb_workfn [ 4253.790188] workqueue xprtiod: flags=0xa [ 4253.793848] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4253.797515] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 4253.801919] workqueue ipv6_addrconf: flags=0x40008 [ 4253.804746] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4253.808959] pending: addrconf_verify_work [ 4253.810402] workqueue knbd0-recv: flags=0x1a [ 4253.816990] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4253.823302] in-flight: 657:recv_work [ 4253.826602] workqueue ptlrpc_pinger: flags=0x2 [ 4253.830990] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4253.834433] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 4253.837423] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2294s workers=4 idle: 20831 12642 28859 [ 4253.843520] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4244s workers=2 idle: 659 [ 4256.059364] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 4256.065138] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4256.123568] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4256.133558] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4256.140497] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4256.145605] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4256.160848] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4256.165479] RAX: 0000000000004607 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 4256.169590] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 4256.176463] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4256.183229] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 4256.189264] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 4256.194473] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 4256.205432] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4256.211434] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 4256.219596] Call Trace: [ 4256.221494] [ 4256.224201] ? show_regs.cold.9+0x22/0x2f [ 4256.229729] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4256.236580] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4256.244435] ? watchdog+0x40/0x40 [ 4256.247835] ? hrtimer_interrupt+0x128/0x2c0 [ 4256.252916] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4256.258785] ? apic_timer_interrupt+0xf/0x20 [ 4256.266275] [ 4256.267449] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4256.270023] do_raw_spin_lock+0x177/0x190 [ 4256.272547] _raw_spin_lock+0x21/0x30 [ 4256.275466] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4256.279501] ? do_raw_spin_unlock+0x75/0x190 [ 4256.285058] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4256.291218] ? do_wait_intr+0xf0/0xf0 [ 4256.299745] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4256.308513] mdc_close+0x53b/0xd70 [mdc] [ 4256.309964] lmv_close+0x18d/0x490 [lmv] [ 4256.315525] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 4256.320334] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4256.328681] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4256.331749] ll_md_close+0x274/0x800 [lustre] [ 4256.338566] ll_file_release+0x128/0xc50 [lustre] [ 4256.339954] __fput+0x10d/0x440 [ 4256.342527] ____fput+0x12/0x20 [ 4256.343812] task_work_run+0xc6/0x110 [ 4256.348173] exit_to_usermode_loop+0x1dd/0x1f0 [ 4256.352962] do_syscall_64+0x3d6/0x440 [ 4256.355920] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4256.361313] RIP: 0033:0x7ff54af45cd8 [ 4256.365379] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4256.376811] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4256.387522] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 4256.390722] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 4256.395762] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 4256.401854] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 4256.414719] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4272.062625] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 4272.068562] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4272.106160] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4272.112693] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4272.119062] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4272.122606] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4272.144409] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4272.148796] RAX: 0000000000007f88 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 4272.159350] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 4272.168423] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4272.178139] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 4272.182917] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 4272.192626] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 4272.196441] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4272.199171] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 4272.203153] Call Trace: [ 4272.205733] [ 4272.206683] ? show_regs.cold.9+0x22/0x2f [ 4272.208352] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4272.210813] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4272.212727] ? watchdog+0x40/0x40 [ 4272.214378] ? hrtimer_interrupt+0x128/0x2c0 [ 4272.216283] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4272.218367] ? apic_timer_interrupt+0xf/0x20 [ 4272.220453] [ 4272.221352] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4272.224091] do_raw_spin_lock+0x177/0x190 [ 4272.225956] _raw_spin_lock+0x21/0x30 [ 4272.229371] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4272.236579] ? do_raw_spin_unlock+0x75/0x190 [ 4272.240867] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4272.244398] ? do_wait_intr+0xf0/0xf0 [ 4272.247140] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4272.250256] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4272.253424] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4272.256236] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4272.258763] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4272.261878] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4272.263777] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4272.265605] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4272.267552] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4272.270332] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4272.272761] lmv_intent_open+0x338/0x1150 [lmv] [ 4272.274571] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4272.285907] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4272.287935] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4272.289498] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4272.294409] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4272.297062] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4272.300361] ? do_raw_spin_unlock+0x75/0x190 [ 4272.302234] ? _raw_spin_unlock+0x12/0x30 [ 4272.304020] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4272.306415] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4272.308672] ll_atomic_open+0x36a/0xd70 [lustre] [ 4272.311192] lookup_open+0x356/0xb40 [ 4272.312875] do_last+0x313/0xfc0 [ 4272.314384] ? nd_jump_root+0xe5/0x160 [ 4272.316381] ? path_init+0x437/0x520 [ 4272.319673] path_openat+0xf7/0x500 [ 4272.321677] ? do_raw_spin_unlock+0x75/0x190 [ 4272.324255] do_filp_open+0x99/0x140 [ 4272.326711] ? getname_flags+0x6e/0x330 [ 4272.329000] ? __check_object_size+0xff/0x256 [ 4272.331400] ? do_raw_spin_unlock+0x75/0x190 [ 4272.338944] ? _raw_spin_unlock+0x12/0x30 [ 4272.341203] do_sys_openat2+0x2b4/0x410 [ 4272.344016] do_sys_open+0x73/0xa0 [ 4272.345191] __x64_sys_openat+0x24/0x30 [ 4272.349255] do_syscall_64+0xc1/0x440 [ 4272.351174] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4272.354600] RIP: 0033:0x7ff54af45332 [ 4272.357177] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4272.366213] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4272.369559] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 4272.372694] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 4272.375977] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 4272.383842] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 4272.392439] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 4276.059254] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 4276.066754] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4276.088136] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4276.095179] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4276.099127] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4276.101684] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4276.104204] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4276.111884] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4276.116737] RAX: 0000000000007790 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 4276.123069] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 4276.126346] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 4276.133060] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 4276.137757] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 4276.143127] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 4276.149334] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4276.153403] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 4276.156884] Call Trace: [ 4276.157974] [ 4276.158711] ? show_regs.cold.9+0x22/0x2f [ 4276.160275] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4276.163096] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4276.165244] ? watchdog+0x40/0x40 [ 4276.167427] ? hrtimer_interrupt+0x128/0x2c0 [ 4276.169658] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4276.171550] ? apic_timer_interrupt+0xf/0x20 [ 4276.173240] [ 4276.174100] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4276.176533] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4276.178860] do_raw_spin_lock+0x177/0x190 [ 4276.180536] _raw_spin_lock+0x21/0x30 [ 4276.182524] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4276.184709] process_one_work+0x2c8/0x700 [ 4276.186364] worker_thread+0x296/0x6e0 [ 4276.187886] ? rescuer_thread+0x570/0x570 [ 4276.189427] kthread+0x1d1/0x200 [ 4276.190657] ? set_kthread_struct+0x70/0x70 [ 4276.192348] ret_from_fork+0x1f/0x30 [ 4280.055482] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [ 4280.060272] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4280.124843] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4280.138078] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4280.144265] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4280.154585] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4280.171828] RSP: 0000:ffffaf9fc1b97bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4280.178983] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 4280.186165] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 4280.189452] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 4280.193608] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 4280.200484] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4280.208703] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 4280.227353] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4280.236637] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 4280.248219] Call Trace: [ 4280.257284] [ 4280.261465] ? show_regs.cold.9+0x22/0x2f [ 4280.266392] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4280.269501] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4280.278398] ? watchdog+0x40/0x40 [ 4280.282133] ? hrtimer_interrupt+0x128/0x2c0 [ 4280.290584] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4280.296574] ? apic_timer_interrupt+0xf/0x20 [ 4280.301625] [ 4280.304596] ? native_safe_halt+0x2e/0x50 [ 4280.306698] kvm_wait+0xcd/0xe0 [ 4280.308611] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4280.314408] do_raw_spin_lock+0x177/0x190 [ 4280.324038] _raw_spin_lock+0x21/0x30 [ 4280.328017] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4280.331699] obd_mod_free+0x46/0x140 [ptlrpc] [ 4280.338655] mdc_commit_open+0x91/0xb0 [mdc] [ 4280.343015] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4280.347905] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4280.351567] after_reply+0xe2c/0x1780 [ptlrpc] [ 4280.356442] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4280.362034] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4280.365411] ? do_wait_intr+0xf0/0xf0 [ 4280.366955] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4280.370063] kthread+0x1d1/0x200 [ 4280.372003] ? set_kthread_struct+0x70/0x70 [ 4280.375420] ret_from_fork+0x1f/0x30 [ 4283.872661] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2324s! [ 4283.909309] Showing busy workqueues and worker pools: [ 4283.912629] workqueue events: flags=0x0 [ 4283.921249] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4283.924389] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 4283.939240] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4283.950970] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 4283.963518] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4283.968887] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 4283.972445] workqueue events_long: flags=0x0 [ 4283.973943] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4283.976202] pending: lu_objects_flush [obdclass] [ 4283.978531] workqueue events_unbound: flags=0x2 [ 4283.980351] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4283.982406] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4283.986446] workqueue events_power_efficient: flags=0x80 [ 4283.989410] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4283.992499] pending: neigh_periodic_work [ 4283.994203] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4283.996423] pending: do_cache_clean, check_lifetime [ 4283.998278] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4284.000230] pending: neigh_periodic_work [ 4284.001717] workqueue writeback: flags=0x4a [ 4284.002941] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4284.004524] pending: wb_workfn, wb_workfn [ 4284.005956] workqueue xprtiod: flags=0xa [ 4284.007185] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4284.010424] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 4284.012712] workqueue ipv6_addrconf: flags=0x40008 [ 4284.014350] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4284.017166] pending: addrconf_verify_work [ 4284.018317] workqueue knbd0-recv: flags=0x1a [ 4284.019500] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4284.023859] in-flight: 657:recv_work [ 4284.025076] workqueue ptlrpc_pinger: flags=0x2 [ 4284.026101] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4284.027477] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 4284.033874] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2325s workers=4 idle: 20831 12642 28859 [ 4284.041886] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4274s workers=2 idle: 659 [ 4284.059248] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 4284.064828] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4284.082421] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4284.086992] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4284.094511] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4284.098590] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4284.112826] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4284.116035] RAX: 000000000000758d RBX: ffff96848811cae0 RCX: 0000000000000001 [ 4284.119581] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 4284.125447] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4284.128416] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 4284.131376] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 4284.137898] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 4284.144188] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4284.149560] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 4284.153517] Call Trace: [ 4284.156530] [ 4284.160273] ? show_regs.cold.9+0x22/0x2f [ 4284.161661] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4284.172110] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4284.175579] ? watchdog+0x40/0x40 [ 4284.177267] ? hrtimer_interrupt+0x128/0x2c0 [ 4284.182769] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4284.185246] ? apic_timer_interrupt+0xf/0x20 [ 4284.186957] [ 4284.187818] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4284.190080] do_raw_spin_lock+0x177/0x190 [ 4284.191583] _raw_spin_lock+0x21/0x30 [ 4284.192954] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4284.195068] ? do_raw_spin_unlock+0x75/0x190 [ 4284.196572] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4284.200372] ? do_wait_intr+0xf0/0xf0 [ 4284.205714] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4284.207792] mdc_close+0x53b/0xd70 [mdc] [ 4284.209232] lmv_close+0x18d/0x490 [lmv] [ 4284.211148] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 4284.213901] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4284.216542] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4284.218535] ll_md_close+0x274/0x800 [lustre] [ 4284.220605] ll_file_release+0x128/0xc50 [lustre] [ 4284.222994] __fput+0x10d/0x440 [ 4284.224421] ____fput+0x12/0x20 [ 4284.225959] task_work_run+0xc6/0x110 [ 4284.227549] exit_to_usermode_loop+0x1dd/0x1f0 [ 4284.229513] do_syscall_64+0x3d6/0x440 [ 4284.231124] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4284.233361] RIP: 0033:0x7ff54af45cd8 [ 4284.234992] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4284.242912] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4284.245688] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 4284.248302] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 4284.251259] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 4284.254262] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 4284.256784] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4300.060273] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 4300.062816] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4300.102541] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4300.107440] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4300.111168] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4300.113377] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4300.132498] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4300.142482] RAX: 0000000000004b8c RBX: ffff96848811cae0 RCX: 0000000000000001 [ 4300.154649] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 4300.158850] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4300.162337] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 4300.167673] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 4300.173290] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 4300.175253] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4300.179279] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 4300.180669] Call Trace: [ 4300.186262] [ 4300.187491] ? show_regs.cold.9+0x22/0x2f [ 4300.190732] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4300.197169] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4300.200830] ? watchdog+0x40/0x40 [ 4300.207888] ? hrtimer_interrupt+0x128/0x2c0 [ 4300.210253] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4300.216935] ? apic_timer_interrupt+0xf/0x20 [ 4300.220701] [ 4300.222527] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4300.226366] do_raw_spin_lock+0x177/0x190 [ 4300.231146] _raw_spin_lock+0x21/0x30 [ 4300.233463] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4300.238835] ? do_raw_spin_unlock+0x75/0x190 [ 4300.248922] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4300.254103] ? do_wait_intr+0xf0/0xf0 [ 4300.255855] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4300.258264] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4300.260317] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4300.262515] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4300.264540] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4300.267261] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4300.268773] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4300.270595] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4300.272426] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4300.275235] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4300.277358] lmv_intent_open+0x338/0x1150 [lmv] [ 4300.279003] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4300.281233] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4300.283258] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4300.284798] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4300.286334] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4300.288000] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4300.289531] ? do_raw_spin_unlock+0x75/0x190 [ 4300.290508] ? _raw_spin_unlock+0x12/0x30 [ 4300.291795] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4300.293410] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4300.295433] ll_atomic_open+0x36a/0xd70 [lustre] [ 4300.297824] lookup_open+0x356/0xb40 [ 4300.299398] do_last+0x313/0xfc0 [ 4300.300777] ? nd_jump_root+0xe5/0x160 [ 4300.302474] ? path_init+0x437/0x520 [ 4300.303856] path_openat+0xf7/0x500 [ 4300.304690] ? do_raw_spin_unlock+0x75/0x190 [ 4300.306234] do_filp_open+0x99/0x140 [ 4300.307662] ? getname_flags+0x6e/0x330 [ 4300.308849] ? __check_object_size+0xff/0x256 [ 4300.310257] ? do_raw_spin_unlock+0x75/0x190 [ 4300.311490] ? _raw_spin_unlock+0x12/0x30 [ 4300.312741] do_sys_openat2+0x2b4/0x410 [ 4300.314017] do_sys_open+0x73/0xa0 [ 4300.315715] __x64_sys_openat+0x24/0x30 [ 4300.317225] do_syscall_64+0xc1/0x440 [ 4300.318734] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4300.320605] RIP: 0033:0x7ff54af45332 [ 4300.321800] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4300.327647] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4300.330005] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 4300.332645] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 4300.335261] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 4300.337298] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 4300.339745] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 4304.069010] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 4304.071559] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4304.129387] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4304.133489] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4304.141288] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4304.143210] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4304.145357] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4304.152855] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4304.158445] RAX: 0000000000005a0d RBX: ffff96848811cae0 RCX: 0000000000000001 [ 4304.162601] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 4304.165459] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 4304.168750] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 4304.171455] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 4304.178370] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 4304.183345] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4304.185436] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 4304.188010] Call Trace: [ 4304.188736] [ 4304.189456] ? show_regs.cold.9+0x22/0x2f [ 4304.191158] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4304.192617] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4304.194531] ? watchdog+0x40/0x40 [ 4304.195655] ? hrtimer_interrupt+0x128/0x2c0 [ 4304.199396] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4304.201866] ? apic_timer_interrupt+0xf/0x20 [ 4304.202826] [ 4304.203326] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4304.207750] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4304.209791] do_raw_spin_lock+0x177/0x190 [ 4304.211301] _raw_spin_lock+0x21/0x30 [ 4304.212196] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4304.213529] process_one_work+0x2c8/0x700 [ 4304.214420] worker_thread+0x296/0x6e0 [ 4304.217016] ? rescuer_thread+0x570/0x570 [ 4304.218236] kthread+0x1d1/0x200 [ 4304.221477] ? set_kthread_struct+0x70/0x70 [ 4304.225309] ret_from_fork+0x1f/0x30 [ 4308.060290] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [ 4308.063147] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4308.096614] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4308.111567] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4308.119934] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4308.124257] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4308.134725] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4308.139163] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 4308.144036] RDX: 0000000000000430 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 4308.146148] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 4308.149111] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 4308.153679] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4308.156267] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 4308.161277] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4308.164272] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 4308.168551] Call Trace: [ 4308.169701] [ 4308.170501] ? show_regs.cold.9+0x22/0x2f [ 4308.172873] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4308.175266] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4308.177997] ? watchdog+0x40/0x40 [ 4308.180998] ? hrtimer_interrupt+0x128/0x2c0 [ 4308.182966] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4308.184497] ? apic_timer_interrupt+0xf/0x20 [ 4308.186006] [ 4308.187257] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4308.190258] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4308.192041] do_raw_spin_lock+0x177/0x190 [ 4308.195200] _raw_spin_lock+0x21/0x30 [ 4308.197344] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4308.200346] obd_mod_free+0x46/0x140 [ptlrpc] [ 4308.203116] mdc_commit_open+0x91/0xb0 [mdc] [ 4308.204789] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4308.208717] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4308.211334] after_reply+0xe2c/0x1780 [ptlrpc] [ 4308.213712] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4308.218139] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4308.223876] ? do_wait_intr+0xf0/0xf0 [ 4308.227846] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4308.229997] kthread+0x1d1/0x200 [ 4308.233626] ? set_kthread_struct+0x70/0x70 [ 4308.235281] ret_from_fork+0x1f/0x30 [ 4312.061684] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 4312.064542] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4312.090281] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4312.095236] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4312.099719] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4312.102311] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4312.111154] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 4312.116681] RAX: 0000000000007d0a RBX: ffff96848811cae0 RCX: 0000000000000001 [ 4312.121149] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 4312.125184] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4312.129510] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 4312.133027] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 4312.136697] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 4312.140497] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4312.143618] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 4312.147233] Call Trace: [ 4312.148252] [ 4312.149269] ? show_regs.cold.9+0x22/0x2f [ 4312.150980] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4312.152887] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4312.155545] ? watchdog+0x40/0x40 [ 4312.156899] ? hrtimer_interrupt+0x128/0x2c0 [ 4312.159067] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4312.161753] ? apic_timer_interrupt+0xf/0x20 [ 4312.163962] [ 4312.165060] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4312.168343] do_raw_spin_lock+0x177/0x190 [ 4312.171328] _raw_spin_lock+0x21/0x30 [ 4312.173721] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4312.178359] ? do_raw_spin_unlock+0x75/0x190 [ 4312.181194] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4312.187965] ? do_wait_intr+0xf0/0xf0 [ 4312.194576] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4312.203260] mdc_close+0x53b/0xd70 [mdc] [ 4312.206265] lmv_close+0x18d/0x490 [lmv] [ 4312.213901] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 4312.223695] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4312.231951] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4312.234956] ll_md_close+0x274/0x800 [lustre] [ 4312.241965] ll_file_release+0x128/0xc50 [lustre] [ 4312.247580] __fput+0x10d/0x440 [ 4312.252405] ____fput+0x12/0x20 [ 4312.258213] task_work_run+0xc6/0x110 [ 4312.267219] exit_to_usermode_loop+0x1dd/0x1f0 [ 4312.273412] do_syscall_64+0x3d6/0x440 [ 4312.275195] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4312.280344] RIP: 0033:0x7ff54af45cd8 [ 4312.287301] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4312.302327] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4312.314254] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 4312.324229] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 4312.333915] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 4312.347131] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 4312.354986] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4314.085486] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2355s! [ 4314.100319] Showing busy workqueues and worker pools: [ 4314.107433] workqueue events: flags=0x0 [ 4314.109153] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4314.123935] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 4314.147287] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4314.164738] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 4314.190290] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4314.202355] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 4314.223862] workqueue events_long: flags=0x0 [ 4314.225419] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4314.245287] pending: lu_objects_flush [obdclass] [ 4314.249850] workqueue events_unbound: flags=0x2 [ 4314.264822] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4314.271625] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4314.288177] workqueue events_power_efficient: flags=0x80 [ 4314.300669] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4314.317252] pending: neigh_periodic_work [ 4314.318786] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4314.330720] pending: do_cache_clean, check_lifetime [ 4314.333931] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4314.340315] pending: neigh_periodic_work [ 4314.352733] workqueue writeback: flags=0x4a [ 4314.361891] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4314.366321] pending: wb_workfn, wb_workfn [ 4314.368121] workqueue xprtiod: flags=0xa [ 4314.370793] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4314.372439] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 4314.379381] workqueue ipv6_addrconf: flags=0x40008 [ 4314.382748] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4314.384843] pending: addrconf_verify_work [ 4314.386300] workqueue knbd0-recv: flags=0x1a [ 4314.390596] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4314.394234] in-flight: 657:recv_work [ 4314.396690] workqueue ptlrpc_pinger: flags=0x2 [ 4314.406662] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4314.428134] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 4314.442895] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2355s workers=4 idle: 20831 12642 28859 [ 4314.457126] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4304s workers=2 idle: 659 [ 4328.065427] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 4328.075637] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4328.107213] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4328.134149] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4328.143748] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4328.150915] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4328.173601] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4328.184824] RAX: 0000000000007586 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 4328.192139] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 4328.198587] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4328.203591] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 4328.207690] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 4328.211528] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 4328.215736] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4328.218717] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 4328.222257] Call Trace: [ 4328.223752] [ 4328.225210] ? show_regs.cold.9+0x22/0x2f [ 4328.227344] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4328.229846] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4328.232342] ? watchdog+0x40/0x40 [ 4328.234170] ? hrtimer_interrupt+0x128/0x2c0 [ 4328.236170] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4328.240970] ? apic_timer_interrupt+0xf/0x20 [ 4328.245091] [ 4328.246240] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4328.249328] do_raw_spin_lock+0x177/0x190 [ 4328.251517] _raw_spin_lock+0x21/0x30 [ 4328.253918] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4328.260240] ? do_raw_spin_unlock+0x75/0x190 [ 4328.264123] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4328.267350] ? do_wait_intr+0xf0/0xf0 [ 4328.269678] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4328.274004] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4328.277561] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4328.280955] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4328.284262] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4328.287270] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4328.290228] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4328.293094] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4328.296073] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4328.299397] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4328.302675] lmv_intent_open+0x338/0x1150 [lmv] [ 4328.305208] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4328.308505] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4328.313659] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4328.316961] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4328.320401] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4328.324005] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4328.327519] ? do_raw_spin_unlock+0x75/0x190 [ 4328.329828] ? _raw_spin_unlock+0x12/0x30 [ 4328.332610] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4328.336049] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4328.339725] ll_atomic_open+0x36a/0xd70 [lustre] [ 4328.344312] lookup_open+0x356/0xb40 [ 4328.346563] do_last+0x313/0xfc0 [ 4328.348552] ? nd_jump_root+0xe5/0x160 [ 4328.351163] ? path_init+0x437/0x520 [ 4328.353422] path_openat+0xf7/0x500 [ 4328.355847] ? do_raw_spin_unlock+0x75/0x190 [ 4328.358724] do_filp_open+0x99/0x140 [ 4328.361063] ? getname_flags+0x6e/0x330 [ 4328.364260] ? __check_object_size+0xff/0x256 [ 4328.366503] ? do_raw_spin_unlock+0x75/0x190 [ 4328.369177] ? _raw_spin_unlock+0x12/0x30 [ 4328.371785] do_sys_openat2+0x2b4/0x410 [ 4328.374344] do_sys_open+0x73/0xa0 [ 4328.376866] __x64_sys_openat+0x24/0x30 [ 4328.379608] do_syscall_64+0xc1/0x440 [ 4328.382038] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4328.385665] RIP: 0033:0x7ff54af45332 [ 4328.388964] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4328.403048] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4328.407618] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 4328.414262] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 4328.419252] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 4328.423894] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 4328.428766] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 4332.059613] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 4332.065084] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4332.111987] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4332.122568] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4332.136212] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4332.143151] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4332.147426] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4332.170893] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4332.180888] RAX: 0000000000007092 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 4332.187866] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 4332.191205] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 4332.194113] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 4332.203946] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 4332.210298] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 4332.219367] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4332.223620] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 4332.229233] Call Trace: [ 4332.232680] [ 4332.233370] ? show_regs.cold.9+0x22/0x2f [ 4332.238153] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4332.243019] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4332.251944] ? watchdog+0x40/0x40 [ 4332.255187] ? hrtimer_interrupt+0x128/0x2c0 [ 4332.259796] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4332.265665] ? apic_timer_interrupt+0xf/0x20 [ 4332.270545] [ 4332.271151] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4332.275539] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4332.280956] do_raw_spin_lock+0x177/0x190 [ 4332.283266] _raw_spin_lock+0x21/0x30 [ 4332.287300] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4332.290352] process_one_work+0x2c8/0x700 [ 4332.292248] worker_thread+0x296/0x6e0 [ 4332.294008] ? rescuer_thread+0x570/0x570 [ 4332.299825] kthread+0x1d1/0x200 [ 4332.302520] ? set_kthread_struct+0x70/0x70 [ 4332.308120] ret_from_fork+0x1f/0x30 [ 4340.057224] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 4340.059228] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 4340.063723] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4340.063806] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4340.063814] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4340.063817] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4340.063832] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4340.063837] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4340.063842] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 4340.063844] RDX: 00000000000019a5 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 4340.063847] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 4340.063849] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 4340.063851] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4340.063855] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 4340.063858] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4340.063860] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 4340.063868] Call Trace: [ 4340.063874] [ 4340.063879] ? show_regs.cold.9+0x22/0x2f [ 4340.063889] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4340.063896] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4340.063901] ? watchdog+0x40/0x40 [ 4340.063906] ? hrtimer_interrupt+0x128/0x2c0 [ 4340.063909] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4340.063917] ? apic_timer_interrupt+0xf/0x20 [ 4340.063922] [ 4340.063925] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4340.063930] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4340.063935] do_raw_spin_lock+0x177/0x190 [ 4340.063939] _raw_spin_lock+0x21/0x30 [ 4340.063945] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4340.064719] obd_mod_free+0x46/0x140 [ptlrpc] [ 4340.064928] mdc_commit_open+0x91/0xb0 [mdc] [ 4340.064970] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4340.074902] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4340.074988] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4340.074997] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4340.075001] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4340.075018] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4340.075023] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4340.075028] RAX: 0000000000003385 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 4340.075030] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 4340.075032] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4340.075034] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 4340.075037] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 4340.075040] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 4340.075043] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4340.075045] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 4340.075055] Call Trace: [ 4340.075061] [ 4340.075068] ? show_regs.cold.9+0x22/0x2f [ 4340.075079] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4340.075086] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4340.075091] ? watchdog+0x40/0x40 [ 4340.075096] ? hrtimer_interrupt+0x128/0x2c0 [ 4340.075101] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4340.075111] ? apic_timer_interrupt+0xf/0x20 [ 4340.075116] [ 4340.075119] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4340.075126] do_raw_spin_lock+0x177/0x190 [ 4340.075131] _raw_spin_lock+0x21/0x30 [ 4340.075137] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4340.107946] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4340.108382] after_reply+0xe2c/0x1780 [ptlrpc] [ 4340.108613] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4340.108947] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4340.109228] ? do_wait_intr+0xf0/0xf0 [ 4340.109238] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4340.114846] ? do_raw_spin_unlock+0x75/0x190 [ 4340.114868] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4340.115153] ? do_wait_intr+0xf0/0xf0 [ 4340.115216] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4340.115498] mdc_close+0x53b/0xd70 [mdc] [ 4340.115615] lmv_close+0x18d/0x490 [lmv] [ 4340.115646] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 4340.120074] kthread+0x1d1/0x200 [ 4340.120097] ? set_kthread_struct+0x70/0x70 [ 4340.120103] ret_from_fork+0x1f/0x30 [ 4340.474707] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4340.477271] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4340.479724] ll_md_close+0x274/0x800 [lustre] [ 4340.481651] ll_file_release+0x128/0xc50 [lustre] [ 4340.483754] __fput+0x10d/0x440 [ 4340.485052] ____fput+0x12/0x20 [ 4340.486494] task_work_run+0xc6/0x110 [ 4340.488026] exit_to_usermode_loop+0x1dd/0x1f0 [ 4340.491558] do_syscall_64+0x3d6/0x440 [ 4340.493771] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4340.496768] RIP: 0033:0x7ff54af45cd8 [ 4340.500132] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4340.516068] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4340.520842] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 4340.525224] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 4340.546309] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 4340.550556] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 4340.555097] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4344.804008] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2385s! [ 4344.818273] Showing busy workqueues and worker pools: [ 4344.825754] workqueue events: flags=0x0 [ 4344.829789] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4344.839741] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 4344.847875] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4344.856044] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 4344.867417] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4344.874807] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 4344.882061] workqueue events_long: flags=0x0 [ 4344.884969] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4344.891988] pending: lu_objects_flush [obdclass] [ 4344.898748] workqueue events_unbound: flags=0x2 [ 4344.901303] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4344.908772] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4344.921963] workqueue events_power_efficient: flags=0x80 [ 4344.923848] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4344.935749] pending: neigh_periodic_work [ 4344.939706] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4344.946965] pending: do_cache_clean, check_lifetime [ 4344.954085] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4344.960240] pending: neigh_periodic_work [ 4344.969423] workqueue writeback: flags=0x4a [ 4344.974409] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4344.982553] pending: wb_workfn, wb_workfn [ 4344.987961] workqueue xprtiod: flags=0xa [ 4344.992323] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4344.997713] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 4345.010239] workqueue ipv6_addrconf: flags=0x40008 [ 4345.014703] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4345.023747] pending: addrconf_verify_work [ 4345.029747] workqueue knbd0-recv: flags=0x1a [ 4345.033696] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4345.042722] in-flight: 657:recv_work [ 4345.046166] workqueue ptlrpc_pinger: flags=0x2 [ 4345.050032] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4345.056821] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 4345.062467] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2386s workers=4 idle: 20831 12642 28859 [ 4345.071858] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4335s workers=2 idle: 659 [ 4353.911022] rcu: INFO: rcu_sched self-detected stall on CPU [ 4353.915457] rcu: 0-...!: (1261776 ticks this GP) idle=98a/1/0x4000000000000002 softirq=125154/125154 fqs=959 [ 4353.921885] (t=2400071 jiffies g=253585 q=3533) [ 4353.924060] rcu: rcu_sched kthread starved for 2395031 jiffies! g253585 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [ 4353.929352] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 4353.933155] rcu: RCU grace-period kthread stack dump: [ 4353.935240] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 4353.939188] Call Trace: [ 4353.940363] __schedule+0x351/0xcb0 [ 4353.941452] schedule+0xc0/0x180 [ 4353.942269] schedule_timeout+0xb4/0x190 [ 4353.946184] ? do_raw_spin_unlock+0x75/0x190 [ 4353.948471] ? __next_timer_interrupt+0x160/0x160 [ 4353.954395] ? prepare_to_swait_event+0xf3/0x140 [ 4353.957970] rcu_gp_kthread+0x8dd/0x15b0 [ 4353.963523] ? force_qs_rnp+0x2c0/0x2c0 [ 4353.964697] kthread+0x1d1/0x200 [ 4353.965671] ? set_kthread_struct+0x70/0x70 [ 4353.970960] ret_from_fork+0x1f/0x30 [ 4353.972478] rcu: Stack dump where RCU GP kthread last ran: [ 4353.975886] Sending NMI from CPU 0 to CPUs 2: [ 4353.992124] NMI backtrace for cpu 2 [ 4353.992130] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4353.992132] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4353.992134] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4353.992137] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4353.992139] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4353.992141] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 [ 4353.992143] RAX: 000000000000751f RBX: ffff96848811cae0 RCX: 0000000000000001 [ 4353.992144] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 4353.992146] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 4353.992148] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 4353.992149] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 4353.992151] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 4353.992152] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4353.992154] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 4353.992155] Call Trace: [ 4353.992156] [ 4353.992157] ? show_regs.cold.9+0x22/0x2f [ 4353.992158] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 4353.992159] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 4353.992160] ? nmi_handle+0xbd/0x210 [ 4353.992161] ? default_do_nmi+0x4b/0x1b0 [ 4353.992162] ? do_nmi+0x280/0x380 [ 4353.992163] ? end_repeat_nmi+0x16/0x1f [ 4353.992164] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4353.992166] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4353.992167] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4353.992167] [ 4353.992168] do_raw_spin_lock+0x177/0x190 [ 4353.992169] _raw_spin_lock+0x21/0x30 [ 4353.992170] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4353.992171] process_one_work+0x2c8/0x700 [ 4353.992171] worker_thread+0x296/0x6e0 [ 4353.992172] ? rescuer_thread+0x570/0x570 [ 4353.992173] kthread+0x1d1/0x200 [ 4353.992174] ? set_kthread_struct+0x70/0x70 [ 4353.992175] ret_from_fork+0x1f/0x30 [ 4353.992932] NMI backtrace for cpu 0 [ 4354.324503] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4354.344752] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4354.353547] Call Trace: [ 4354.370157] [ 4354.370777] ? dump_stack+0xbb/0x10e [ 4354.371802] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 4354.384102] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 4354.388715] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 4354.393289] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 4354.396273] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 4354.399412] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 4354.401445] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 4354.403631] ? update_process_times+0x6d/0xc0 [ 4354.407264] ? tick_sched_handle.isra.18+0x27/0x90 [ 4354.411865] ? tick_sched_timer+0x53/0xb0 [ 4354.414392] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4354.416099] ? tick_sched_handle.isra.18+0x90/0x90 [ 4354.419936] ? hrtimer_interrupt+0x128/0x2c0 [ 4354.427652] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4354.429005] ? apic_timer_interrupt+0xf/0x20 [ 4354.433534] [ 4354.435262] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4354.438688] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4354.448241] ? do_raw_spin_lock+0x177/0x190 [ 4354.451000] ? _raw_spin_lock+0x21/0x30 [ 4354.454312] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4354.460001] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 4354.465832] ? mdc_commit_open+0x91/0xb0 [mdc] [ 4354.467331] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4354.471678] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4354.473639] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 4354.482682] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4354.490847] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4354.492350] ? do_wait_intr+0xf0/0xf0 [ 4354.494823] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4354.497572] ? kthread+0x1d1/0x200 [ 4354.499168] ? set_kthread_struct+0x70/0x70 [ 4354.500505] ? ret_from_fork+0x1f/0x30 [ 4356.060124] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 4356.079221] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4356.114892] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4356.128722] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4356.137491] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4356.142294] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4356.152958] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4356.158940] RAX: 0000000000005785 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 4356.162672] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 4356.165632] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4356.173807] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 4356.180324] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 4356.182831] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 4356.186540] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4356.192685] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 4356.196828] Call Trace: [ 4356.201115] [ 4356.201666] ? show_regs.cold.9+0x22/0x2f [ 4356.203666] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4356.210474] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4356.213860] ? watchdog+0x40/0x40 [ 4356.216819] ? hrtimer_interrupt+0x128/0x2c0 [ 4356.221882] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4356.224815] ? apic_timer_interrupt+0xf/0x20 [ 4356.226400] [ 4356.226920] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4356.230629] do_raw_spin_lock+0x177/0x190 [ 4356.237532] _raw_spin_lock+0x21/0x30 [ 4356.240297] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4356.242661] ? do_raw_spin_unlock+0x75/0x190 [ 4356.244856] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4356.250659] ? do_wait_intr+0xf0/0xf0 [ 4356.253565] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4356.258658] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4356.262571] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4356.268238] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4356.270316] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4356.273189] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4356.274776] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4356.276597] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4356.278614] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4356.281410] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4356.286801] lmv_intent_open+0x338/0x1150 [lmv] [ 4356.288549] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4356.290794] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4356.292730] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4356.303183] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4356.304995] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4356.308986] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4356.310332] ? do_raw_spin_unlock+0x75/0x190 [ 4356.311379] ? _raw_spin_unlock+0x12/0x30 [ 4356.313617] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4356.317526] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4356.319642] ll_atomic_open+0x36a/0xd70 [lustre] [ 4356.321624] lookup_open+0x356/0xb40 [ 4356.322786] do_last+0x313/0xfc0 [ 4356.323880] ? nd_jump_root+0xe5/0x160 [ 4356.326567] ? path_init+0x437/0x520 [ 4356.328495] path_openat+0xf7/0x500 [ 4356.329822] ? do_raw_spin_unlock+0x75/0x190 [ 4356.333620] do_filp_open+0x99/0x140 [ 4356.334791] ? getname_flags+0x6e/0x330 [ 4356.336708] ? __check_object_size+0xff/0x256 [ 4356.338323] ? do_raw_spin_unlock+0x75/0x190 [ 4356.340081] ? _raw_spin_unlock+0x12/0x30 [ 4356.343801] do_sys_openat2+0x2b4/0x410 [ 4356.344905] do_sys_open+0x73/0xa0 [ 4356.346458] __x64_sys_openat+0x24/0x30 [ 4356.347716] do_syscall_64+0xc1/0x440 [ 4356.349081] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4356.350853] RIP: 0033:0x7ff54af45332 [ 4356.353643] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4356.371086] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4356.377639] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 4356.384659] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 4356.391293] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 4356.397622] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 4356.411321] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 4368.059089] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [ 4368.064123] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4368.186673] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4368.213371] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4368.226623] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4368.240342] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4368.277623] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4368.292419] RAX: 0000000000007f85 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 4368.304307] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 4368.312679] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4368.317471] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 4368.322319] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 4368.336431] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 4368.356269] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4368.365426] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 4368.383090] Call Trace: [ 4368.384001] [ 4368.398136] ? show_regs.cold.9+0x22/0x2f [ 4368.407748] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4368.425817] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4368.434968] ? watchdog+0x40/0x40 [ 4368.438787] ? hrtimer_interrupt+0x128/0x2c0 [ 4368.460778] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4368.467535] ? apic_timer_interrupt+0xf/0x20 [ 4368.482331] [ 4368.487152] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4368.500900] do_raw_spin_lock+0x177/0x190 [ 4368.508702] _raw_spin_lock+0x21/0x30 [ 4368.514821] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4368.531608] ? do_raw_spin_unlock+0x75/0x190 [ 4368.537979] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4368.548181] ? do_wait_intr+0xf0/0xf0 [ 4368.552408] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4368.557269] mdc_close+0x53b/0xd70 [mdc] [ 4368.558674] lmv_close+0x18d/0x490 [lmv] [ 4368.571416] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 4368.582755] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4368.599612] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4368.611346] ll_md_close+0x274/0x800 [lustre] [ 4368.618819] ll_file_release+0x128/0xc50 [lustre] [ 4368.632123] __fput+0x10d/0x440 [ 4368.640799] ____fput+0x12/0x20 [ 4368.648613] task_work_run+0xc6/0x110 [ 4368.651471] exit_to_usermode_loop+0x1dd/0x1f0 [ 4368.668621] do_syscall_64+0x3d6/0x440 [ 4368.680707] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4368.683529] RIP: 0033:0x7ff54af45cd8 [ 4368.687073] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4368.714415] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4368.731609] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 4368.742495] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 4368.752552] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 4368.765619] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 4368.779592] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4375.519115] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2416s! [ 4375.520794] Showing busy workqueues and worker pools: [ 4375.528622] workqueue events: flags=0x0 [ 4375.530514] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4375.534725] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 4375.543129] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4375.545612] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 4375.550251] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4375.553089] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 4375.555018] workqueue events_long: flags=0x0 [ 4375.556570] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4375.559023] pending: lu_objects_flush [obdclass] [ 4375.560905] workqueue events_unbound: flags=0x2 [ 4375.562492] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4375.565040] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4375.567727] workqueue events_power_efficient: flags=0x80 [ 4375.569202] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4375.571529] pending: neigh_periodic_work [ 4375.573205] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4375.575629] pending: do_cache_clean, check_lifetime [ 4375.577641] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4375.580057] pending: neigh_periodic_work [ 4375.581566] workqueue writeback: flags=0x4a [ 4375.583196] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4375.585485] pending: wb_workfn, wb_workfn [ 4375.587190] workqueue xprtiod: flags=0xa [ 4375.588844] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4375.598055] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 4375.600689] workqueue ipv6_addrconf: flags=0x40008 [ 4375.602419] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4375.606221] pending: addrconf_verify_work [ 4375.608814] workqueue knbd0-recv: flags=0x1a [ 4375.611703] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4375.613758] in-flight: 657:recv_work [ 4375.615303] workqueue ptlrpc_pinger: flags=0x2 [ 4375.616888] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4375.618217] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 4375.619854] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2416s workers=4 idle: 20831 12642 28859 [ 4375.622922] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4365s workers=2 idle: 659 [ 4380.059014] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 4380.060759] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4380.102257] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4380.114469] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4380.123295] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4380.128562] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4380.133117] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4380.151139] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4380.158334] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 4380.164244] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 4380.178253] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 4380.188278] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 4380.199697] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 4380.209612] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 4380.223761] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4380.231630] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 4380.234478] Call Trace: [ 4380.236507] [ 4380.247466] ? show_regs.cold.9+0x22/0x2f [ 4380.248834] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4380.250545] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4380.262537] ? watchdog+0x40/0x40 [ 4380.265644] ? hrtimer_interrupt+0x128/0x2c0 [ 4380.270777] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4380.275704] ? apic_timer_interrupt+0xf/0x20 [ 4380.281390] [ 4380.283376] ? native_safe_halt+0x2e/0x50 [ 4380.288639] kvm_wait+0xcd/0xe0 [ 4380.292486] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4380.296538] do_raw_spin_lock+0x177/0x190 [ 4380.303935] _raw_spin_lock+0x21/0x30 [ 4380.306691] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4380.313302] process_one_work+0x2c8/0x700 [ 4380.318673] worker_thread+0x296/0x6e0 [ 4380.325504] ? rescuer_thread+0x570/0x570 [ 4380.331761] kthread+0x1d1/0x200 [ 4380.335648] ? set_kthread_struct+0x70/0x70 [ 4380.342752] ret_from_fork+0x1f/0x30 [ 4384.060068] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 4384.063643] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4384.085246] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4384.091223] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4384.096206] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4384.098501] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4384.110242] RSP: 0000:ffffaf9fc15234f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4384.114273] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 4384.117974] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 4384.123255] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4384.127084] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 4384.130654] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 4384.133633] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 4384.137154] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4384.140190] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 4384.144067] Call Trace: [ 4384.145298] [ 4384.146298] ? show_regs.cold.9+0x22/0x2f [ 4384.148372] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4384.150867] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4384.153449] ? watchdog+0x40/0x40 [ 4384.155631] ? hrtimer_interrupt+0x128/0x2c0 [ 4384.157644] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4384.160132] ? apic_timer_interrupt+0xf/0x20 [ 4384.162232] [ 4384.163311] ? native_safe_halt+0x2e/0x50 [ 4384.165311] kvm_wait+0xcd/0xe0 [ 4384.166819] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4384.170126] do_raw_spin_lock+0x177/0x190 [ 4384.172090] _raw_spin_lock+0x21/0x30 [ 4384.173558] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4384.177304] ? do_raw_spin_unlock+0x75/0x190 [ 4384.179512] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4384.182843] ? do_wait_intr+0xf0/0xf0 [ 4384.185682] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4384.189140] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4384.197739] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4384.202130] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4384.206480] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4384.210124] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4384.214276] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4384.218107] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4384.221274] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4384.224690] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4384.227889] lmv_intent_open+0x338/0x1150 [lmv] [ 4384.230367] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4384.233420] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4384.236853] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4384.241285] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4384.244848] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4384.247142] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4384.249252] ? do_raw_spin_unlock+0x75/0x190 [ 4384.252150] ? _raw_spin_unlock+0x12/0x30 [ 4384.256341] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4384.262245] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4384.268635] ll_atomic_open+0x36a/0xd70 [lustre] [ 4384.272520] lookup_open+0x356/0xb40 [ 4384.277813] do_last+0x313/0xfc0 [ 4384.279820] ? nd_jump_root+0xe5/0x160 [ 4384.283114] ? path_init+0x437/0x520 [ 4384.285310] path_openat+0xf7/0x500 [ 4384.288659] ? do_raw_spin_unlock+0x75/0x190 [ 4384.291441] do_filp_open+0x99/0x140 [ 4384.293299] ? getname_flags+0x6e/0x330 [ 4384.295095] ? __check_object_size+0xff/0x256 [ 4384.297279] ? do_raw_spin_unlock+0x75/0x190 [ 4384.299440] ? _raw_spin_unlock+0x12/0x30 [ 4384.301152] do_sys_openat2+0x2b4/0x410 [ 4384.303153] do_sys_open+0x73/0xa0 [ 4384.305016] __x64_sys_openat+0x24/0x30 [ 4384.306852] do_syscall_64+0xc1/0x440 [ 4384.311355] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4384.316023] RIP: 0033:0x7ff54af45332 [ 4384.320013] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4384.345218] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4384.350958] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 4384.354637] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 4384.358610] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 4384.362835] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 4384.366846] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 4396.062062] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [ 4396.071614] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4396.123680] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4396.141412] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4396.154412] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4396.161505] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4396.187914] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 4396.201354] RAX: 0000000000001389 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 4396.216747] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 4396.222951] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4396.230602] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 4396.235230] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 4396.242720] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 4396.250413] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4396.254751] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 4396.260000] Call Trace: [ 4396.262420] [ 4396.263606] ? show_regs.cold.9+0x22/0x2f [ 4396.265864] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4396.268943] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4396.272281] ? watchdog+0x40/0x40 [ 4396.274439] ? hrtimer_interrupt+0x128/0x2c0 [ 4396.276831] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4396.279116] ? apic_timer_interrupt+0xf/0x20 [ 4396.282290] [ 4396.283533] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4396.286584] do_raw_spin_lock+0x177/0x190 [ 4396.288744] _raw_spin_lock+0x21/0x30 [ 4396.290924] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4396.294368] ? do_raw_spin_unlock+0x75/0x190 [ 4396.296848] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4396.299980] ? do_wait_intr+0xf0/0xf0 [ 4396.302148] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4396.304649] mdc_close+0x53b/0xd70 [mdc] [ 4396.307181] lmv_close+0x18d/0x490 [lmv] [ 4396.309408] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 4396.313756] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4396.317819] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4396.321465] ll_md_close+0x274/0x800 [lustre] [ 4396.324412] ll_file_release+0x128/0xc50 [lustre] [ 4396.327520] __fput+0x10d/0x440 [ 4396.329565] ____fput+0x12/0x20 [ 4396.331444] task_work_run+0xc6/0x110 [ 4396.333729] exit_to_usermode_loop+0x1dd/0x1f0 [ 4396.335868] do_syscall_64+0x3d6/0x440 [ 4396.337490] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4396.340422] RIP: 0033:0x7ff54af45cd8 [ 4396.343365] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4396.355796] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4396.359555] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 4396.362710] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 4396.367206] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 4396.372211] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 4396.376953] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4400.055976] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 4400.059827] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4400.092614] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4400.101398] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4400.107502] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4400.110997] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4400.133954] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4400.143765] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 4400.150535] RDX: 0000000000007e86 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 4400.154546] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 4400.157044] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 4400.164431] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4400.176473] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 4400.193337] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4400.200283] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 4400.206993] Call Trace: [ 4400.210378] [ 4400.218035] ? show_regs.cold.9+0x22/0x2f [ 4400.223342] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4400.225498] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4400.227959] ? watchdog+0x40/0x40 [ 4400.229573] ? hrtimer_interrupt+0x128/0x2c0 [ 4400.231233] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4400.234120] ? apic_timer_interrupt+0xf/0x20 [ 4400.238305] [ 4400.239234] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4400.241354] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4400.243798] do_raw_spin_lock+0x177/0x190 [ 4400.245018] _raw_spin_lock+0x21/0x30 [ 4400.250914] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4400.253474] obd_mod_free+0x46/0x140 [ptlrpc] [ 4400.255710] mdc_commit_open+0x91/0xb0 [mdc] [ 4400.259313] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4400.262169] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4400.264241] after_reply+0xe2c/0x1780 [ptlrpc] [ 4400.267938] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4400.275811] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4400.277562] ? do_wait_intr+0xf0/0xf0 [ 4400.283721] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4400.289668] kthread+0x1d1/0x200 [ 4400.291525] ? set_kthread_struct+0x70/0x70 [ 4400.293835] ret_from_fork+0x1f/0x30 [ 4405.728571] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2446s! [ 4405.733725] Showing busy workqueues and worker pools: [ 4405.738806] workqueue events: flags=0x0 [ 4405.748190] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4405.756163] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 4405.767735] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4405.774367] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 4405.784989] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4405.791625] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 4405.795963] workqueue events_long: flags=0x0 [ 4405.799774] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4405.804452] pending: lu_objects_flush [obdclass] [ 4405.810046] workqueue events_unbound: flags=0x2 [ 4405.816213] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4405.818647] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4405.827637] workqueue events_power_efficient: flags=0x80 [ 4405.830686] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4405.836420] pending: neigh_periodic_work [ 4405.840153] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4405.844727] pending: do_cache_clean, check_lifetime [ 4405.849461] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4405.853363] pending: neigh_periodic_work [ 4405.855728] workqueue writeback: flags=0x4a [ 4405.863607] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4405.868011] pending: wb_workfn, wb_workfn [ 4405.870163] workqueue xprtiod: flags=0xa [ 4405.872112] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4405.875204] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 4405.879420] workqueue ipv6_addrconf: flags=0x40008 [ 4405.881845] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4405.884673] pending: addrconf_verify_work [ 4405.892508] workqueue knbd0-recv: flags=0x1a [ 4405.895146] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4405.902683] in-flight: 657:recv_work [ 4405.905752] workqueue ptlrpc_pinger: flags=0x2 [ 4405.909032] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4405.913202] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 4405.917381] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2447s workers=4 idle: 20831 12642 28859 [ 4405.927677] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4396s workers=2 idle: 659 [ 4408.060455] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 4408.069472] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4408.121506] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4408.148814] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4408.162982] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4408.172369] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4408.182060] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4408.204259] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4408.215924] RAX: 0000000000007643 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 4408.225688] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 4408.229858] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 4408.236153] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 4408.243138] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 4408.252738] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 4408.264464] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4408.272196] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 4408.277923] Call Trace: [ 4408.282465] [ 4408.283254] ? show_regs.cold.9+0x22/0x2f [ 4408.287268] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4408.292972] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4408.297748] ? watchdog+0x40/0x40 [ 4408.305137] ? hrtimer_interrupt+0x128/0x2c0 [ 4408.312870] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4408.323352] ? apic_timer_interrupt+0xf/0x20 [ 4408.332170] [ 4408.338515] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4408.347349] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4408.363355] do_raw_spin_lock+0x177/0x190 [ 4408.369942] _raw_spin_lock+0x21/0x30 [ 4408.375215] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4408.386458] process_one_work+0x2c8/0x700 [ 4408.390773] worker_thread+0x296/0x6e0 [ 4408.396049] ? rescuer_thread+0x570/0x570 [ 4408.399479] kthread+0x1d1/0x200 [ 4408.403781] ? set_kthread_struct+0x70/0x70 [ 4408.414999] ret_from_fork+0x1f/0x30 [ 4412.062195] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [ 4412.065995] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4412.111886] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4412.123959] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4412.129693] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4412.139138] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4412.148428] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 4412.154711] RAX: 0000000000007f97 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 4412.158347] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 4412.161061] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4412.164808] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 4412.167946] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 4412.171197] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 4412.175020] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4412.177266] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 4412.183695] Call Trace: [ 4412.188959] [ 4412.190546] ? show_regs.cold.9+0x22/0x2f [ 4412.192023] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4412.195968] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4412.202135] ? watchdog+0x40/0x40 [ 4412.203617] ? hrtimer_interrupt+0x128/0x2c0 [ 4412.207133] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4412.214869] ? apic_timer_interrupt+0xf/0x20 [ 4412.216358] [ 4412.218422] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4412.220354] do_raw_spin_lock+0x177/0x190 [ 4412.223306] _raw_spin_lock+0x21/0x30 [ 4412.231723] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4412.242110] ? do_raw_spin_unlock+0x75/0x190 [ 4412.244488] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4412.247983] ? do_wait_intr+0xf0/0xf0 [ 4412.249583] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4412.252073] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4412.254946] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4412.260184] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4412.262968] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4412.265055] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4412.269036] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4412.272054] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4412.276576] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4412.281964] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4412.287339] lmv_intent_open+0x338/0x1150 [lmv] [ 4412.290151] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4412.292075] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4412.297534] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4412.299052] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4412.302787] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4412.310523] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4412.312710] ? do_raw_spin_unlock+0x75/0x190 [ 4412.314680] ? _raw_spin_unlock+0x12/0x30 [ 4412.316372] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4412.319248] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4412.325191] ll_atomic_open+0x36a/0xd70 [lustre] [ 4412.328308] lookup_open+0x356/0xb40 [ 4412.331158] do_last+0x313/0xfc0 [ 4412.332720] ? nd_jump_root+0xe5/0x160 [ 4412.334056] ? path_init+0x437/0x520 [ 4412.335247] path_openat+0xf7/0x500 [ 4412.337755] ? do_raw_spin_unlock+0x75/0x190 [ 4412.340592] do_filp_open+0x99/0x140 [ 4412.343233] ? getname_flags+0x6e/0x330 [ 4412.344581] ? __check_object_size+0xff/0x256 [ 4412.353318] ? do_raw_spin_unlock+0x75/0x190 [ 4412.354601] ? _raw_spin_unlock+0x12/0x30 [ 4412.366660] do_sys_openat2+0x2b4/0x410 [ 4412.368223] do_sys_open+0x73/0xa0 [ 4412.373232] __x64_sys_openat+0x24/0x30 [ 4412.374526] do_syscall_64+0xc1/0x440 [ 4412.378946] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4412.385825] RIP: 0033:0x7ff54af45332 [ 4412.389056] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4412.399005] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4412.401916] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 4412.412333] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 4412.419204] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 4412.433661] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 4412.442017] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 4424.062106] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [ 4424.068511] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4424.107784] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4424.117178] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4424.124497] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4424.130791] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4424.145786] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4424.149004] RAX: 0000000000004784 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 4424.156616] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 4424.164708] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4424.170709] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 4424.176333] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 4424.186868] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 4424.190373] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4424.196766] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 4424.203143] Call Trace: [ 4424.204022] [ 4424.204812] ? show_regs.cold.9+0x22/0x2f [ 4424.216103] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4424.218133] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4424.229198] ? watchdog+0x40/0x40 [ 4424.234255] ? hrtimer_interrupt+0x128/0x2c0 [ 4424.239179] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4424.246697] ? apic_timer_interrupt+0xf/0x20 [ 4424.254111] [ 4424.258954] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4424.270801] do_raw_spin_lock+0x177/0x190 [ 4424.281620] _raw_spin_lock+0x21/0x30 [ 4424.293257] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4424.299659] ? do_raw_spin_unlock+0x75/0x190 [ 4424.303977] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4424.306686] ? do_wait_intr+0xf0/0xf0 [ 4424.307505] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4424.317546] mdc_close+0x53b/0xd70 [mdc] [ 4424.323169] lmv_close+0x18d/0x490 [lmv] [ 4424.325454] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 4424.329622] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4424.335092] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4424.337285] ll_md_close+0x274/0x800 [lustre] [ 4424.339459] ll_file_release+0x128/0xc50 [lustre] [ 4424.344182] __fput+0x10d/0x440 [ 4424.348288] ____fput+0x12/0x20 [ 4424.350451] task_work_run+0xc6/0x110 [ 4424.356703] exit_to_usermode_loop+0x1dd/0x1f0 [ 4424.361962] do_syscall_64+0x3d6/0x440 [ 4424.364652] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4424.366630] RIP: 0033:0x7ff54af45cd8 [ 4424.369048] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4424.379546] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4424.381631] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 4424.386000] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 4424.388960] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 4424.394110] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 4424.397490] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4432.055141] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 4432.068117] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4432.105183] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4432.112230] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4432.118207] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4432.123581] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4432.140381] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4432.146666] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 4432.152978] RDX: 0000000000007cff RSI: 0000000000000003 RDI: ffff96848811cae0 [ 4432.160160] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 4432.166215] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 4432.170082] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4432.177720] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 4432.182137] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4432.185043] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 4432.188977] Call Trace: [ 4432.192007] [ 4432.193691] ? show_regs.cold.9+0x22/0x2f [ 4432.196476] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4432.203838] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4432.205812] ? watchdog+0x40/0x40 [ 4432.215125] ? hrtimer_interrupt+0x128/0x2c0 [ 4432.224128] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4432.229487] ? apic_timer_interrupt+0xf/0x20 [ 4432.237872] [ 4432.238726] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4432.243690] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4432.247782] do_raw_spin_lock+0x177/0x190 [ 4432.252200] _raw_spin_lock+0x21/0x30 [ 4432.256160] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4432.258910] obd_mod_free+0x46/0x140 [ptlrpc] [ 4432.262054] mdc_commit_open+0x91/0xb0 [mdc] [ 4432.265881] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4432.272187] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4432.279797] after_reply+0xe2c/0x1780 [ptlrpc] [ 4432.284858] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4432.287284] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4432.301166] ? do_wait_intr+0xf0/0xf0 [ 4432.304710] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4432.307881] kthread+0x1d1/0x200 [ 4432.317471] ? set_kthread_struct+0x70/0x70 [ 4432.321883] ret_from_fork+0x1f/0x30 [ 4435.935189] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2477s! [ 4435.939573] Showing busy workqueues and worker pools: [ 4435.952118] workqueue events: flags=0x0 [ 4435.954148] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4435.967853] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 4435.978981] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4435.984198] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 4435.994182] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4435.998489] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 4436.002563] workqueue events_long: flags=0x0 [ 4436.004648] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4436.010093] pending: lu_objects_flush [obdclass] [ 4436.017505] workqueue events_unbound: flags=0x2 [ 4436.028771] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4436.030598] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4436.039889] workqueue events_power_efficient: flags=0x80 [ 4436.049035] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4436.056564] pending: neigh_periodic_work [ 4436.059021] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 4436.065039] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4436.065062] pending: do_cache_clean, check_lifetime [ 4436.065080] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4436.065088] pending: neigh_periodic_work [ 4436.065103] workqueue writeback: flags=0x4a [ 4436.065105] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4436.065113] pending: wb_workfn, wb_workfn [ 4436.065134] workqueue xprtiod: flags=0xa [ 4436.065136] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4436.065142] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 4436.065162] workqueue ipv6_addrconf: flags=0x40008 [ 4436.065165] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4436.065173] pending: addrconf_verify_work [ 4436.065186] workqueue knbd0-recv: flags=0x1a [ 4436.065188] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4436.065196] in-flight: 657:recv_work [ 4436.065207] workqueue ptlrpc_pinger: flags=0x2 [ 4436.065209] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4436.065214] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 4436.074569] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4436.075199] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4436.075206] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4436.075212] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4436.078885] [ 4436.078933] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2477s workers=4 idle: 20831 12642 28859 [ 4436.078955] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4426s workers=2 idle: 659 [ 4436.315996] [ 4436.317821] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4436.326562] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4436.341428] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4436.356855] RAX: 00000000000068c5 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 4436.361804] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 4436.369158] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 4436.374611] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 4436.380624] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 4436.385697] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 4436.392370] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4436.398507] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 4436.404581] Call Trace: [ 4436.405893] [ 4436.407046] ? show_regs.cold.9+0x22/0x2f [ 4436.409185] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4436.412354] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4436.415072] ? watchdog+0x40/0x40 [ 4436.420754] ? hrtimer_interrupt+0x128/0x2c0 [ 4436.423529] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4436.425134] ? apic_timer_interrupt+0xf/0x20 [ 4436.439221] [ 4436.443137] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4436.456788] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4436.466170] do_raw_spin_lock+0x177/0x190 [ 4436.477751] _raw_spin_lock+0x21/0x30 [ 4436.488527] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4436.495284] process_one_work+0x2c8/0x700 [ 4436.500888] worker_thread+0x296/0x6e0 [ 4436.512967] ? rescuer_thread+0x570/0x570 [ 4436.516673] kthread+0x1d1/0x200 [ 4436.521181] ? set_kthread_struct+0x70/0x70 [ 4436.528077] ret_from_fork+0x1f/0x30 [ 4440.060056] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [ 4440.065094] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4440.098180] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4440.105181] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4440.112106] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4440.117422] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4440.125049] RSP: 0000:ffffaf9fc15234f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4440.128308] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 4440.132133] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 4440.134073] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4440.142410] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 4440.148296] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 4440.154550] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 4440.158114] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4440.160262] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 4440.165491] Call Trace: [ 4440.166509] [ 4440.167258] ? show_regs.cold.9+0x22/0x2f [ 4440.172135] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4440.174081] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4440.176039] ? watchdog+0x40/0x40 [ 4440.177633] ? hrtimer_interrupt+0x128/0x2c0 [ 4440.179150] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4440.181074] ? apic_timer_interrupt+0xf/0x20 [ 4440.182756] [ 4440.183531] ? native_safe_halt+0x2e/0x50 [ 4440.184948] kvm_wait+0xcd/0xe0 [ 4440.186153] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4440.188540] do_raw_spin_lock+0x177/0x190 [ 4440.190262] _raw_spin_lock+0x21/0x30 [ 4440.191529] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4440.198906] ? do_raw_spin_unlock+0x75/0x190 [ 4440.202709] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4440.207981] ? do_wait_intr+0xf0/0xf0 [ 4440.212090] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4440.214570] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4440.217952] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4440.221607] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4440.225782] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4440.231965] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4440.238306] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4440.241511] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4440.244680] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4440.248945] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4440.252737] lmv_intent_open+0x338/0x1150 [lmv] [ 4440.255592] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4440.260059] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4440.263904] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4440.266692] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4440.269783] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4440.276056] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4440.280986] ? do_raw_spin_unlock+0x75/0x190 [ 4440.284468] ? _raw_spin_unlock+0x12/0x30 [ 4440.288709] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4440.291935] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4440.295075] ll_atomic_open+0x36a/0xd70 [lustre] [ 4440.298170] lookup_open+0x356/0xb40 [ 4440.300110] do_last+0x313/0xfc0 [ 4440.302111] ? nd_jump_root+0xe5/0x160 [ 4440.304339] ? path_init+0x437/0x520 [ 4440.307574] path_openat+0xf7/0x500 [ 4440.310348] ? do_raw_spin_unlock+0x75/0x190 [ 4440.314180] do_filp_open+0x99/0x140 [ 4440.316032] ? getname_flags+0x6e/0x330 [ 4440.320448] ? __check_object_size+0xff/0x256 [ 4440.325371] ? do_raw_spin_unlock+0x75/0x190 [ 4440.327277] ? _raw_spin_unlock+0x12/0x30 [ 4440.328282] do_sys_openat2+0x2b4/0x410 [ 4440.329191] do_sys_open+0x73/0xa0 [ 4440.337995] __x64_sys_openat+0x24/0x30 [ 4440.340926] do_syscall_64+0xc1/0x440 [ 4440.341837] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4440.349563] RIP: 0033:0x7ff54af45332 [ 4440.352493] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4440.365574] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4440.369981] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 4440.374504] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 4440.379035] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 4440.380954] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 4440.384518] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 4452.064912] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [ 4452.074719] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4452.129525] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4452.147513] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4452.159779] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4452.167938] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4452.183785] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4452.193411] RAX: 0000000000001412 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 4452.200620] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 4452.212885] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4452.221001] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 4452.231457] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 4452.241611] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 4452.247897] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4452.251958] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 4452.263336] Call Trace: [ 4452.264731] [ 4452.265935] ? show_regs.cold.9+0x22/0x2f [ 4452.272018] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4452.276149] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4452.283502] ? watchdog+0x40/0x40 [ 4452.288400] ? hrtimer_interrupt+0x128/0x2c0 [ 4452.294657] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4452.297561] ? apic_timer_interrupt+0xf/0x20 [ 4452.301547] [ 4452.304527] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4452.308484] do_raw_spin_lock+0x177/0x190 [ 4452.310153] _raw_spin_lock+0x21/0x30 [ 4452.311772] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4452.314926] ? do_raw_spin_unlock+0x75/0x190 [ 4452.316736] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4452.319707] ? do_wait_intr+0xf0/0xf0 [ 4452.323710] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4452.329523] mdc_close+0x53b/0xd70 [mdc] [ 4452.332207] lmv_close+0x18d/0x490 [lmv] [ 4452.333958] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 4452.337757] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4452.341784] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4452.348156] ll_md_close+0x274/0x800 [lustre] [ 4452.351825] ll_file_release+0x128/0xc50 [lustre] [ 4452.363962] __fput+0x10d/0x440 [ 4452.366684] ____fput+0x12/0x20 [ 4452.368132] task_work_run+0xc6/0x110 [ 4452.369950] exit_to_usermode_loop+0x1dd/0x1f0 [ 4452.377061] do_syscall_64+0x3d6/0x440 [ 4452.381842] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4452.385843] RIP: 0033:0x7ff54af45cd8 [ 4452.389475] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4452.404051] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4452.415598] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 4452.420446] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 4452.423853] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 4452.427844] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 4452.431477] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4464.057867] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 4464.059126] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 4464.062154] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4464.062230] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4464.062237] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4464.062239] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4464.062254] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4464.062258] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4464.062262] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 4464.062264] RDX: 00000000000064a0 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 4464.062267] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 4464.062269] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 4464.062271] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4464.062274] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 4464.062276] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4464.062278] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 4464.062286] Call Trace: [ 4464.062292] [ 4464.062298] ? show_regs.cold.9+0x22/0x2f [ 4464.062310] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4464.062316] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4464.062320] ? watchdog+0x40/0x40 [ 4464.062324] ? hrtimer_interrupt+0x128/0x2c0 [ 4464.062327] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4464.062336] ? apic_timer_interrupt+0xf/0x20 [ 4464.062341] [ 4464.062344] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4464.062349] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4464.062354] do_raw_spin_lock+0x177/0x190 [ 4464.062359] _raw_spin_lock+0x21/0x30 [ 4464.062365] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4464.062673] obd_mod_free+0x46/0x140 [ptlrpc] [ 4464.072941] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4464.073020] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4464.073027] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4464.073035] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4464.073354] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4464.073366] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4464.073371] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4464.073375] RAX: 0000000000004d49 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 4464.073377] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 4464.073379] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 4464.073381] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 4464.073383] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 4464.073386] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 4464.073388] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4464.073390] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 4464.073399] Call Trace: [ 4464.073405] [ 4464.073411] ? show_regs.cold.9+0x22/0x2f [ 4464.073420] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4464.073426] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4464.073430] ? watchdog+0x40/0x40 [ 4464.073434] ? hrtimer_interrupt+0x128/0x2c0 [ 4464.073438] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4464.073446] ? apic_timer_interrupt+0xf/0x20 [ 4464.073451] [ 4464.073454] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4464.073458] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4464.073463] do_raw_spin_lock+0x177/0x190 [ 4464.073467] _raw_spin_lock+0x21/0x30 [ 4464.073472] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4464.073676] process_one_work+0x2c8/0x700 [ 4464.073683] worker_thread+0x296/0x6e0 [ 4464.073687] ? rescuer_thread+0x570/0x570 [ 4464.073691] kthread+0x1d1/0x200 [ 4464.073695] ? set_kthread_struct+0x70/0x70 [ 4464.073699] ret_from_fork+0x1f/0x30 [ 4464.465845] mdc_commit_open+0x91/0xb0 [mdc] [ 4464.475513] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4464.480921] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4464.490262] after_reply+0xe2c/0x1780 [ptlrpc] [ 4464.492146] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4464.497120] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4464.500625] ? do_wait_intr+0xf0/0xf0 [ 4464.502937] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4464.510681] kthread+0x1d1/0x200 [ 4464.514666] ? set_kthread_struct+0x70/0x70 [ 4464.518552] ret_from_fork+0x1f/0x30 [ 4466.143031] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2507s! [ 4466.147386] Showing busy workqueues and worker pools: [ 4466.150406] workqueue events: flags=0x0 [ 4466.152447] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4466.155702] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 4466.164024] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4466.170912] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 4466.176214] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4466.178723] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 4466.181853] workqueue events_long: flags=0x0 [ 4466.183540] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4466.187661] pending: lu_objects_flush [obdclass] [ 4466.190735] workqueue events_unbound: flags=0x2 [ 4466.192876] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4466.195846] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4466.205411] workqueue events_power_efficient: flags=0x80 [ 4466.210269] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4466.215450] pending: neigh_periodic_work [ 4466.218228] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4466.225702] pending: do_cache_clean, check_lifetime [ 4466.233454] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4466.237643] pending: neigh_periodic_work [ 4466.239793] workqueue writeback: flags=0x4a [ 4466.242308] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4466.245503] pending: wb_workfn, wb_workfn [ 4466.247713] workqueue xprtiod: flags=0xa [ 4466.249900] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4466.256242] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 4466.259883] workqueue ipv6_addrconf: flags=0x40008 [ 4466.263088] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4466.266650] pending: addrconf_verify_work [ 4466.270397] workqueue knbd0-recv: flags=0x1a [ 4466.274372] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4466.278728] in-flight: 657:recv_work [ 4466.283519] workqueue ptlrpc_pinger: flags=0x2 [ 4466.286623] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4466.290494] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 4466.294130] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2507s workers=4 idle: 20831 12642 28859 [ 4466.299358] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4456s workers=2 idle: 659 [ 4468.060031] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [ 4468.061913] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4468.079467] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4468.082711] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4468.086212] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4468.088202] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4468.095695] RSP: 0000:ffffaf9fc15234f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4468.098459] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 4468.100083] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 4468.102087] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4468.103760] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 4468.105972] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 4468.109530] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 4468.112801] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4468.115794] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 4468.118299] Call Trace: [ 4468.119758] [ 4468.120559] ? show_regs.cold.9+0x22/0x2f [ 4468.121785] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4468.124124] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4468.129018] ? watchdog+0x40/0x40 [ 4468.130044] ? hrtimer_interrupt+0x128/0x2c0 [ 4468.131158] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4468.132973] ? apic_timer_interrupt+0xf/0x20 [ 4468.134834] [ 4468.135723] ? native_safe_halt+0x2e/0x50 [ 4468.138295] kvm_wait+0xcd/0xe0 [ 4468.139742] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4468.142073] do_raw_spin_lock+0x177/0x190 [ 4468.143977] _raw_spin_lock+0x21/0x30 [ 4468.145652] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4468.147863] ? do_raw_spin_unlock+0x75/0x190 [ 4468.149613] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4468.151591] ? do_wait_intr+0xf0/0xf0 [ 4468.153214] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4468.155797] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4468.158114] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4468.160452] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4468.162651] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4468.165456] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4468.167247] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4468.169140] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4468.171237] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4468.174219] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4468.176682] lmv_intent_open+0x338/0x1150 [lmv] [ 4468.178331] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4468.180372] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4468.182530] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4468.184605] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4468.187502] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4468.190186] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4468.192239] ? do_raw_spin_unlock+0x75/0x190 [ 4468.194456] ? _raw_spin_unlock+0x12/0x30 [ 4468.195842] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4468.198543] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4468.200683] ll_atomic_open+0x36a/0xd70 [lustre] [ 4468.202853] lookup_open+0x356/0xb40 [ 4468.204627] do_last+0x313/0xfc0 [ 4468.206043] ? nd_jump_root+0xe5/0x160 [ 4468.207558] ? path_init+0x437/0x520 [ 4468.208800] path_openat+0xf7/0x500 [ 4468.210443] ? do_raw_spin_unlock+0x75/0x190 [ 4468.212443] do_filp_open+0x99/0x140 [ 4468.213850] ? getname_flags+0x6e/0x330 [ 4468.215572] ? __check_object_size+0xff/0x256 [ 4468.217264] ? do_raw_spin_unlock+0x75/0x190 [ 4468.219079] ? _raw_spin_unlock+0x12/0x30 [ 4468.220632] do_sys_openat2+0x2b4/0x410 [ 4468.222182] do_sys_open+0x73/0xa0 [ 4468.223845] __x64_sys_openat+0x24/0x30 [ 4468.225486] do_syscall_64+0xc1/0x440 [ 4468.226944] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4468.229432] RIP: 0033:0x7ff54af45332 [ 4468.230779] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4468.238251] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4468.241392] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 4468.244394] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 4468.246606] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 4468.249181] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 4468.251862] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 4480.060215] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 4480.063354] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4480.103684] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4480.112393] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4480.121087] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4480.127899] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4480.153960] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4480.159597] RAX: 0000000000006998 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 4480.169956] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 4480.177381] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4480.182464] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 4480.194883] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 4480.200590] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 4480.213496] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4480.220638] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 4480.226946] Call Trace: [ 4480.233301] [ 4480.234134] ? show_regs.cold.9+0x22/0x2f [ 4480.235577] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4480.240171] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4480.243397] ? watchdog+0x40/0x40 [ 4480.245114] ? hrtimer_interrupt+0x128/0x2c0 [ 4480.247990] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4480.254406] ? apic_timer_interrupt+0xf/0x20 [ 4480.257703] [ 4480.258738] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4480.263397] do_raw_spin_lock+0x177/0x190 [ 4480.264962] _raw_spin_lock+0x21/0x30 [ 4480.267281] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4480.271621] ? do_raw_spin_unlock+0x75/0x190 [ 4480.275151] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4480.280668] ? do_wait_intr+0xf0/0xf0 [ 4480.283515] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4480.285569] mdc_close+0x53b/0xd70 [mdc] [ 4480.294931] lmv_close+0x18d/0x490 [lmv] [ 4480.298789] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 4480.304417] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4480.307789] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4480.311121] ll_md_close+0x274/0x800 [lustre] [ 4480.313897] ll_file_release+0x128/0xc50 [lustre] [ 4480.316758] __fput+0x10d/0x440 [ 4480.318868] ____fput+0x12/0x20 [ 4480.321710] task_work_run+0xc6/0x110 [ 4480.332415] exit_to_usermode_loop+0x1dd/0x1f0 [ 4480.333990] do_syscall_64+0x3d6/0x440 [ 4480.340543] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4480.342485] RIP: 0033:0x7ff54af45cd8 [ 4480.352340] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4480.378101] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4480.386288] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 4480.392696] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 4480.398166] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 4480.403952] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 4480.414162] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4492.055074] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 4492.059271] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 4492.062232] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4492.062313] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4492.062321] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4492.062324] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4492.062339] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4492.062343] RSP: 0000:ffffaf9fc1b97bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4492.062347] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 4492.062350] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 4492.062352] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 4492.062354] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 4492.063025] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4492.063028] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 4492.063031] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4492.063033] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 4492.063042] Call Trace: [ 4492.063048] [ 4492.063053] ? show_regs.cold.9+0x22/0x2f [ 4492.063066] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4492.063074] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4492.063078] ? watchdog+0x40/0x40 [ 4492.063082] ? hrtimer_interrupt+0x128/0x2c0 [ 4492.063086] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4492.063092] ? apic_timer_interrupt+0xf/0x20 [ 4492.063097] [ 4492.063100] ? native_safe_halt+0x2e/0x50 [ 4492.063104] kvm_wait+0xcd/0xe0 [ 4492.063112] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4492.063120] do_raw_spin_lock+0x177/0x190 [ 4492.063124] _raw_spin_lock+0x21/0x30 [ 4492.063130] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4492.064124] obd_mod_free+0x46/0x140 [ptlrpc] [ 4492.070354] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4492.070430] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4492.070437] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4492.070442] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4492.071071] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4492.071084] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4492.071089] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4492.071094] RAX: 0000000000006f92 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 4492.071096] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 4492.071098] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 4492.071100] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 4492.071102] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 4492.071105] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 4492.071107] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4492.071110] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 4492.071120] Call Trace: [ 4492.071126] [ 4492.071134] ? show_regs.cold.9+0x22/0x2f [ 4492.071233] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4492.071239] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4492.071243] ? watchdog+0x40/0x40 [ 4492.071247] ? hrtimer_interrupt+0x128/0x2c0 [ 4492.071252] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4492.071262] ? apic_timer_interrupt+0xf/0x20 [ 4492.071266] [ 4492.071269] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4492.071273] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4492.071279] do_raw_spin_lock+0x177/0x190 [ 4492.071284] _raw_spin_lock+0x21/0x30 [ 4492.071288] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4492.071513] process_one_work+0x2c8/0x700 [ 4492.071521] worker_thread+0x296/0x6e0 [ 4492.071525] ? rescuer_thread+0x570/0x570 [ 4492.071529] kthread+0x1d1/0x200 [ 4492.071534] ? set_kthread_struct+0x70/0x70 [ 4492.071538] ret_from_fork+0x1f/0x30 [ 4492.641205] mdc_commit_open+0x91/0xb0 [mdc] [ 4492.647272] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4492.656961] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4492.664712] after_reply+0xe2c/0x1780 [ptlrpc] [ 4492.668000] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4492.673474] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4492.676510] ? do_wait_intr+0xf0/0xf0 [ 4492.678567] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4492.692467] kthread+0x1d1/0x200 [ 4492.693676] ? set_kthread_struct+0x70/0x70 [ 4492.715567] ret_from_fork+0x1f/0x30 [ 4496.060029] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [ 4496.062868] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4496.084793] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4496.097189] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4496.104534] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4496.108005] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4496.116979] RSP: 0000:ffffaf9fc15234f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4496.120554] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 4496.124015] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 4496.127151] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4496.132728] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 4496.136073] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 4496.142531] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 4496.147652] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4496.157575] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 4496.163528] Call Trace: [ 4496.165777] [ 4496.167871] ? show_regs.cold.9+0x22/0x2f [ 4496.170891] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4496.174090] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4496.177361] ? watchdog+0x40/0x40 [ 4496.179705] ? hrtimer_interrupt+0x128/0x2c0 [ 4496.182243] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4496.184158] ? apic_timer_interrupt+0xf/0x20 [ 4496.186255] [ 4496.187534] ? native_safe_halt+0x2e/0x50 [ 4496.190487] kvm_wait+0xcd/0xe0 [ 4496.192243] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4496.195466] do_raw_spin_lock+0x177/0x190 [ 4496.197882] _raw_spin_lock+0x21/0x30 [ 4496.199969] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4496.203892] ? do_raw_spin_unlock+0x75/0x190 [ 4496.206853] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4496.210351] ? do_wait_intr+0xf0/0xf0 [ 4496.212559] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4496.215326] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4496.218640] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4496.221883] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4496.223886] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4496.226549] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4496.229237] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4496.231890] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4496.234867] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4496.238388] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4496.241502] lmv_intent_open+0x338/0x1150 [lmv] [ 4496.243740] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4496.247850] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4496.250903] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4496.253585] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4496.256203] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4496.258968] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4496.261783] ? do_raw_spin_unlock+0x75/0x190 [ 4496.264934] ? _raw_spin_unlock+0x12/0x30 [ 4496.271279] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4496.278219] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4496.286466] ll_atomic_open+0x36a/0xd70 [lustre] [ 4496.294217] lookup_open+0x356/0xb40 [ 4496.297573] do_last+0x313/0xfc0 [ 4496.304217] ? nd_jump_root+0xe5/0x160 [ 4496.307178] ? path_init+0x437/0x520 [ 4496.311857] path_openat+0xf7/0x500 [ 4496.313792] ? do_raw_spin_unlock+0x75/0x190 [ 4496.319759] do_filp_open+0x99/0x140 [ 4496.321980] ? getname_flags+0x6e/0x330 [ 4496.327964] ? __check_object_size+0xff/0x256 [ 4496.332911] ? do_raw_spin_unlock+0x75/0x190 [ 4496.339988] ? _raw_spin_unlock+0x12/0x30 [ 4496.343907] do_sys_openat2+0x2b4/0x410 [ 4496.349303] do_sys_open+0x73/0xa0 [ 4496.351040] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2537s! [ 4496.351622] __x64_sys_openat+0x24/0x30 [ 4496.363992] Showing busy workqueues and worker pools: [ 4496.370177] do_syscall_64+0xc1/0x440 [ 4496.371852] workqueue events: flags=0x0 [ 4496.375558] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4496.376882] pwq 6: [ 4496.381995] RIP: 0033:0x7ff54af45332 [ 4496.381993] cpus=3 [ 4496.385311] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4496.386508] node=0 [ 4496.389142] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 [ 4496.395326] flags=0x0 nice=0 active=2/256 refcnt=3 [ 4496.395339] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 4496.395781] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4496.395790] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 4496.395943] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4496.395949] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 4496.396312] workqueue events_long: flags=0x0 [ 4496.396314] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4496.396319] pending: lu_objects_flush [obdclass] [ 4496.396452] workqueue events_unbound: flags=0x2 [ 4496.396454] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4496.396458] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4496.396474] workqueue events_power_efficient: flags=0x80 [ 4496.396475] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4496.396479] pending: neigh_periodic_work [ 4496.396484] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4496.396488] pending: do_cache_clean, check_lifetime [ 4496.396498] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4496.396502] pending: neigh_periodic_work [ 4496.396509] workqueue writeback: flags=0x4a [ 4496.396510] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4496.396514] pending: wb_workfn, wb_workfn [ 4496.396523] workqueue xprtiod: flags=0xa [ 4496.396526] pwq 8: [ 4496.397613] ORIG_RAX: 0000000000000101 [ 4496.397619] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 4496.397621] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 4496.397624] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 4496.397626] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 4496.397628] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 4496.450838] cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4496.452369] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 4496.454591] workqueue ipv6_addrconf: flags=0x40008 [ 4496.456315] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4496.458195] pending: addrconf_verify_work [ 4496.459379] workqueue knbd0-recv: flags=0x1a [ 4496.460532] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4496.462349] in-flight: 657:recv_work [ 4496.463468] workqueue ptlrpc_pinger: flags=0x2 [ 4496.464699] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4496.466350] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 4496.468106] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2537s workers=4 idle: 20831 12642 28859 [ 4496.470346] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4486s workers=2 idle: 659 [ 4508.062184] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 4508.078944] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4508.167151] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4508.184807] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4508.206824] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4508.217458] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4508.248937] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4508.262540] RAX: 0000000000003486 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 4508.273572] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 4508.284742] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4508.297194] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 4508.311586] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 4508.324192] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 4508.340956] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4508.353325] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 4508.365596] Call Trace: [ 4508.376694] [ 4508.382227] ? show_regs.cold.9+0x22/0x2f [ 4508.383663] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4508.396806] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4508.398343] ? watchdog+0x40/0x40 [ 4508.406972] ? hrtimer_interrupt+0x128/0x2c0 [ 4508.413726] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4508.422769] ? apic_timer_interrupt+0xf/0x20 [ 4508.427962] [ 4508.429251] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4508.432738] do_raw_spin_lock+0x177/0x190 [ 4508.437819] _raw_spin_lock+0x21/0x30 [ 4508.439650] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4508.452493] ? do_raw_spin_unlock+0x75/0x190 [ 4508.459818] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4508.470979] ? do_wait_intr+0xf0/0xf0 [ 4508.475926] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4508.489835] mdc_close+0x53b/0xd70 [mdc] [ 4508.493197] lmv_close+0x18d/0x490 [lmv] [ 4508.494877] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 4508.507293] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4508.513380] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4508.522665] ll_md_close+0x274/0x800 [lustre] [ 4508.528741] ll_file_release+0x128/0xc50 [lustre] [ 4508.536890] __fput+0x10d/0x440 [ 4508.539483] ____fput+0x12/0x20 [ 4508.547602] task_work_run+0xc6/0x110 [ 4508.550765] exit_to_usermode_loop+0x1dd/0x1f0 [ 4508.562879] do_syscall_64+0x3d6/0x440 [ 4508.566482] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4508.574819] RIP: 0033:0x7ff54af45cd8 [ 4508.577927] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4508.613056] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4508.620841] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 4508.630295] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 4508.642685] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 4508.656893] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 4508.665810] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4520.060440] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 4520.067983] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4520.116375] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4520.128689] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4520.142431] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4520.147040] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4520.160897] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4520.184843] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4520.192734] RAX: 0000000000006332 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 4520.205995] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 4520.211568] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 4520.217030] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 4520.223679] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 4520.231540] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 4520.241952] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4520.248966] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 4520.262336] Call Trace: [ 4520.264368] [ 4520.265064] ? show_regs.cold.9+0x22/0x2f [ 4520.272571] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4520.277630] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4520.279869] ? watchdog+0x40/0x40 [ 4520.286772] ? hrtimer_interrupt+0x128/0x2c0 [ 4520.289407] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4520.296679] ? apic_timer_interrupt+0xf/0x20 [ 4520.304241] [ 4520.306769] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4520.313609] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4520.323668] do_raw_spin_lock+0x177/0x190 [ 4520.337639] _raw_spin_lock+0x21/0x30 [ 4520.342796] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4520.356988] process_one_work+0x2c8/0x700 [ 4520.364309] worker_thread+0x296/0x6e0 [ 4520.369385] ? rescuer_thread+0x570/0x570 [ 4520.384643] kthread+0x1d1/0x200 [ 4520.385682] ? set_kthread_struct+0x70/0x70 [ 4520.389514] ret_from_fork+0x1f/0x30 [ 4524.058073] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 4524.060028] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 4524.061748] Modules linked in: [ 4524.071722] Modules linked in: [ 4524.071728] lustre(O) osp(O) [ 4524.076506] lustre(O) [ 4524.077341] ofd(O) [ 4524.078438] osp(O) [ 4524.079526] lod(O) [ 4524.087165] ofd(O) [ 4524.088186] mdt(O) [ 4524.089814] lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) [ 4524.090680] mdd(O) [ 4524.091819] lov(O) [ 4524.095604] mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4524.095838] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4524.095845] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4524.095848] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4524.095862] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4524.095866] RSP: 0000:ffffaf9fc1b97bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4524.095871] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 4524.095873] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 4524.095875] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 4524.095877] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 4524.095880] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4524.095882] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 4524.095885] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4524.095887] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 4524.095896] Call Trace: [ 4524.095903] [ 4524.095909] ? show_regs.cold.9+0x22/0x2f [ 4524.095920] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4524.095926] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4524.095930] ? watchdog+0x40/0x40 [ 4524.095934] ? hrtimer_interrupt+0x128/0x2c0 [ 4524.095939] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4524.095947] ? apic_timer_interrupt+0xf/0x20 [ 4524.095952] [ 4524.095955] ? native_safe_halt+0x2e/0x50 [ 4524.095959] kvm_wait+0xcd/0xe0 [ 4524.095967] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4524.095974] do_raw_spin_lock+0x177/0x190 [ 4524.095978] _raw_spin_lock+0x21/0x30 [ 4524.095984] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4524.096287] obd_mod_free+0x46/0x140 [ptlrpc] [ 4524.096490] mdc_commit_open+0x91/0xb0 [mdc] [ 4524.096530] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4524.096897] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4524.097134] after_reply+0xe2c/0x1780 [ptlrpc] [ 4524.097322] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4524.098446] osc(O) [ 4524.099338] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4524.161163] lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4524.161223] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4524.161232] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4524.161234] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4524.165843] ? do_wait_intr+0xf0/0xf0 [ 4524.165871] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4524.172347] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4524.172362] RSP: 0000:ffffaf9fc15234f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4524.172369] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 4524.172372] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 4524.172374] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4524.172376] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 4524.172379] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 4524.172382] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 4524.172385] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4524.172387] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 4524.172396] Call Trace: [ 4524.172402] [ 4524.172409] ? show_regs.cold.9+0x22/0x2f [ 4524.172421] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4524.172428] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4524.172432] ? watchdog+0x40/0x40 [ 4524.172437] ? hrtimer_interrupt+0x128/0x2c0 [ 4524.172441] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4524.172450] ? apic_timer_interrupt+0xf/0x20 [ 4524.172454] [ 4524.172457] ? native_safe_halt+0x2e/0x50 [ 4524.172460] kvm_wait+0xcd/0xe0 [ 4524.172468] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4524.172479] do_raw_spin_lock+0x177/0x190 [ 4524.172484] _raw_spin_lock+0x21/0x30 [ 4524.172489] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4524.174460] kthread+0x1d1/0x200 [ 4524.184276] ? do_raw_spin_unlock+0x75/0x190 [ 4524.187343] ? set_kthread_struct+0x70/0x70 [ 4524.190303] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4524.193149] ret_from_fork+0x1f/0x30 [ 4524.196068] ? do_wait_intr+0xf0/0xf0 [ 4524.547649] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4524.552830] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4524.556936] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4524.563082] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4524.569055] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4524.576037] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4524.580360] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4524.583044] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4524.593682] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4524.598319] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4524.600269] lmv_intent_open+0x338/0x1150 [lmv] [ 4524.604717] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4524.607584] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4524.610515] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4524.612741] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4524.615113] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4524.617211] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4524.619725] ? do_raw_spin_unlock+0x75/0x190 [ 4524.622135] ? _raw_spin_unlock+0x12/0x30 [ 4524.626121] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4524.632660] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4524.635533] ll_atomic_open+0x36a/0xd70 [lustre] [ 4524.638381] lookup_open+0x356/0xb40 [ 4524.639667] do_last+0x313/0xfc0 [ 4524.642033] ? nd_jump_root+0xe5/0x160 [ 4524.648790] ? path_init+0x437/0x520 [ 4524.650769] path_openat+0xf7/0x500 [ 4524.653803] ? do_raw_spin_unlock+0x75/0x190 [ 4524.660374] do_filp_open+0x99/0x140 [ 4524.663630] ? getname_flags+0x6e/0x330 [ 4524.668217] ? __check_object_size+0xff/0x256 [ 4524.675327] ? do_raw_spin_unlock+0x75/0x190 [ 4524.678164] ? _raw_spin_unlock+0x12/0x30 [ 4524.681082] do_sys_openat2+0x2b4/0x410 [ 4524.683324] do_sys_open+0x73/0xa0 [ 4524.687291] __x64_sys_openat+0x24/0x30 [ 4524.689030] do_syscall_64+0xc1/0x440 [ 4524.691185] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4524.695672] RIP: 0033:0x7ff54af45332 [ 4524.696911] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4524.712048] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4524.716148] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 4524.722164] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 4524.726467] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 4524.730731] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 4524.735369] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 4526.559083] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2567s! [ 4526.566950] Showing busy workqueues and worker pools: [ 4526.572256] workqueue events: flags=0x0 [ 4526.575094] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4526.581125] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 4526.593162] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4526.605329] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 4526.617344] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4526.628300] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 4526.643491] workqueue events_long: flags=0x0 [ 4526.648663] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4526.668438] pending: lu_objects_flush [obdclass] [ 4526.670301] workqueue events_unbound: flags=0x2 [ 4526.685221] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4526.690748] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4526.704118] workqueue events_power_efficient: flags=0x80 [ 4526.708436] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4526.717876] pending: neigh_periodic_work [ 4526.723056] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4526.740339] pending: do_cache_clean, check_lifetime [ 4526.746727] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4526.752909] pending: neigh_periodic_work [ 4526.754364] workqueue writeback: flags=0x4a [ 4526.763809] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4526.768941] pending: wb_workfn, wb_workfn [ 4526.774131] workqueue xprtiod: flags=0xa [ 4526.777471] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4526.781998] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 4526.786230] workqueue ipv6_addrconf: flags=0x40008 [ 4526.790981] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4526.796361] pending: addrconf_verify_work [ 4526.798008] workqueue knbd0-recv: flags=0x1a [ 4526.809664] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4526.811947] in-flight: 657:recv_work [ 4526.821179] workqueue ptlrpc_pinger: flags=0x2 [ 4526.822629] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4526.830965] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 4526.837289] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2567s workers=4 idle: 20831 12642 28859 [ 4526.840186] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4517s workers=2 idle: 659 [ 4533.916979] rcu: INFO: rcu_sched self-detected stall on CPU [ 4533.926068] rcu: 0-...!: (1352987 ticks this GP) idle=98a/1/0x4000000000000002 softirq=125154/125154 fqs=959 [ 4533.945117] (t=2580076 jiffies g=253585 q=3591) [ 4533.948527] rcu: rcu_sched kthread starved for 2575036 jiffies! g253585 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [ 4533.959176] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 4533.974054] rcu: RCU grace-period kthread stack dump: [ 4533.981366] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 4533.995514] Call Trace: [ 4533.999146] __schedule+0x351/0xcb0 [ 4534.006701] schedule+0xc0/0x180 [ 4534.012157] schedule_timeout+0xb4/0x190 [ 4534.018476] ? do_raw_spin_unlock+0x75/0x190 [ 4534.022850] ? __next_timer_interrupt+0x160/0x160 [ 4534.027460] ? prepare_to_swait_event+0xf3/0x140 [ 4534.030863] rcu_gp_kthread+0x8dd/0x15b0 [ 4534.033406] ? force_qs_rnp+0x2c0/0x2c0 [ 4534.037039] kthread+0x1d1/0x200 [ 4534.039063] ? set_kthread_struct+0x70/0x70 [ 4534.041466] ret_from_fork+0x1f/0x30 [ 4534.043948] rcu: Stack dump where RCU GP kthread last ran: [ 4534.047433] Sending NMI from CPU 0 to CPUs 1: [ 4534.056983] NMI backtrace for cpu 1 [ 4534.056990] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4534.056992] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4534.056994] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4534.056996] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4534.056998] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 [ 4534.057001] RAX: 000000000000570e RBX: ffff96848811cae0 RCX: 0000000000000001 [ 4534.057002] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 4534.057004] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4534.057005] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 4534.057007] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 4534.057008] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 4534.057010] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4534.057011] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 4534.057012] Call Trace: [ 4534.057013] [ 4534.057013] ? show_regs.cold.9+0x22/0x2f [ 4534.057015] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 4534.057016] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 4534.057017] ? nmi_handle+0xbd/0x210 [ 4534.057018] ? default_do_nmi+0x4b/0x1b0 [ 4534.057019] ? do_nmi+0x280/0x380 [ 4534.057020] ? end_repeat_nmi+0x16/0x1f [ 4534.057021] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4534.057022] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4534.057023] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4534.057024] [ 4534.057025] do_raw_spin_lock+0x177/0x190 [ 4534.057026] _raw_spin_lock+0x21/0x30 [ 4534.057026] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4534.057027] ? do_raw_spin_unlock+0x75/0x190 [ 4534.057028] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4534.057029] ? do_wait_intr+0xf0/0xf0 [ 4534.057030] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4534.057030] mdc_close+0x53b/0xd70 [mdc] [ 4534.057031] lmv_close+0x18d/0x490 [lmv] [ 4534.057032] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 4534.057033] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4534.057034] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4534.057035] ll_md_close+0x274/0x800 [lustre] [ 4534.057036] ll_file_release+0x128/0xc50 [lustre] [ 4534.057037] __fput+0x10d/0x440 [ 4534.057038] ____fput+0x12/0x20 [ 4534.057039] task_work_run+0xc6/0x110 [ 4534.057040] exit_to_usermode_loop+0x1dd/0x1f0 [ 4534.057041] do_syscall_64+0x3d6/0x440 [ 4534.057041] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4534.057042] RIP: 0033:0x7ff54af45cd8 [ 4534.057043] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4534.057045] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4534.057047] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 4534.057048] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 4534.057050] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 4534.057051] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 4534.057052] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4534.057507] NMI backtrace for cpu 0 [ 4534.268852] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4534.272939] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4534.276543] Call Trace: [ 4534.279425] [ 4534.282271] ? dump_stack+0xbb/0x10e [ 4534.287976] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 4534.295523] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 4534.299938] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 4534.303842] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 4534.309563] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 4534.317194] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 4534.323828] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 4534.330511] ? update_process_times+0x6d/0xc0 [ 4534.335932] ? tick_sched_handle.isra.18+0x27/0x90 [ 4534.337629] ? tick_sched_timer+0x53/0xb0 [ 4534.340041] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4534.345979] ? tick_sched_handle.isra.18+0x90/0x90 [ 4534.353233] ? hrtimer_interrupt+0x128/0x2c0 [ 4534.357996] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4534.365938] ? apic_timer_interrupt+0xf/0x20 [ 4534.371472] [ 4534.377820] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4534.381658] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4534.385562] ? do_raw_spin_lock+0x177/0x190 [ 4534.390346] ? _raw_spin_lock+0x21/0x30 [ 4534.393506] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4534.401368] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 4534.406579] ? mdc_commit_open+0x91/0xb0 [mdc] [ 4534.413411] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4534.418836] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4534.420995] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 4534.430508] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4534.433926] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4534.436510] ? do_wait_intr+0xf0/0xf0 [ 4534.438962] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4534.441973] ? kthread+0x1d1/0x200 [ 4534.443947] ? set_kthread_struct+0x70/0x70 [ 4534.446899] ? ret_from_fork+0x1f/0x30 [ 4548.059085] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 4548.065911] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4548.102440] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4548.110986] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4548.115436] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4548.119206] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4548.121755] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4548.132574] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4548.136739] RAX: 0000000000005609 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 4548.141422] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 4548.144483] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 4548.149696] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 4548.154259] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 4548.158196] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 4548.162632] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4548.165529] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 4548.169853] Call Trace: [ 4548.171250] [ 4548.172525] ? show_regs.cold.9+0x22/0x2f [ 4548.175339] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4548.177706] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4548.180197] ? watchdog+0x40/0x40 [ 4548.181680] ? hrtimer_interrupt+0x128/0x2c0 [ 4548.183220] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4548.185461] ? apic_timer_interrupt+0xf/0x20 [ 4548.187497] [ 4548.188778] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4548.191583] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4548.194354] do_raw_spin_lock+0x177/0x190 [ 4548.196651] _raw_spin_lock+0x21/0x30 [ 4548.198294] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4548.201434] process_one_work+0x2c8/0x700 [ 4548.204420] worker_thread+0x296/0x6e0 [ 4548.206509] ? rescuer_thread+0x570/0x570 [ 4548.208505] kthread+0x1d1/0x200 [ 4548.210148] ? set_kthread_struct+0x70/0x70 [ 4548.212333] ret_from_fork+0x1f/0x30 [ 4552.060032] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 4552.063672] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4552.091362] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4552.103191] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4552.112286] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4552.116619] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4552.127086] RSP: 0000:ffffaf9fc15234f0 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 4552.130890] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 4552.134253] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 4552.139952] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4552.143686] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 4552.147844] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 4552.151900] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 4552.159294] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4552.163429] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 4552.171804] Call Trace: [ 4552.174176] [ 4552.175014] ? show_regs.cold.9+0x22/0x2f [ 4552.178738] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4552.182842] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4552.186635] ? watchdog+0x40/0x40 [ 4552.187797] ? hrtimer_interrupt+0x128/0x2c0 [ 4552.191235] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4552.194910] ? apic_timer_interrupt+0xf/0x20 [ 4552.199254] [ 4552.201132] ? native_safe_halt+0x2e/0x50 [ 4552.203422] kvm_wait+0xcd/0xe0 [ 4552.205114] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4552.212818] do_raw_spin_lock+0x177/0x190 [ 4552.214952] _raw_spin_lock+0x21/0x30 [ 4552.217055] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4552.220331] ? do_raw_spin_unlock+0x75/0x190 [ 4552.224737] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4552.228673] ? do_wait_intr+0xf0/0xf0 [ 4552.230467] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4552.235862] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4552.239490] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4552.243572] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4552.246419] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4552.249420] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4552.251896] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4552.254167] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4552.256966] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4552.265631] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4552.270205] lmv_intent_open+0x338/0x1150 [lmv] [ 4552.272877] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4552.275736] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4552.278404] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4552.281489] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4552.284787] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4552.287680] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4552.290675] ? do_raw_spin_unlock+0x75/0x190 [ 4552.292935] ? _raw_spin_unlock+0x12/0x30 [ 4552.295216] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4552.298848] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4552.301115] ll_atomic_open+0x36a/0xd70 [lustre] [ 4552.303626] lookup_open+0x356/0xb40 [ 4552.305356] do_last+0x313/0xfc0 [ 4552.306654] ? nd_jump_root+0xe5/0x160 [ 4552.308417] ? path_init+0x437/0x520 [ 4552.310152] path_openat+0xf7/0x500 [ 4552.311482] ? do_raw_spin_unlock+0x75/0x190 [ 4552.315048] do_filp_open+0x99/0x140 [ 4552.317416] ? getname_flags+0x6e/0x330 [ 4552.318995] ? __check_object_size+0xff/0x256 [ 4552.321346] ? do_raw_spin_unlock+0x75/0x190 [ 4552.322983] ? _raw_spin_unlock+0x12/0x30 [ 4552.324832] do_sys_openat2+0x2b4/0x410 [ 4552.327002] do_sys_open+0x73/0xa0 [ 4552.328914] __x64_sys_openat+0x24/0x30 [ 4552.335362] do_syscall_64+0xc1/0x440 [ 4552.336981] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4552.341196] RIP: 0033:0x7ff54af45332 [ 4552.344819] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4552.361481] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4552.368680] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 4552.375059] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 4552.377192] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 4552.383652] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 4552.393200] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 4557.280541] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2598s! [ 4557.286305] Showing busy workqueues and worker pools: [ 4557.290631] workqueue events: flags=0x0 [ 4557.298207] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4557.305553] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 4557.317984] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4557.325669] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 4557.336187] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4557.343868] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 4557.352751] workqueue events_long: flags=0x0 [ 4557.357231] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4557.362879] pending: lu_objects_flush [obdclass] [ 4557.368243] workqueue events_unbound: flags=0x2 [ 4557.372547] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4557.378470] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4557.388354] workqueue events_power_efficient: flags=0x80 [ 4557.395434] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4557.405000] pending: neigh_periodic_work [ 4557.410951] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4557.416695] pending: do_cache_clean, check_lifetime [ 4557.422681] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4557.430239] pending: neigh_periodic_work [ 4557.436765] workqueue writeback: flags=0x4a [ 4557.438851] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4557.446298] pending: wb_workfn, wb_workfn [ 4557.451174] workqueue xprtiod: flags=0xa [ 4557.456556] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4557.463523] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 4557.471823] workqueue ipv6_addrconf: flags=0x40008 [ 4557.478718] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4557.483477] pending: addrconf_verify_work [ 4557.487240] workqueue knbd0-recv: flags=0x1a [ 4557.496346] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4557.502730] in-flight: 657:recv_work [ 4557.506646] workqueue ptlrpc_pinger: flags=0x2 [ 4557.512281] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4557.516515] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 4557.523544] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2598s workers=4 idle: 20831 12642 28859 [ 4557.532281] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4547s workers=2 idle: 659 [ 4560.061426] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 4560.067900] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4560.112592] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4560.122604] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4560.128584] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4560.133185] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4560.152814] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 4560.161455] RAX: 0000000000007c87 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 4560.170244] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 4560.173524] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4560.184342] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 4560.192667] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 4560.194998] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 4560.203190] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4560.209674] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 4560.220610] Call Trace: [ 4560.222469] [ 4560.223961] ? show_regs.cold.9+0x22/0x2f [ 4560.229912] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4560.234822] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4560.239455] ? watchdog+0x40/0x40 [ 4560.241369] ? hrtimer_interrupt+0x128/0x2c0 [ 4560.243861] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4560.246359] ? apic_timer_interrupt+0xf/0x20 [ 4560.248392] [ 4560.249399] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4560.252099] do_raw_spin_lock+0x177/0x190 [ 4560.254204] _raw_spin_lock+0x21/0x30 [ 4560.255616] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4560.258705] ? do_raw_spin_unlock+0x75/0x190 [ 4560.260661] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4560.263785] ? do_wait_intr+0xf0/0xf0 [ 4560.265442] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4560.267769] mdc_close+0x53b/0xd70 [mdc] [ 4560.276393] lmv_close+0x18d/0x490 [lmv] [ 4560.278567] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 4560.286226] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4560.293613] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4560.305072] ll_md_close+0x274/0x800 [lustre] [ 4560.306444] ll_file_release+0x128/0xc50 [lustre] [ 4560.310590] __fput+0x10d/0x440 [ 4560.311375] ____fput+0x12/0x20 [ 4560.313600] task_work_run+0xc6/0x110 [ 4560.318191] exit_to_usermode_loop+0x1dd/0x1f0 [ 4560.326841] do_syscall_64+0x3d6/0x440 [ 4560.328725] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4560.332613] RIP: 0033:0x7ff54af45cd8 [ 4560.335000] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4560.347842] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4560.351526] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 4560.356184] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 4560.360636] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 4560.365602] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 4560.370669] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4576.059992] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 4576.062686] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4576.095752] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4576.104683] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4576.109569] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4576.114079] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4576.116504] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4576.123105] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4576.125713] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 4576.128730] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 4576.131019] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 4576.133725] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 4576.136846] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 4576.140267] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 4576.143856] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4576.146281] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 4576.149395] Call Trace: [ 4576.150443] [ 4576.151616] ? show_regs.cold.9+0x22/0x2f [ 4576.157502] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4576.161778] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4576.168451] ? watchdog+0x40/0x40 [ 4576.173831] ? hrtimer_interrupt+0x128/0x2c0 [ 4576.176502] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4576.179500] ? apic_timer_interrupt+0xf/0x20 [ 4576.181732] [ 4576.183074] ? native_safe_halt+0x2e/0x50 [ 4576.187406] kvm_wait+0xcd/0xe0 [ 4576.190022] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4576.196887] do_raw_spin_lock+0x177/0x190 [ 4576.199941] _raw_spin_lock+0x21/0x30 [ 4576.203534] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4576.208202] process_one_work+0x2c8/0x700 [ 4576.213655] worker_thread+0x296/0x6e0 [ 4576.215086] ? rescuer_thread+0x570/0x570 [ 4576.218463] kthread+0x1d1/0x200 [ 4576.219663] ? set_kthread_struct+0x70/0x70 [ 4576.221404] ret_from_fork+0x1f/0x30 [ 4580.061537] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 4580.066663] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4580.104799] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4580.109968] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4580.114770] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4580.117653] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4580.125201] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4580.128231] RAX: 0000000000007e18 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 4580.134202] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 4580.142462] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4580.150163] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 4580.156793] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 4580.161454] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 4580.168588] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4580.171229] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 4580.176887] Call Trace: [ 4580.177706] [ 4580.178368] ? show_regs.cold.9+0x22/0x2f [ 4580.183092] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4580.190063] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4580.193338] ? watchdog+0x40/0x40 [ 4580.197219] ? hrtimer_interrupt+0x128/0x2c0 [ 4580.201118] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4580.212268] ? apic_timer_interrupt+0xf/0x20 [ 4580.217820] [ 4580.222099] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4580.228369] do_raw_spin_lock+0x177/0x190 [ 4580.232122] _raw_spin_lock+0x21/0x30 [ 4580.235130] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4580.244197] ? do_raw_spin_unlock+0x75/0x190 [ 4580.250347] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4580.253326] ? do_wait_intr+0xf0/0xf0 [ 4580.256963] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4580.264870] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4580.268967] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4580.279751] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4580.288048] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4580.294258] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4580.300343] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4580.306677] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4580.310315] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4580.315261] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4580.320091] lmv_intent_open+0x338/0x1150 [lmv] [ 4580.322055] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4580.325438] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4580.329098] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4580.331497] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4580.336777] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4580.341243] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4580.344229] ? do_raw_spin_unlock+0x75/0x190 [ 4580.348262] ? _raw_spin_unlock+0x12/0x30 [ 4580.350935] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4580.354273] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4580.364379] ll_atomic_open+0x36a/0xd70 [lustre] [ 4580.365944] lookup_open+0x356/0xb40 [ 4580.371234] do_last+0x313/0xfc0 [ 4580.373125] ? nd_jump_root+0xe5/0x160 [ 4580.377066] ? path_init+0x437/0x520 [ 4580.384397] path_openat+0xf7/0x500 [ 4580.385470] ? do_raw_spin_unlock+0x75/0x190 [ 4580.388883] do_filp_open+0x99/0x140 [ 4580.389787] ? getname_flags+0x6e/0x330 [ 4580.390629] ? __check_object_size+0xff/0x256 [ 4580.393171] ? do_raw_spin_unlock+0x75/0x190 [ 4580.394936] ? _raw_spin_unlock+0x12/0x30 [ 4580.397432] do_sys_openat2+0x2b4/0x410 [ 4580.399233] do_sys_open+0x73/0xa0 [ 4580.401959] __x64_sys_openat+0x24/0x30 [ 4580.404015] do_syscall_64+0xc1/0x440 [ 4580.408801] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4580.412860] RIP: 0033:0x7ff54af45332 [ 4580.415963] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4580.427608] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4580.432770] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 4580.440758] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 4580.449052] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 4580.454713] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 4580.459102] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 4584.055083] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [ 4584.059631] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4584.098456] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4584.104455] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4584.108453] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4584.111434] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4584.139693] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4584.153708] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 4584.168175] RDX: 0000000000002c6f RSI: 0000000000000003 RDI: ffff96848811cae0 [ 4584.176197] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 4584.182546] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 4584.186365] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4584.189636] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 4584.193531] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4584.195709] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 4584.199358] Call Trace: [ 4584.200554] [ 4584.201359] ? show_regs.cold.9+0x22/0x2f [ 4584.203418] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4584.206638] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4584.208684] ? watchdog+0x40/0x40 [ 4584.210349] ? hrtimer_interrupt+0x128/0x2c0 [ 4584.212327] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4584.214566] ? apic_timer_interrupt+0xf/0x20 [ 4584.216657] [ 4584.217592] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4584.220205] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4584.222445] do_raw_spin_lock+0x177/0x190 [ 4584.224194] _raw_spin_lock+0x21/0x30 [ 4584.225632] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4584.228020] obd_mod_free+0x46/0x140 [ptlrpc] [ 4584.234835] mdc_commit_open+0x91/0xb0 [mdc] [ 4584.243941] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4584.247885] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4584.250497] after_reply+0xe2c/0x1780 [ptlrpc] [ 4584.252953] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4584.255879] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4584.258134] ? do_wait_intr+0xf0/0xf0 [ 4584.259998] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4584.263098] kthread+0x1d1/0x200 [ 4584.264760] ? set_kthread_struct+0x70/0x70 [ 4584.266731] ret_from_fork+0x1f/0x30 [ 4588.001016] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2629s! [ 4588.007349] Showing busy workqueues and worker pools: [ 4588.009014] workqueue events: flags=0x0 [ 4588.011905] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4588.016626] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 4588.025359] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4588.028587] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 4588.035633] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4588.038415] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 4588.047691] workqueue events_long: flags=0x0 [ 4588.049665] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4588.054515] pending: lu_objects_flush [obdclass] [ 4588.059368] workqueue events_unbound: flags=0x2 [ 4588.062337] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4588.062446] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 4588.067417] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4588.067500] workqueue events_power_efficient: flags=0x80 [ 4588.067506] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4588.067518] pending: neigh_periodic_work [ 4588.067528] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4588.067535] pending: do_cache_clean, check_lifetime [ 4588.067550] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4588.067558] pending: neigh_periodic_work [ 4588.067569] workqueue writeback: flags=0x4a [ 4588.067572] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4588.067578] pending: wb_workfn, wb_workfn [ 4588.067592] workqueue xprtiod: flags=0xa [ 4588.067594] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4588.067599] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 4588.067619] workqueue ipv6_addrconf: flags=0x40008 [ 4588.067622] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4588.067629] pending: addrconf_verify_work [ 4588.067641] workqueue knbd0-recv: flags=0x1a [ 4588.067644] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4588.067651] in-flight: 657:recv_work [ 4588.067661] workqueue ptlrpc_pinger: flags=0x2 [ 4588.067663] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4588.067668] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 4588.068052] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2629s workers=4 idle: 20831 12642 28859 [ 4588.068066] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4578s workers=2 idle: 659 [ 4588.233894] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4588.290631] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4588.299295] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4588.311139] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4588.313207] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4588.329969] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4588.334145] RAX: 0000000000006f8d RBX: ffff96848811cae0 RCX: 0000000000000001 [ 4588.337051] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 4588.340423] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4588.343593] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 4588.346678] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 4588.350791] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 4588.357137] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4588.363600] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 4588.374639] Call Trace: [ 4588.375522] [ 4588.380495] ? show_regs.cold.9+0x22/0x2f [ 4588.392456] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4588.396708] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4588.400463] ? watchdog+0x40/0x40 [ 4588.402402] ? hrtimer_interrupt+0x128/0x2c0 [ 4588.405012] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4588.407458] ? apic_timer_interrupt+0xf/0x20 [ 4588.409823] [ 4588.411174] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4588.414622] do_raw_spin_lock+0x177/0x190 [ 4588.416320] _raw_spin_lock+0x21/0x30 [ 4588.418208] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4588.421699] ? do_raw_spin_unlock+0x75/0x190 [ 4588.423486] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4588.426630] ? do_wait_intr+0xf0/0xf0 [ 4588.428444] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4588.434539] mdc_close+0x53b/0xd70 [mdc] [ 4588.438849] lmv_close+0x18d/0x490 [lmv] [ 4588.441346] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 4588.444494] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4588.449713] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4588.452535] ll_md_close+0x274/0x800 [lustre] [ 4588.456688] ll_file_release+0x128/0xc50 [lustre] [ 4588.463281] __fput+0x10d/0x440 [ 4588.466436] ____fput+0x12/0x20 [ 4588.468726] task_work_run+0xc6/0x110 [ 4588.473548] exit_to_usermode_loop+0x1dd/0x1f0 [ 4588.476162] do_syscall_64+0x3d6/0x440 [ 4588.477465] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4588.489291] RIP: 0033:0x7ff54af45cd8 [ 4588.498642] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4588.516477] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4588.525660] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 4588.531600] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 4588.537286] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 4588.546944] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 4588.551636] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4604.059031] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 4604.068270] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4604.107459] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4604.118359] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4604.127246] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4604.134717] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4604.137499] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4604.164439] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4604.175715] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 4604.185396] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 4604.191928] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 4604.204923] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 4604.217323] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 4604.230396] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 4604.247513] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4604.253237] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 4604.263487] Call Trace: [ 4604.272127] [ 4604.272951] ? show_regs.cold.9+0x22/0x2f [ 4604.274481] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4604.283476] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4604.289383] ? watchdog+0x40/0x40 [ 4604.292912] ? hrtimer_interrupt+0x128/0x2c0 [ 4604.298036] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4604.308438] ? apic_timer_interrupt+0xf/0x20 [ 4604.313333] [ 4604.320260] ? native_safe_halt+0x2e/0x50 [ 4604.332511] kvm_wait+0xcd/0xe0 [ 4604.339143] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4604.345142] do_raw_spin_lock+0x177/0x190 [ 4604.347874] _raw_spin_lock+0x21/0x30 [ 4604.351318] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4604.355806] process_one_work+0x2c8/0x700 [ 4604.360479] worker_thread+0x296/0x6e0 [ 4604.363835] ? rescuer_thread+0x570/0x570 [ 4604.370912] kthread+0x1d1/0x200 [ 4604.374335] ? set_kthread_struct+0x70/0x70 [ 4604.376707] ret_from_fork+0x1f/0x30 [ 4608.061450] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 4608.068820] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4608.113216] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4608.128873] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4608.137281] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4608.141955] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4608.164471] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4608.171199] RAX: 0000000000007f85 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 4608.187410] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 4608.199569] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4608.216520] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 4608.224697] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 4608.235804] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 4608.240168] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4608.247245] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 4608.256223] Call Trace: [ 4608.261089] [ 4608.265031] ? show_regs.cold.9+0x22/0x2f [ 4608.267457] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4608.273934] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4608.285201] ? watchdog+0x40/0x40 [ 4608.288690] ? hrtimer_interrupt+0x128/0x2c0 [ 4608.293872] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4608.295707] ? apic_timer_interrupt+0xf/0x20 [ 4608.306975] [ 4608.308632] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4608.319050] do_raw_spin_lock+0x177/0x190 [ 4608.324151] _raw_spin_lock+0x21/0x30 [ 4608.329823] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4608.337777] ? do_raw_spin_unlock+0x75/0x190 [ 4608.347237] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4608.354948] ? do_wait_intr+0xf0/0xf0 [ 4608.359514] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4608.367613] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4608.374293] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4608.376456] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4608.380620] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4608.389123] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4608.403648] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4608.407859] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4608.425137] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4608.434050] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4608.444752] lmv_intent_open+0x338/0x1150 [lmv] [ 4608.446421] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4608.452104] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4608.456907] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4608.462718] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4608.466686] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4608.472619] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4608.474282] ? do_raw_spin_unlock+0x75/0x190 [ 4608.478107] ? _raw_spin_unlock+0x12/0x30 [ 4608.481326] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4608.498704] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4608.506858] ll_atomic_open+0x36a/0xd70 [lustre] [ 4608.511423] lookup_open+0x356/0xb40 [ 4608.512695] do_last+0x313/0xfc0 [ 4608.520305] ? nd_jump_root+0xe5/0x160 [ 4608.530010] ? path_init+0x437/0x520 [ 4608.537298] path_openat+0xf7/0x500 [ 4608.538384] ? do_raw_spin_unlock+0x75/0x190 [ 4608.544564] do_filp_open+0x99/0x140 [ 4608.546021] ? getname_flags+0x6e/0x330 [ 4608.547475] ? __check_object_size+0xff/0x256 [ 4608.557634] ? do_raw_spin_unlock+0x75/0x190 [ 4608.559583] ? _raw_spin_unlock+0x12/0x30 [ 4608.566627] do_sys_openat2+0x2b4/0x410 [ 4608.571449] do_sys_open+0x73/0xa0 [ 4608.573010] __x64_sys_openat+0x24/0x30 [ 4608.574910] do_syscall_64+0xc1/0x440 [ 4608.576833] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4608.582133] RIP: 0033:0x7ff54af45332 [ 4608.584701] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4608.602677] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4608.607087] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 4608.609897] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 4608.611676] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 4608.617157] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 4608.620997] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 4616.058716] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [ 4616.060328] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [ 4616.073759] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4616.073843] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4616.073859] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4616.073862] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4616.073876] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4616.073881] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4616.073886] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 4616.073888] RDX: 0000000000005fda RSI: 0000000000000003 RDI: ffff96848811cae0 [ 4616.073891] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 4616.073893] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 4616.073896] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4616.073899] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 4616.073902] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4616.073904] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 4616.073911] Call Trace: [ 4616.073916] [ 4616.073921] ? show_regs.cold.9+0x22/0x2f [ 4616.073929] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4616.073935] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4616.073939] ? watchdog+0x40/0x40 [ 4616.073943] ? hrtimer_interrupt+0x128/0x2c0 [ 4616.073947] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4616.073955] ? apic_timer_interrupt+0xf/0x20 [ 4616.073960] [ 4616.073963] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4616.073967] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4616.073972] do_raw_spin_lock+0x177/0x190 [ 4616.073986] _raw_spin_lock+0x21/0x30 [ 4616.073991] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4616.081639] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) [ 4616.144292] obd_mod_free+0x46/0x140 [ptlrpc] [ 4616.144590] mdc_commit_open+0x91/0xb0 [mdc] [ 4616.144629] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4616.160907] mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4616.160976] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4616.160983] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4616.160985] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4616.160999] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4616.161004] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4616.161007] RAX: 0000000000004787 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 4616.161009] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 4616.161012] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4616.161014] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 4616.161016] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 4616.161019] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 4616.161022] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4616.161024] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 4616.161032] Call Trace: [ 4616.161038] [ 4616.161044] ? show_regs.cold.9+0x22/0x2f [ 4616.162197] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4616.162203] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4616.162208] ? watchdog+0x40/0x40 [ 4616.162212] ? hrtimer_interrupt+0x128/0x2c0 [ 4616.162217] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4616.162224] ? apic_timer_interrupt+0xf/0x20 [ 4616.162229] [ 4616.162232] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4616.162241] do_raw_spin_lock+0x177/0x190 [ 4616.162247] _raw_spin_lock+0x21/0x30 [ 4616.162253] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4616.163374] ? do_raw_spin_unlock+0x75/0x190 [ 4616.163382] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4616.174367] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4616.177359] after_reply+0xe2c/0x1780 [ptlrpc] [ 4616.177820] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4616.178320] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4616.186775] ? do_wait_intr+0xf0/0xf0 [ 4616.186799] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4616.188343] mdc_close+0x53b/0xd70 [mdc] [ 4616.188384] lmv_close+0x18d/0x490 [lmv] [ 4616.188414] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 4616.188768] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4616.188886] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4616.188970] ll_md_close+0x274/0x800 [lustre] [ 4616.189814] ll_file_release+0x128/0xc50 [lustre] [ 4616.189913] __fput+0x10d/0x440 [ 4616.189926] ____fput+0x12/0x20 [ 4616.189931] task_work_run+0xc6/0x110 [ 4616.189941] exit_to_usermode_loop+0x1dd/0x1f0 [ 4616.189948] do_syscall_64+0x3d6/0x440 [ 4616.189953] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4616.189963] RIP: 0033:0x7ff54af45cd8 [ 4616.189972] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4616.189976] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4616.189982] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 4616.189984] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 4616.189987] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 4616.189989] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 4616.189992] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4616.838896] ? do_wait_intr+0xf0/0xf0 [ 4616.841155] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4616.846598] kthread+0x1d1/0x200 [ 4616.849627] ? set_kthread_struct+0x70/0x70 [ 4616.852816] ret_from_fork+0x1f/0x30 [ 4618.209990] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2659s! [ 4618.223667] Showing busy workqueues and worker pools: [ 4618.226725] workqueue events: flags=0x0 [ 4618.230546] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4618.238397] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 4618.252415] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4618.265315] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 4618.278215] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4618.288330] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 4618.299526] workqueue events_long: flags=0x0 [ 4618.301199] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4618.316777] pending: lu_objects_flush [obdclass] [ 4618.332969] workqueue events_unbound: flags=0x2 [ 4618.339331] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4618.348509] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4618.369705] workqueue events_power_efficient: flags=0x80 [ 4618.376470] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4618.390575] pending: neigh_periodic_work [ 4618.392244] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4618.403961] pending: do_cache_clean, check_lifetime [ 4618.412548] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4618.419346] pending: neigh_periodic_work [ 4618.424197] workqueue writeback: flags=0x4a [ 4618.425591] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4618.434248] pending: wb_workfn, wb_workfn [ 4618.439479] workqueue xprtiod: flags=0xa [ 4618.440993] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4618.455705] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 4618.468392] workqueue ipv6_addrconf: flags=0x40008 [ 4618.472034] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4618.481890] pending: addrconf_verify_work [ 4618.485871] workqueue knbd0-recv: flags=0x1a [ 4618.495245] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4618.504153] in-flight: 657:recv_work [ 4618.505623] workqueue ptlrpc_pinger: flags=0x2 [ 4618.527196] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4618.544637] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 4618.550554] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2659s workers=4 idle: 20831 12642 28859 [ 4618.557971] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4608s workers=2 idle: 659 [ 4632.059032] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 4632.063833] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4632.111026] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4632.127859] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4632.142645] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4632.147011] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4632.148327] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4632.171119] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4632.177261] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 4632.184131] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 4632.193393] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 4632.198458] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 4632.206105] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 4632.211164] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 4632.216873] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4632.221616] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 4632.227443] Call Trace: [ 4632.234183] [ 4632.234934] ? show_regs.cold.9+0x22/0x2f [ 4632.236302] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4632.240454] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4632.243659] ? watchdog+0x40/0x40 [ 4632.244944] ? hrtimer_interrupt+0x128/0x2c0 [ 4632.248491] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4632.250185] ? apic_timer_interrupt+0xf/0x20 [ 4632.255675] [ 4632.259577] ? native_safe_halt+0x2e/0x50 [ 4632.264533] kvm_wait+0xcd/0xe0 [ 4632.269687] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4632.275128] do_raw_spin_lock+0x177/0x190 [ 4632.277640] _raw_spin_lock+0x21/0x30 [ 4632.280618] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4632.286997] process_one_work+0x2c8/0x700 [ 4632.293531] worker_thread+0x296/0x6e0 [ 4632.299667] ? rescuer_thread+0x570/0x570 [ 4632.301588] kthread+0x1d1/0x200 [ 4632.302781] ? set_kthread_struct+0x70/0x70 [ 4632.313014] ret_from_fork+0x1f/0x30 [ 4636.061555] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 4636.069127] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4636.114961] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4636.123263] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4636.133836] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4636.148606] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4636.181069] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4636.193683] RAX: 0000000000006494 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 4636.205598] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 4636.213835] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4636.221548] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 4636.225651] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 4636.229851] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 4636.233392] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4636.236812] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 4636.241246] Call Trace: [ 4636.242614] [ 4636.243635] ? show_regs.cold.9+0x22/0x2f [ 4636.245646] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4636.248246] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4636.250255] ? watchdog+0x40/0x40 [ 4636.251701] ? hrtimer_interrupt+0x128/0x2c0 [ 4636.256481] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4636.259577] ? apic_timer_interrupt+0xf/0x20 [ 4636.261644] [ 4636.263700] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4636.268570] do_raw_spin_lock+0x177/0x190 [ 4636.271006] _raw_spin_lock+0x21/0x30 [ 4636.273275] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4636.282626] ? do_raw_spin_unlock+0x75/0x190 [ 4636.288432] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4636.292783] ? do_wait_intr+0xf0/0xf0 [ 4636.294041] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4636.298142] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4636.301844] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4636.305248] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4636.309164] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4636.312181] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4636.314781] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4636.317158] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4636.320159] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4636.322261] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4636.324525] lmv_intent_open+0x338/0x1150 [lmv] [ 4636.326869] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4636.329601] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4636.337078] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4636.338390] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4636.346085] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4636.352530] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4636.356182] ? do_raw_spin_unlock+0x75/0x190 [ 4636.360727] ? _raw_spin_unlock+0x12/0x30 [ 4636.362889] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4636.365841] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4636.368799] ll_atomic_open+0x36a/0xd70 [lustre] [ 4636.372528] lookup_open+0x356/0xb40 [ 4636.374348] do_last+0x313/0xfc0 [ 4636.376115] ? nd_jump_root+0xe5/0x160 [ 4636.377771] ? path_init+0x437/0x520 [ 4636.379257] path_openat+0xf7/0x500 [ 4636.381149] ? do_raw_spin_unlock+0x75/0x190 [ 4636.382771] do_filp_open+0x99/0x140 [ 4636.384368] ? getname_flags+0x6e/0x330 [ 4636.386162] ? __check_object_size+0xff/0x256 [ 4636.388662] ? do_raw_spin_unlock+0x75/0x190 [ 4636.390268] ? _raw_spin_unlock+0x12/0x30 [ 4636.392611] do_sys_openat2+0x2b4/0x410 [ 4636.394597] do_sys_open+0x73/0xa0 [ 4636.396142] __x64_sys_openat+0x24/0x30 [ 4636.397563] do_syscall_64+0xc1/0x440 [ 4636.398590] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4636.399758] RIP: 0033:0x7ff54af45332 [ 4636.400741] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4636.407280] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4636.413052] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 4636.416913] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 4636.420236] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 4636.424954] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 4636.429207] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 4644.055056] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [ 4644.058029] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [ 4644.061738] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4644.061875] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4644.061883] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4644.061885] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4644.061903] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4644.061908] RSP: 0000:ffffaf9fc1b97bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4644.061913] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 4644.061916] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 4644.061918] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 4644.061920] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 4644.061923] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4644.061927] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 4644.061929] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4644.061932] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 4644.061939] Call Trace: [ 4644.061946] [ 4644.061952] ? show_regs.cold.9+0x22/0x2f [ 4644.061965] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4644.061972] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4644.061977] ? watchdog+0x40/0x40 [ 4644.061981] ? hrtimer_interrupt+0x128/0x2c0 [ 4644.061985] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4644.061991] ? apic_timer_interrupt+0xf/0x20 [ 4644.062046] [ 4644.062051] ? native_safe_halt+0x2e/0x50 [ 4644.062055] kvm_wait+0xcd/0xe0 [ 4644.062063] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4644.062069] do_raw_spin_lock+0x177/0x190 [ 4644.062074] _raw_spin_lock+0x21/0x30 [ 4644.062079] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4644.062401] obd_mod_free+0x46/0x140 [ptlrpc] [ 4644.062590] mdc_commit_open+0x91/0xb0 [mdc] [ 4644.062633] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4644.062842] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4644.063072] after_reply+0xe2c/0x1780 [ptlrpc] [ 4644.063286] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4644.063532] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4644.063756] ? do_wait_intr+0xf0/0xf0 [ 4644.063766] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4644.063998] kthread+0x1d1/0x200 [ 4644.064004] ? set_kthread_struct+0x70/0x70 [ 4644.064008] ret_from_fork+0x1f/0x30 [ 4644.192708] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4644.221440] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4644.226924] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4644.231364] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4644.234483] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4644.249902] RSP: 0018:ffffaf9fc1543ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4644.254404] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 4644.259786] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 4644.263982] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4644.267497] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 4644.271519] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 4644.275060] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 4644.279856] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4644.282612] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 4644.286388] Call Trace: [ 4644.287580] [ 4644.288566] ? show_regs.cold.9+0x22/0x2f [ 4644.290600] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4644.296482] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4644.299847] ? watchdog+0x40/0x40 [ 4644.301680] ? hrtimer_interrupt+0x128/0x2c0 [ 4644.306671] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4644.309079] ? apic_timer_interrupt+0xf/0x20 [ 4644.311446] [ 4644.312246] ? native_safe_halt+0x2e/0x50 [ 4644.314827] kvm_wait+0xcd/0xe0 [ 4644.316568] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4644.321420] do_raw_spin_lock+0x177/0x190 [ 4644.334258] _raw_spin_lock+0x21/0x30 [ 4644.335904] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4644.348524] ? do_raw_spin_unlock+0x75/0x190 [ 4644.358640] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4644.365714] ? do_wait_intr+0xf0/0xf0 [ 4644.370903] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4644.379776] mdc_close+0x53b/0xd70 [mdc] [ 4644.384958] lmv_close+0x18d/0x490 [lmv] [ 4644.387199] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 4644.390653] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4644.394402] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4644.397297] ll_md_close+0x274/0x800 [lustre] [ 4644.399038] ll_file_release+0x128/0xc50 [lustre] [ 4644.403543] __fput+0x10d/0x440 [ 4644.409032] ____fput+0x12/0x20 [ 4644.410222] task_work_run+0xc6/0x110 [ 4644.414984] exit_to_usermode_loop+0x1dd/0x1f0 [ 4644.418761] do_syscall_64+0x3d6/0x440 [ 4644.422658] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4644.425049] RIP: 0033:0x7ff54af45cd8 [ 4644.429947] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4644.438316] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4644.442344] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 4644.445902] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 4644.449108] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 4644.452965] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 4644.458420] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4648.927313] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2690s! [ 4648.930653] Showing busy workqueues and worker pools: [ 4648.933664] workqueue events: flags=0x0 [ 4648.936777] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4648.943678] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 4648.950521] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4648.955500] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 4648.964865] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4648.973649] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 4648.985050] workqueue events_long: flags=0x0 [ 4648.989097] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4648.993624] pending: lu_objects_flush [obdclass] [ 4648.998541] workqueue events_unbound: flags=0x2 [ 4649.002425] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4649.007602] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4649.016329] workqueue events_power_efficient: flags=0x80 [ 4649.019226] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4649.023694] pending: neigh_periodic_work [ 4649.028061] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4649.031686] pending: do_cache_clean, check_lifetime [ 4649.035245] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4649.039628] pending: neigh_periodic_work [ 4649.042109] workqueue writeback: flags=0x4a [ 4649.045239] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4649.047567] pending: wb_workfn, wb_workfn [ 4649.049572] workqueue xprtiod: flags=0xa [ 4649.051430] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4649.054322] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 4649.057555] workqueue ipv6_addrconf: flags=0x40008 [ 4649.059632] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4649.062633] pending: addrconf_verify_work [ 4649.067020] workqueue knbd0-recv: flags=0x1a [ 4649.073625] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4649.077615] in-flight: 657:recv_work [ 4649.081448] workqueue ptlrpc_pinger: flags=0x2 [ 4649.084360] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4649.086862] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 4649.090842] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2690s workers=4 idle: 20831 12642 28859 [ 4649.098958] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4639s workers=2 idle: 659 [ 4660.059145] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 4660.064902] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4660.100424] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4660.110466] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4660.118303] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4660.122454] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4660.128031] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4660.146244] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4660.153509] RAX: 0000000000005a83 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 4660.162678] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 4660.167952] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 4660.174134] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 4660.179887] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 4660.186840] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 4660.194299] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4660.199488] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 4660.207472] Call Trace: [ 4660.208408] [ 4660.209135] ? show_regs.cold.9+0x22/0x2f [ 4660.214924] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4660.220681] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4660.224368] ? watchdog+0x40/0x40 [ 4660.228186] ? hrtimer_interrupt+0x128/0x2c0 [ 4660.232721] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4660.236411] ? apic_timer_interrupt+0xf/0x20 [ 4660.240388] [ 4660.242459] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4660.248681] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4660.251412] do_raw_spin_lock+0x177/0x190 [ 4660.256898] _raw_spin_lock+0x21/0x30 [ 4660.260486] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4660.264539] process_one_work+0x2c8/0x700 [ 4660.266649] worker_thread+0x296/0x6e0 [ 4660.268698] ? rescuer_thread+0x570/0x570 [ 4660.271768] kthread+0x1d1/0x200 [ 4660.273526] ? set_kthread_struct+0x70/0x70 [ 4660.275802] ret_from_fork+0x1f/0x30 [ 4664.062757] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 4664.067152] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4664.114507] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4664.127744] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4664.134304] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4664.142450] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4664.155417] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4664.166115] RAX: 0000000000007f86 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 4664.179514] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 4664.188136] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4664.195220] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 4664.200990] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 4664.205610] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 4664.214397] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4664.222159] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 4664.227695] Call Trace: [ 4664.233233] [ 4664.233999] ? show_regs.cold.9+0x22/0x2f [ 4664.242889] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4664.244377] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4664.245459] ? watchdog+0x40/0x40 [ 4664.255983] ? hrtimer_interrupt+0x128/0x2c0 [ 4664.262530] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4664.271157] ? apic_timer_interrupt+0xf/0x20 [ 4664.274549] [ 4664.275444] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4664.279084] do_raw_spin_lock+0x177/0x190 [ 4664.284205] _raw_spin_lock+0x21/0x30 [ 4664.288129] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4664.296364] ? do_raw_spin_unlock+0x75/0x190 [ 4664.300503] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4664.303596] ? do_wait_intr+0xf0/0xf0 [ 4664.306005] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4664.309902] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4664.315547] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4664.318539] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4664.321165] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4664.326180] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4664.329944] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4664.333079] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4664.339218] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4664.341504] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4664.352997] lmv_intent_open+0x338/0x1150 [lmv] [ 4664.356060] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4664.362306] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4664.367902] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4664.376483] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4664.383828] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4664.391424] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4664.396560] ? do_raw_spin_unlock+0x75/0x190 [ 4664.400469] ? _raw_spin_unlock+0x12/0x30 [ 4664.407826] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4664.418916] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4664.420479] ll_atomic_open+0x36a/0xd70 [lustre] [ 4664.421929] lookup_open+0x356/0xb40 [ 4664.429999] do_last+0x313/0xfc0 [ 4664.434217] ? nd_jump_root+0xe5/0x160 [ 4664.435177] ? path_init+0x437/0x520 [ 4664.436026] path_openat+0xf7/0x500 [ 4664.444106] ? do_raw_spin_unlock+0x75/0x190 [ 4664.446815] do_filp_open+0x99/0x140 [ 4664.453453] ? getname_flags+0x6e/0x330 [ 4664.455006] ? __check_object_size+0xff/0x256 [ 4664.460563] ? do_raw_spin_unlock+0x75/0x190 [ 4664.465838] ? _raw_spin_unlock+0x12/0x30 [ 4664.470541] do_sys_openat2+0x2b4/0x410 [ 4664.471802] do_sys_open+0x73/0xa0 [ 4664.472935] __x64_sys_openat+0x24/0x30 [ 4664.483457] do_syscall_64+0xc1/0x440 [ 4664.486213] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4664.493499] RIP: 0033:0x7ff54af45332 [ 4664.494689] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4664.520401] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4664.522977] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 4664.531696] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 4664.538992] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 4664.544878] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 4664.556635] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 4672.063775] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [ 4672.073700] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4672.131280] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4672.148788] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4672.159964] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4672.166509] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4672.188953] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 4672.216335] RAX: 0000000000007cad RBX: ffff96848811cae0 RCX: 0000000000000001 [ 4672.227909] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 4672.240221] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4672.259290] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 4672.273313] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 4672.289289] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 4672.299232] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4672.303171] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 4672.313299] Call Trace: [ 4672.316862] [ 4672.320906] ? show_regs.cold.9+0x22/0x2f [ 4672.325342] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4672.338851] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4672.359405] ? watchdog+0x40/0x40 [ 4672.366310] ? hrtimer_interrupt+0x128/0x2c0 [ 4672.379474] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4672.385654] ? apic_timer_interrupt+0xf/0x20 [ 4672.389884] [ 4672.392230] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4672.396589] do_raw_spin_lock+0x177/0x190 [ 4672.398844] _raw_spin_lock+0x21/0x30 [ 4672.403230] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4672.409592] ? do_raw_spin_unlock+0x75/0x190 [ 4672.414004] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4672.420363] ? do_wait_intr+0xf0/0xf0 [ 4672.424485] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4672.431827] mdc_close+0x53b/0xd70 [mdc] [ 4672.433879] lmv_close+0x18d/0x490 [lmv] [ 4672.441700] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 4672.449280] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4672.456769] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4672.467675] ll_md_close+0x274/0x800 [lustre] [ 4672.486246] ll_file_release+0x128/0xc50 [lustre] [ 4672.494713] __fput+0x10d/0x440 [ 4672.496643] ____fput+0x12/0x20 [ 4672.497919] task_work_run+0xc6/0x110 [ 4672.499487] exit_to_usermode_loop+0x1dd/0x1f0 [ 4672.501146] do_syscall_64+0x3d6/0x440 [ 4672.502594] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4672.504432] RIP: 0033:0x7ff54af45cd8 [ 4672.505630] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4672.512120] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4672.514656] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 4672.517555] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 4672.520282] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 4672.523180] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 4672.525467] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4676.055052] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 4676.059029] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4676.081382] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4676.086406] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4676.091616] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4676.096411] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4676.115343] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4676.122020] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 4676.129996] RDX: 000000000000439a RSI: 0000000000000003 RDI: ffff96848811cae0 [ 4676.135759] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 4676.144526] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 4676.148858] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4676.152474] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 4676.155821] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4676.158349] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 4676.161704] Call Trace: [ 4676.162868] [ 4676.163983] ? show_regs.cold.9+0x22/0x2f [ 4676.165664] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4676.167644] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4676.169565] ? watchdog+0x40/0x40 [ 4676.170991] ? hrtimer_interrupt+0x128/0x2c0 [ 4676.172656] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4676.174578] ? apic_timer_interrupt+0xf/0x20 [ 4676.176213] [ 4676.177026] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4676.179717] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4676.182323] do_raw_spin_lock+0x177/0x190 [ 4676.184309] _raw_spin_lock+0x21/0x30 [ 4676.186128] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4676.188733] obd_mod_free+0x46/0x140 [ptlrpc] [ 4676.190933] mdc_commit_open+0x91/0xb0 [mdc] [ 4676.192780] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4676.195292] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4676.197863] after_reply+0xe2c/0x1780 [ptlrpc] [ 4676.200149] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4676.202615] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4676.204686] ? do_wait_intr+0xf0/0xf0 [ 4676.207177] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4676.209204] kthread+0x1d1/0x200 [ 4676.219001] ? set_kthread_struct+0x70/0x70 [ 4676.220519] ret_from_fork+0x1f/0x30 [ 4679.142332] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2720s! [ 4679.161866] Showing busy workqueues and worker pools: [ 4679.173487] workqueue events: flags=0x0 [ 4679.185515] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4679.193471] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 4679.213569] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4679.223546] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 4679.240361] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4679.250986] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 4679.261991] workqueue events_long: flags=0x0 [ 4679.268937] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4679.276410] pending: lu_objects_flush [obdclass] [ 4679.283630] workqueue events_unbound: flags=0x2 [ 4679.293882] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4679.299686] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4679.312470] workqueue events_power_efficient: flags=0x80 [ 4679.318415] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4679.331772] pending: neigh_periodic_work [ 4679.340538] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4679.348642] pending: do_cache_clean, check_lifetime [ 4679.353510] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4679.363840] pending: neigh_periodic_work [ 4679.369858] workqueue writeback: flags=0x4a [ 4679.374984] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4679.385212] pending: wb_workfn, wb_workfn [ 4679.387148] workqueue xprtiod: flags=0xa [ 4679.389653] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4679.395364] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 4679.404482] workqueue ipv6_addrconf: flags=0x40008 [ 4679.415414] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4679.417702] pending: addrconf_verify_work [ 4679.431771] workqueue knbd0-recv: flags=0x1a [ 4679.433515] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4679.447704] in-flight: 657:recv_work [ 4679.450695] workqueue ptlrpc_pinger: flags=0x2 [ 4679.456745] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4679.465340] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 4679.468529] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2720s workers=4 idle: 20831 12642 28859 [ 4679.475351] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4669s workers=2 idle: 659 [ 4688.059014] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 4688.061472] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4688.078002] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4688.085434] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4688.096889] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4688.099815] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4688.101651] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4688.108622] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4688.114185] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 4688.117012] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 4688.123230] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 4688.178031] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 4688.181663] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 4688.187724] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 4688.202539] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4688.208009] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 4688.215978] Call Trace: [ 4688.217014] [ 4688.217804] ? show_regs.cold.9+0x22/0x2f [ 4688.220163] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4688.222353] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4688.224543] ? watchdog+0x40/0x40 [ 4688.226111] ? hrtimer_interrupt+0x128/0x2c0 [ 4688.228764] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4688.230599] ? apic_timer_interrupt+0xf/0x20 [ 4688.235169] [ 4688.237924] ? native_safe_halt+0x2e/0x50 [ 4688.242144] kvm_wait+0xcd/0xe0 [ 4688.243113] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4688.246085] do_raw_spin_lock+0x177/0x190 [ 4688.248231] _raw_spin_lock+0x21/0x30 [ 4688.250673] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4688.254491] process_one_work+0x2c8/0x700 [ 4688.256685] worker_thread+0x296/0x6e0 [ 4688.258917] ? rescuer_thread+0x570/0x570 [ 4688.260615] kthread+0x1d1/0x200 [ 4688.261746] ? set_kthread_struct+0x70/0x70 [ 4688.263555] ret_from_fork+0x1f/0x30 [ 4692.060029] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 4692.067967] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4692.132803] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4692.144520] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4692.152150] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4692.156887] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4692.176978] RSP: 0000:ffffaf9fc15234f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4692.182843] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 4692.188266] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 4692.194414] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4692.206835] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 4692.218446] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 4692.224801] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 4692.245780] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4692.252618] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 4692.261741] Call Trace: [ 4692.267055] [ 4692.276283] ? show_regs.cold.9+0x22/0x2f [ 4692.286455] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4692.294023] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4692.296768] ? watchdog+0x40/0x40 [ 4692.302927] ? hrtimer_interrupt+0x128/0x2c0 [ 4692.305985] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4692.310128] ? apic_timer_interrupt+0xf/0x20 [ 4692.315907] [ 4692.320473] ? native_safe_halt+0x2e/0x50 [ 4692.325027] kvm_wait+0xcd/0xe0 [ 4692.329114] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4692.337940] do_raw_spin_lock+0x177/0x190 [ 4692.343949] _raw_spin_lock+0x21/0x30 [ 4692.352130] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4692.361147] ? do_raw_spin_unlock+0x75/0x190 [ 4692.374138] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4692.378229] ? do_wait_intr+0xf0/0xf0 [ 4692.383645] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4692.386332] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4692.396167] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4692.411977] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4692.415258] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4692.427085] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4692.434170] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4692.439269] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4692.447064] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4692.460863] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4692.470007] lmv_intent_open+0x338/0x1150 [lmv] [ 4692.489712] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4692.503137] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4692.514298] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4692.520480] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4692.524722] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4692.529190] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4692.533642] ? do_raw_spin_unlock+0x75/0x190 [ 4692.540066] ? _raw_spin_unlock+0x12/0x30 [ 4692.547008] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4692.550203] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4692.554431] ll_atomic_open+0x36a/0xd70 [lustre] [ 4692.561483] lookup_open+0x356/0xb40 [ 4692.565324] do_last+0x313/0xfc0 [ 4692.569066] ? nd_jump_root+0xe5/0x160 [ 4692.574497] ? path_init+0x437/0x520 [ 4692.579247] path_openat+0xf7/0x500 [ 4692.581239] ? do_raw_spin_unlock+0x75/0x190 [ 4692.584399] do_filp_open+0x99/0x140 [ 4692.586732] ? getname_flags+0x6e/0x330 [ 4692.589609] ? __check_object_size+0xff/0x256 [ 4692.592226] ? do_raw_spin_unlock+0x75/0x190 [ 4692.594697] ? _raw_spin_unlock+0x12/0x30 [ 4692.597067] do_sys_openat2+0x2b4/0x410 [ 4692.599519] do_sys_open+0x73/0xa0 [ 4692.601582] __x64_sys_openat+0x24/0x30 [ 4692.604252] do_syscall_64+0xc1/0x440 [ 4692.606438] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4692.613958] RIP: 0033:0x7ff54af45332 [ 4692.619105] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4692.639013] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4692.645087] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 4692.652261] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 4692.661010] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 4692.669835] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 4692.680770] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 4700.058051] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [ 4700.060426] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4700.091299] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4700.097604] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4700.105682] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4700.109266] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4700.125914] RSP: 0018:ffffaf9fc1543ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4700.130549] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 4700.138311] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 4700.146644] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4700.151137] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 4700.155116] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 4700.158601] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 4700.166457] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4700.173722] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 4700.182375] Call Trace: [ 4700.184599] [ 4700.191340] ? show_regs.cold.9+0x22/0x2f [ 4700.197637] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4700.202630] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4700.209631] ? watchdog+0x40/0x40 [ 4700.213021] ? hrtimer_interrupt+0x128/0x2c0 [ 4700.217367] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4700.220392] ? apic_timer_interrupt+0xf/0x20 [ 4700.223087] [ 4700.223862] ? native_safe_halt+0x2e/0x50 [ 4700.229302] kvm_wait+0xcd/0xe0 [ 4700.230515] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4700.235458] do_raw_spin_lock+0x177/0x190 [ 4700.237586] _raw_spin_lock+0x21/0x30 [ 4700.238561] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4700.243767] ? do_raw_spin_unlock+0x75/0x190 [ 4700.248200] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4700.255619] ? do_wait_intr+0xf0/0xf0 [ 4700.259222] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4700.263526] mdc_close+0x53b/0xd70 [mdc] [ 4700.268416] lmv_close+0x18d/0x490 [lmv] [ 4700.273375] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 4700.280552] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4700.286475] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4700.295183] ll_md_close+0x274/0x800 [lustre] [ 4700.298512] ll_file_release+0x128/0xc50 [lustre] [ 4700.303245] __fput+0x10d/0x440 [ 4700.305307] ____fput+0x12/0x20 [ 4700.306664] task_work_run+0xc6/0x110 [ 4700.308464] exit_to_usermode_loop+0x1dd/0x1f0 [ 4700.311051] do_syscall_64+0x3d6/0x440 [ 4700.312968] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4700.315532] RIP: 0033:0x7ff54af45cd8 [ 4700.317602] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4700.326495] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4700.331122] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 4700.335581] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 4700.339006] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 4700.341260] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 4700.347818] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4704.055014] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 4704.065359] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4704.120730] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4704.134418] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4704.140403] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4704.146604] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4704.166503] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4704.182927] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 4704.195295] RDX: 0000000000000a40 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 4704.207756] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 4704.215589] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 4704.230467] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4704.239967] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 4704.244963] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4704.251404] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 4704.261923] Call Trace: [ 4704.263454] [ 4704.267261] ? show_regs.cold.9+0x22/0x2f [ 4704.275429] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4704.281188] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4704.287618] ? watchdog+0x40/0x40 [ 4704.292991] ? hrtimer_interrupt+0x128/0x2c0 [ 4704.300806] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4704.305425] ? apic_timer_interrupt+0xf/0x20 [ 4704.313463] [ 4704.316804] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4704.319481] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4704.337679] do_raw_spin_lock+0x177/0x190 [ 4704.341142] _raw_spin_lock+0x21/0x30 [ 4704.342956] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4704.347372] obd_mod_free+0x46/0x140 [ptlrpc] [ 4704.352377] mdc_commit_open+0x91/0xb0 [mdc] [ 4704.358000] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4704.364500] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4704.373215] after_reply+0xe2c/0x1780 [ptlrpc] [ 4704.379370] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4704.386446] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4704.391142] ? do_wait_intr+0xf0/0xf0 [ 4704.394551] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4704.400394] kthread+0x1d1/0x200 [ 4704.404676] ? set_kthread_struct+0x70/0x70 [ 4704.410826] ret_from_fork+0x1f/0x30 [ 4709.859568] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2750s! [ 4709.871450] Showing busy workqueues and worker pools: [ 4709.874229] workqueue events: flags=0x0 [ 4709.879993] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4709.895631] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 4709.910184] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4709.916492] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 4709.927181] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4709.931874] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 4709.935944] workqueue events_long: flags=0x0 [ 4709.937963] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4709.943867] pending: lu_objects_flush [obdclass] [ 4709.951410] workqueue events_unbound: flags=0x2 [ 4709.957373] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4709.964880] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4709.993420] workqueue events_power_efficient: flags=0x80 [ 4710.004335] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4710.008985] pending: neigh_periodic_work [ 4710.013748] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4710.017849] pending: do_cache_clean, check_lifetime [ 4710.024462] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4710.033351] pending: neigh_periodic_work [ 4710.035943] workqueue writeback: flags=0x4a [ 4710.040369] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4710.047421] pending: wb_workfn, wb_workfn [ 4710.056114] workqueue xprtiod: flags=0xa [ 4710.059521] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4710.064192] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 4710.071219] workqueue ipv6_addrconf: flags=0x40008 [ 4710.077175] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4710.086001] pending: addrconf_verify_work [ 4710.091832] workqueue knbd0-recv: flags=0x1a [ 4710.099188] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4710.104369] in-flight: 657:recv_work [ 4710.106838] workqueue ptlrpc_pinger: flags=0x2 [ 4710.112265] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4710.120228] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 4710.125210] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2751s workers=4 idle: 20831 12642 28859 [ 4710.132337] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4700s workers=2 idle: 659 [ 4713.919018] rcu: INFO: rcu_sched self-detected stall on CPU [ 4713.923821] rcu: 0-...!: (1440636 ticks this GP) idle=98a/1/0x4000000000000002 softirq=125154/125154 fqs=959 [ 4713.930406] (t=2760079 jiffies g=253585 q=3657) [ 4713.937889] rcu: rcu_sched kthread starved for 2755039 jiffies! g253585 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [ 4713.946877] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 4713.951240] rcu: RCU grace-period kthread stack dump: [ 4713.953094] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 4713.959526] Call Trace: [ 4713.960867] __schedule+0x351/0xcb0 [ 4713.962188] schedule+0xc0/0x180 [ 4713.963416] schedule_timeout+0xb4/0x190 [ 4713.966993] ? do_raw_spin_unlock+0x75/0x190 [ 4713.968994] ? __next_timer_interrupt+0x160/0x160 [ 4713.972168] ? prepare_to_swait_event+0xf3/0x140 [ 4713.974957] rcu_gp_kthread+0x8dd/0x15b0 [ 4713.977201] ? force_qs_rnp+0x2c0/0x2c0 [ 4713.978636] kthread+0x1d1/0x200 [ 4713.979766] ? set_kthread_struct+0x70/0x70 [ 4713.981179] ret_from_fork+0x1f/0x30 [ 4713.982069] rcu: Stack dump where RCU GP kthread last ran: [ 4713.984671] Sending NMI from CPU 0 to CPUs 1: [ 4713.986697] NMI backtrace for cpu 1 skipped: idling at native_safe_halt+0x2e/0x50 [ 4713.990091] NMI backtrace for cpu 0 [ 4714.001220] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4714.014247] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4714.017936] Call Trace: [ 4714.019791] [ 4714.021282] ? dump_stack+0xbb/0x10e [ 4714.031159] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 4714.034452] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 4714.040785] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 4714.043620] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 4714.047715] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 4714.050405] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 4714.053523] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 4714.056361] ? update_process_times+0x6d/0xc0 [ 4714.059785] ? tick_sched_handle.isra.18+0x27/0x90 [ 4714.062789] ? tick_sched_timer+0x53/0xb0 [ 4714.069368] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4714.071399] ? tick_sched_handle.isra.18+0x90/0x90 [ 4714.076497] ? hrtimer_interrupt+0x128/0x2c0 [ 4714.080562] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4714.082825] ? apic_timer_interrupt+0xf/0x20 [ 4714.085622] [ 4714.086790] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4714.093345] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4714.097855] ? do_raw_spin_lock+0x177/0x190 [ 4714.099416] ? _raw_spin_lock+0x21/0x30 [ 4714.101358] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4714.107428] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 4714.109396] ? mdc_commit_open+0x91/0xb0 [mdc] [ 4714.112331] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4714.115223] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4714.122239] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 4714.124965] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4714.126379] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4714.128429] ? do_wait_intr+0xf0/0xf0 [ 4714.131371] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4714.134593] ? kthread+0x1d1/0x200 [ 4714.135721] ? set_kthread_struct+0x70/0x70 [ 4714.136515] ? ret_from_fork+0x1f/0x30 [ 4716.059457] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 4716.067709] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4716.112572] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4716.129310] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4716.137337] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4716.143628] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4716.152248] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4716.172130] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 4716.177271] RAX: 0000000000007f87 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 4716.181563] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 4716.190262] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 4716.197159] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 4716.202324] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 4716.209337] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 4716.217998] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4716.223259] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 4716.228789] Call Trace: [ 4716.232314] [ 4716.234176] ? show_regs.cold.9+0x22/0x2f [ 4716.237815] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4716.241847] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4716.244756] ? watchdog+0x40/0x40 [ 4716.246781] ? hrtimer_interrupt+0x128/0x2c0 [ 4716.258551] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4716.268468] ? apic_timer_interrupt+0xf/0x20 [ 4716.272540] [ 4716.273602] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4716.277740] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4716.280862] do_raw_spin_lock+0x177/0x190 [ 4716.283604] _raw_spin_lock+0x21/0x30 [ 4716.288624] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4716.292079] process_one_work+0x2c8/0x700 [ 4716.294884] worker_thread+0x296/0x6e0 [ 4716.297464] ? rescuer_thread+0x570/0x570 [ 4716.302201] kthread+0x1d1/0x200 [ 4716.303764] ? set_kthread_struct+0x70/0x70 [ 4716.306287] ret_from_fork+0x1f/0x30 [ 4720.060463] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 4720.065586] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4720.146432] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4720.157596] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4720.161544] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4720.169380] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4720.181730] RSP: 0000:ffffaf9fc15234f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4720.186167] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 4720.191236] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 4720.198438] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4720.205918] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 4720.211765] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 4720.217535] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 4720.220305] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4720.225320] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 4720.232557] Call Trace: [ 4720.236002] [ 4720.245265] ? show_regs.cold.9+0x22/0x2f [ 4720.248413] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4720.255720] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4720.262165] ? watchdog+0x40/0x40 [ 4720.270056] ? hrtimer_interrupt+0x128/0x2c0 [ 4720.278379] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4720.286452] ? apic_timer_interrupt+0xf/0x20 [ 4720.293969] [ 4720.297707] ? native_safe_halt+0x2e/0x50 [ 4720.301435] kvm_wait+0xcd/0xe0 [ 4720.303449] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4720.310412] do_raw_spin_lock+0x177/0x190 [ 4720.315084] _raw_spin_lock+0x21/0x30 [ 4720.318398] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4720.324175] ? do_raw_spin_unlock+0x75/0x190 [ 4720.325818] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4720.331210] ? do_wait_intr+0xf0/0xf0 [ 4720.334345] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4720.337680] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4720.342091] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4720.344524] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4720.346513] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4720.349207] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4720.352323] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4720.357264] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4720.361598] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4720.365460] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4720.368757] lmv_intent_open+0x338/0x1150 [lmv] [ 4720.371730] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4720.374438] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4720.377865] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4720.380255] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4720.382714] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4720.385763] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4720.388548] ? do_raw_spin_unlock+0x75/0x190 [ 4720.391366] ? _raw_spin_unlock+0x12/0x30 [ 4720.393868] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4720.396545] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4720.399206] ll_atomic_open+0x36a/0xd70 [lustre] [ 4720.401928] lookup_open+0x356/0xb40 [ 4720.404150] do_last+0x313/0xfc0 [ 4720.406343] ? nd_jump_root+0xe5/0x160 [ 4720.409056] ? path_init+0x437/0x520 [ 4720.411113] path_openat+0xf7/0x500 [ 4720.412956] ? do_raw_spin_unlock+0x75/0x190 [ 4720.415149] do_filp_open+0x99/0x140 [ 4720.417221] ? getname_flags+0x6e/0x330 [ 4720.419768] ? __check_object_size+0xff/0x256 [ 4720.422448] ? do_raw_spin_unlock+0x75/0x190 [ 4720.424342] ? _raw_spin_unlock+0x12/0x30 [ 4720.426295] do_sys_openat2+0x2b4/0x410 [ 4720.428526] do_sys_open+0x73/0xa0 [ 4720.430454] __x64_sys_openat+0x24/0x30 [ 4720.432734] do_syscall_64+0xc1/0x440 [ 4720.434929] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4720.437747] RIP: 0033:0x7ff54af45332 [ 4720.439723] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4720.450346] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4720.454008] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 4720.458238] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 4720.462363] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 4720.466591] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 4720.470948] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 4740.055019] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 4740.059671] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 4740.069209] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul [ 4740.081463] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4740.081548] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4740.081556] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4740.081559] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4740.081575] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4740.081579] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4740.081584] RAX: 0000000000007d93 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 4740.081586] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 4740.081589] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4740.081591] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 4740.081607] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 4740.081611] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 4740.081614] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4740.081616] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 4740.081624] Call Trace: [ 4740.081629] [ 4740.081635] ? show_regs.cold.9+0x22/0x2f [ 4740.081646] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4740.081652] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4740.081657] ? watchdog+0x40/0x40 [ 4740.081662] ? hrtimer_interrupt+0x128/0x2c0 [ 4740.081665] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4740.081673] ? apic_timer_interrupt+0xf/0x20 [ 4740.081679] [ 4740.081682] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4740.081689] do_raw_spin_lock+0x177/0x190 [ 4740.081695] _raw_spin_lock+0x21/0x30 [ 4740.081700] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4740.082016] ? do_raw_spin_unlock+0x75/0x190 [ 4740.082023] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4740.082267] ? do_wait_intr+0xf0/0xf0 [ 4740.082275] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4740.082711] mdc_close+0x53b/0xd70 [mdc] [ 4740.082755] lmv_close+0x18d/0x490 [lmv] [ 4740.082785] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 4740.083296] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4740.083502] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4740.083589] ll_md_close+0x274/0x800 [lustre] [ 4740.083792] ll_file_release+0x128/0xc50 [lustre] [ 4740.084015] __fput+0x10d/0x440 [ 4740.084024] ____fput+0x12/0x20 [ 4740.084030] task_work_run+0xc6/0x110 [ 4740.084038] exit_to_usermode_loop+0x1dd/0x1f0 [ 4740.084046] do_syscall_64+0x3d6/0x440 [ 4740.084106] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4740.084115] RIP: 0033:0x7ff54af45cd8 [ 4740.084122] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4740.084127] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4740.084132] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 4740.084134] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 4740.084137] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 4740.084139] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 4740.084142] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4740.417126] crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4740.424298] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4740.430883] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4740.436416] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4740.440740] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4740.455172] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4740.461229] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 4740.465405] RDX: 0000000000000c76 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 4740.467419] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 4740.472038] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 4740.479221] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4740.485223] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 4740.494809] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4740.503222] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 4740.516512] Call Trace: [ 4740.517818] [ 4740.519485] ? show_regs.cold.9+0x22/0x2f [ 4740.528055] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4740.530071] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4740.532113] ? watchdog+0x40/0x40 [ 4740.533540] ? hrtimer_interrupt+0x128/0x2c0 [ 4740.535217] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4740.536566] ? apic_timer_interrupt+0xf/0x20 [ 4740.538469] [ 4740.541116] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4740.543385] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4740.551980] do_raw_spin_lock+0x177/0x190 [ 4740.555204] _raw_spin_lock+0x21/0x30 [ 4740.561833] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4740.565047] obd_mod_free+0x46/0x140 [ptlrpc] [ 4740.567477] mdc_commit_open+0x91/0xb0 [mdc] [ 4740.569815] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4740.574052] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4740.583557] after_reply+0xe2c/0x1780 [ptlrpc] [ 4740.590289] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4740.592914] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4740.597522] ? do_wait_intr+0xf0/0xf0 [ 4740.604392] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4740.608805] kthread+0x1d1/0x200 [ 4740.610694] ? set_kthread_struct+0x70/0x70 [ 4740.612836] ret_from_fork+0x1f/0x30 [ 4740.616388] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2781s! [ 4740.620262] Showing busy workqueues and worker pools: [ 4740.622667] workqueue events: flags=0x0 [ 4740.624202] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4740.634562] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 4740.645170] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4740.649030] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 4740.658879] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4740.662323] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 4740.676556] workqueue events_long: flags=0x0 [ 4740.680871] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4740.685871] pending: lu_objects_flush [obdclass] [ 4740.691503] workqueue events_unbound: flags=0x2 [ 4740.694385] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4740.699248] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4740.708106] workqueue events_power_efficient: flags=0x80 [ 4740.714084] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4740.723428] pending: neigh_periodic_work [ 4740.731802] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4740.735520] pending: do_cache_clean, check_lifetime [ 4740.740425] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4740.748298] pending: neigh_periodic_work [ 4740.754398] workqueue writeback: flags=0x4a [ 4740.757143] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4740.759584] pending: wb_workfn, wb_workfn [ 4740.762167] workqueue xprtiod: flags=0xa [ 4740.765526] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4740.770190] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 4740.774800] workqueue ipv6_addrconf: flags=0x40008 [ 4740.780426] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4740.788598] pending: addrconf_verify_work [ 4740.789835] workqueue knbd0-recv: flags=0x1a [ 4740.791120] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4740.796175] in-flight: 657:recv_work [ 4740.800370] workqueue ptlrpc_pinger: flags=0x2 [ 4740.803792] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4740.807602] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 4740.811424] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2781s workers=4 idle: 20831 12642 28859 [ 4740.821418] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4731s workers=2 idle: 659 [ 4744.061861] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 4744.072598] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4744.122726] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4744.134814] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4744.139890] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4744.144655] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4744.151430] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4744.167944] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4744.177559] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 4744.181920] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 4744.193863] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 4744.206973] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 4744.211952] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 4744.213975] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 4744.216030] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4744.217473] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 4744.220292] Call Trace: [ 4744.221195] [ 4744.222521] ? show_regs.cold.9+0x22/0x2f [ 4744.224066] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4744.225936] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4744.227506] ? watchdog+0x40/0x40 [ 4744.228821] ? hrtimer_interrupt+0x128/0x2c0 [ 4744.232111] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4744.234939] ? apic_timer_interrupt+0xf/0x20 [ 4744.236334] [ 4744.236807] ? native_safe_halt+0x2e/0x50 [ 4744.239455] kvm_wait+0xcd/0xe0 [ 4744.242603] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4744.244381] do_raw_spin_lock+0x177/0x190 [ 4744.245851] _raw_spin_lock+0x21/0x30 [ 4744.247000] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4744.249105] process_one_work+0x2c8/0x700 [ 4744.251933] worker_thread+0x296/0x6e0 [ 4744.253733] ? rescuer_thread+0x570/0x570 [ 4744.255740] kthread+0x1d1/0x200 [ 4744.257760] ? set_kthread_struct+0x70/0x70 [ 4744.259994] ret_from_fork+0x1f/0x30 [ 4748.061590] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 4748.067895] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4748.107636] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4748.123318] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4748.134786] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4748.140678] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4748.159250] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4748.166322] RAX: 0000000000000692 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 4748.171913] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 4748.177419] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4748.192985] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 4748.197339] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 4748.206538] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 4748.218296] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4748.229937] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 4748.237156] Call Trace: [ 4748.239180] [ 4748.245321] ? show_regs.cold.9+0x22/0x2f [ 4748.251299] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4748.256773] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4748.259457] ? watchdog+0x40/0x40 [ 4748.261584] ? hrtimer_interrupt+0x128/0x2c0 [ 4748.268832] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4748.271894] ? apic_timer_interrupt+0xf/0x20 [ 4748.276454] [ 4748.279398] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4748.293467] do_raw_spin_lock+0x177/0x190 [ 4748.300125] _raw_spin_lock+0x21/0x30 [ 4748.303508] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4748.312273] ? do_raw_spin_unlock+0x75/0x190 [ 4748.315683] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4748.321729] ? do_wait_intr+0xf0/0xf0 [ 4748.331826] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4748.336676] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4748.357478] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4748.359588] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4748.365933] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4748.367793] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4748.378100] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4748.379689] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4748.390444] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4748.395120] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4748.400476] lmv_intent_open+0x338/0x1150 [lmv] [ 4748.403792] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4748.408933] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4748.411765] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4748.416781] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4748.419067] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4748.420967] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4748.431532] ? do_raw_spin_unlock+0x75/0x190 [ 4748.439290] ? _raw_spin_unlock+0x12/0x30 [ 4748.440795] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4748.444680] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4748.447416] ll_atomic_open+0x36a/0xd70 [lustre] [ 4748.451616] lookup_open+0x356/0xb40 [ 4748.453778] do_last+0x313/0xfc0 [ 4748.455814] ? nd_jump_root+0xe5/0x160 [ 4748.457779] ? path_init+0x437/0x520 [ 4748.461986] path_openat+0xf7/0x500 [ 4748.463189] ? do_raw_spin_unlock+0x75/0x190 [ 4748.467149] do_filp_open+0x99/0x140 [ 4748.468308] ? getname_flags+0x6e/0x330 [ 4748.469640] ? __check_object_size+0xff/0x256 [ 4748.476584] ? do_raw_spin_unlock+0x75/0x190 [ 4748.478714] ? _raw_spin_unlock+0x12/0x30 [ 4748.481389] do_sys_openat2+0x2b4/0x410 [ 4748.488480] do_sys_open+0x73/0xa0 [ 4748.491464] __x64_sys_openat+0x24/0x30 [ 4748.494544] do_syscall_64+0xc1/0x440 [ 4748.497668] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4748.501457] RIP: 0033:0x7ff54af45332 [ 4748.505636] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4748.526160] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4748.532492] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 4748.543087] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 4748.552568] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 4748.557358] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 4748.574044] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 4768.055216] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 4768.058171] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 4768.062649] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) [ 4768.070519] Modules linked in: [ 4768.071010] mdd(O) [ 4768.077716] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4768.077799] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4768.077806] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4768.077809] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4768.077823] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4768.077827] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4768.077831] RAX: 0000000000004e34 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 4768.077834] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 4768.077836] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4768.077838] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 4768.077840] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 4768.077844] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 4768.077847] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4768.077849] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 4768.077857] Call Trace: [ 4768.077862] [ 4768.077867] ? show_regs.cold.9+0x22/0x2f [ 4768.077875] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4768.077881] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4768.077886] ? watchdog+0x40/0x40 [ 4768.077890] ? hrtimer_interrupt+0x128/0x2c0 [ 4768.077895] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4768.077903] ? apic_timer_interrupt+0xf/0x20 [ 4768.077908] [ 4768.077911] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4768.077917] do_raw_spin_lock+0x177/0x190 [ 4768.077922] _raw_spin_lock+0x21/0x30 [ 4768.077926] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4768.078225] ? do_raw_spin_unlock+0x75/0x190 [ 4768.078231] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4768.078435] ? do_wait_intr+0xf0/0xf0 [ 4768.078444] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4768.078633] mdc_close+0x53b/0xd70 [mdc] [ 4768.078674] lmv_close+0x18d/0x490 [lmv] [ 4768.078706] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 4768.078881] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4768.078994] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4768.079076] ll_md_close+0x274/0x800 [lustre] [ 4768.080622] ll_file_release+0x128/0xc50 [lustre] [ 4768.080744] __fput+0x10d/0x440 [ 4768.080753] ____fput+0x12/0x20 [ 4768.080757] task_work_run+0xc6/0x110 [ 4768.080765] exit_to_usermode_loop+0x1dd/0x1f0 [ 4768.080772] do_syscall_64+0x3d6/0x440 [ 4768.080776] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4768.080786] RIP: 0033:0x7ff54af45cd8 [ 4768.080793] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4768.080797] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4768.080802] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 4768.080805] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 4768.080807] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 4768.080810] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 4768.080812] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4768.347857] mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4768.367297] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4768.372065] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4768.376836] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4768.380214] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4768.391116] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4768.394438] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 4768.397996] RDX: 0000000000006ded RSI: 0000000000000003 RDI: ffff96848811cae0 [ 4768.403916] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 4768.408151] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 4768.411266] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4768.414186] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 4768.417724] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4768.420271] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 4768.423107] Call Trace: [ 4768.424159] [ 4768.425069] ? show_regs.cold.9+0x22/0x2f [ 4768.426895] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4768.429292] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4768.431357] ? watchdog+0x40/0x40 [ 4768.432990] ? hrtimer_interrupt+0x128/0x2c0 [ 4768.434991] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4768.436944] ? apic_timer_interrupt+0xf/0x20 [ 4768.438950] [ 4768.440114] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4768.443117] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4768.445830] do_raw_spin_lock+0x177/0x190 [ 4768.447167] _raw_spin_lock+0x21/0x30 [ 4768.453594] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4768.456869] obd_mod_free+0x46/0x140 [ptlrpc] [ 4768.463134] mdc_commit_open+0x91/0xb0 [mdc] [ 4768.469218] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4768.472997] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4768.476173] after_reply+0xe2c/0x1780 [ptlrpc] [ 4768.479736] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4768.481969] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4768.486056] ? do_wait_intr+0xf0/0xf0 [ 4768.490354] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4768.493592] kthread+0x1d1/0x200 [ 4768.496571] ? set_kthread_struct+0x70/0x70 [ 4768.499920] ret_from_fork+0x1f/0x30 [ 4771.295993] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2812s! [ 4771.299954] Showing busy workqueues and worker pools: [ 4771.303456] workqueue events: flags=0x0 [ 4771.307871] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4771.315724] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 4771.323732] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4771.333300] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 4771.346339] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4771.351638] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 4771.355463] workqueue events_long: flags=0x0 [ 4771.356835] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4771.359731] pending: lu_objects_flush [obdclass] [ 4771.362851] workqueue events_unbound: flags=0x2 [ 4771.367712] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4771.370546] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4771.378916] workqueue events_power_efficient: flags=0x80 [ 4771.382235] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4771.389090] pending: neigh_periodic_work [ 4771.391260] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4771.394934] pending: do_cache_clean, check_lifetime [ 4771.399199] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4771.404943] pending: neigh_periodic_work [ 4771.412749] workqueue writeback: flags=0x4a [ 4771.415748] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4771.423360] pending: wb_workfn, wb_workfn [ 4771.428165] workqueue xprtiod: flags=0xa [ 4771.433775] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4771.438299] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 4771.445528] workqueue ipv6_addrconf: flags=0x40008 [ 4771.452433] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4771.459061] pending: addrconf_verify_work [ 4771.467743] workqueue knbd0-recv: flags=0x1a [ 4771.469490] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4771.472118] in-flight: 657:recv_work [ 4771.477201] workqueue ptlrpc_pinger: flags=0x2 [ 4771.481311] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4771.487308] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 4771.498270] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2812s workers=4 idle: 20831 12642 28859 [ 4771.509920] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4761s workers=2 idle: 659 [ 4772.059100] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 4772.064504] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4772.087359] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4772.094560] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4772.101130] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4772.104938] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4772.108228] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4772.116866] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4772.120997] RAX: 000000000000404a RBX: ffff96848811cae0 RCX: 0000000000000001 [ 4772.125169] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 4772.128434] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 4772.131003] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 4772.133942] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 4772.136800] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 4772.139951] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4772.142534] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 4772.145378] Call Trace: [ 4772.146243] [ 4772.146990] ? show_regs.cold.9+0x22/0x2f [ 4772.148826] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4772.150374] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4772.152437] ? watchdog+0x40/0x40 [ 4772.153923] ? hrtimer_interrupt+0x128/0x2c0 [ 4772.155887] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4772.157932] ? apic_timer_interrupt+0xf/0x20 [ 4772.159773] [ 4772.160815] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4772.162909] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4772.165412] do_raw_spin_lock+0x177/0x190 [ 4772.167129] _raw_spin_lock+0x21/0x30 [ 4772.168777] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4772.171016] process_one_work+0x2c8/0x700 [ 4772.172728] worker_thread+0x296/0x6e0 [ 4772.174273] ? rescuer_thread+0x570/0x570 [ 4772.175847] kthread+0x1d1/0x200 [ 4772.177192] ? set_kthread_struct+0x70/0x70 [ 4772.178896] ret_from_fork+0x1f/0x30 [ 4776.062187] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [ 4776.071256] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4776.106495] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4776.115403] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4776.125616] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4776.127714] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4776.145367] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4776.154347] RAX: 0000000000000288 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 4776.157343] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 4776.160984] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4776.165386] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 4776.168370] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 4776.171362] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 4776.174496] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4776.177239] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 4776.180336] Call Trace: [ 4776.183396] [ 4776.186569] ? show_regs.cold.9+0x22/0x2f [ 4776.189889] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4776.195516] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4776.198722] ? watchdog+0x40/0x40 [ 4776.200471] ? hrtimer_interrupt+0x128/0x2c0 [ 4776.201972] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4776.205397] ? apic_timer_interrupt+0xf/0x20 [ 4776.207746] [ 4776.208776] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4776.211478] do_raw_spin_lock+0x177/0x190 [ 4776.215074] _raw_spin_lock+0x21/0x30 [ 4776.216897] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4776.220744] ? do_raw_spin_unlock+0x75/0x190 [ 4776.221839] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4776.224187] ? do_wait_intr+0xf0/0xf0 [ 4776.226022] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4776.230454] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4776.233438] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4776.236258] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4776.239613] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4776.243308] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4776.245410] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4776.247329] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4776.249431] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4776.254716] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4776.258202] lmv_intent_open+0x338/0x1150 [lmv] [ 4776.260778] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4776.264385] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4776.267442] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4776.269459] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4776.271328] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4776.273524] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4776.275405] ? do_raw_spin_unlock+0x75/0x190 [ 4776.277191] ? _raw_spin_unlock+0x12/0x30 [ 4776.278646] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4776.280853] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4776.286580] ll_atomic_open+0x36a/0xd70 [lustre] [ 4776.288699] lookup_open+0x356/0xb40 [ 4776.290686] do_last+0x313/0xfc0 [ 4776.292378] ? nd_jump_root+0xe5/0x160 [ 4776.294355] ? path_init+0x437/0x520 [ 4776.296138] path_openat+0xf7/0x500 [ 4776.297727] ? do_raw_spin_unlock+0x75/0x190 [ 4776.299619] do_filp_open+0x99/0x140 [ 4776.301592] ? getname_flags+0x6e/0x330 [ 4776.303141] ? __check_object_size+0xff/0x256 [ 4776.305495] ? do_raw_spin_unlock+0x75/0x190 [ 4776.308322] ? _raw_spin_unlock+0x12/0x30 [ 4776.310775] do_sys_openat2+0x2b4/0x410 [ 4776.312578] do_sys_open+0x73/0xa0 [ 4776.314334] __x64_sys_openat+0x24/0x30 [ 4776.316297] do_syscall_64+0xc1/0x440 [ 4776.319230] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4776.322203] RIP: 0033:0x7ff54af45332 [ 4776.325552] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4776.336723] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4776.339404] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 4776.341730] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 4776.344542] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 4776.347706] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 4776.350844] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 4796.055058] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 4796.058011] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 4796.059659] Modules linked in: [ 4796.073133] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4796.073328] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4796.073335] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4796.073338] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4796.073362] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4796.073366] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4796.073371] RAX: 0000000000001016 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 4796.073373] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 4796.073375] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4796.073377] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 4796.073381] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 4796.073384] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 4796.073387] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4796.073389] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 4796.073397] Call Trace: [ 4796.073413] [ 4796.073421] ? show_regs.cold.9+0x22/0x2f [ 4796.073430] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4796.073437] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4796.073441] ? watchdog+0x40/0x40 [ 4796.073445] ? hrtimer_interrupt+0x128/0x2c0 [ 4796.073449] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4796.073457] ? apic_timer_interrupt+0xf/0x20 [ 4796.073462] [ 4796.073466] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4796.073472] do_raw_spin_lock+0x177/0x190 [ 4796.073477] _raw_spin_lock+0x21/0x30 [ 4796.073482] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4796.073802] ? do_raw_spin_unlock+0x75/0x190 [ 4796.073807] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4796.074050] ? do_wait_intr+0xf0/0xf0 [ 4796.074058] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4796.074261] mdc_close+0x53b/0xd70 [mdc] [ 4796.074301] lmv_close+0x18d/0x490 [lmv] [ 4796.074329] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 4796.074497] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4796.076664] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4796.077030] ll_md_close+0x274/0x800 [lustre] [ 4796.077128] ll_file_release+0x128/0xc50 [lustre] [ 4796.077793] __fput+0x10d/0x440 [ 4796.077805] ____fput+0x12/0x20 [ 4796.077810] task_work_run+0xc6/0x110 [ 4796.077816] exit_to_usermode_loop+0x1dd/0x1f0 [ 4796.077822] do_syscall_64+0x3d6/0x440 [ 4796.077826] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4796.077834] RIP: 0033:0x7ff54af45cd8 [ 4796.077861] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4796.077865] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4796.077870] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 4796.077872] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 4796.077874] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 4796.077877] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 4796.077879] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4796.498752] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4796.536012] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4796.547619] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4796.552559] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4796.565848] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4796.578164] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4796.585071] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 4796.596582] RDX: 000000000000224a RSI: 0000000000000003 RDI: ffff96848811cae0 [ 4796.604151] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 4796.606819] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 4796.621790] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4796.630089] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 4796.640119] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4796.647644] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 4796.661499] Call Trace: [ 4796.662233] [ 4796.662837] ? show_regs.cold.9+0x22/0x2f [ 4796.673157] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4796.680959] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4796.682007] ? watchdog+0x40/0x40 [ 4796.684564] ? hrtimer_interrupt+0x128/0x2c0 [ 4796.686662] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4796.689606] ? apic_timer_interrupt+0xf/0x20 [ 4796.695352] [ 4796.695855] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4796.697074] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4796.704891] do_raw_spin_lock+0x177/0x190 [ 4796.705823] _raw_spin_lock+0x21/0x30 [ 4796.706684] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4796.711910] obd_mod_free+0x46/0x140 [ptlrpc] [ 4796.715123] mdc_commit_open+0x91/0xb0 [mdc] [ 4796.717035] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4796.718868] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4796.722721] after_reply+0xe2c/0x1780 [ptlrpc] [ 4796.726528] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4796.728298] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4796.737875] ? do_wait_intr+0xf0/0xf0 [ 4796.742411] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4796.745508] kthread+0x1d1/0x200 [ 4796.746746] ? set_kthread_struct+0x70/0x70 [ 4796.749574] ret_from_fork+0x1f/0x30 [ 4800.065549] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 4800.077387] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4800.110593] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4800.120862] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4800.127173] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4800.134539] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4800.143884] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4800.174331] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4800.189228] RAX: 0000000000003b85 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 4800.204351] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 4800.220901] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 4800.225734] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 4800.234487] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 4800.241685] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 4800.252642] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4800.260976] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 4800.268106] Call Trace: [ 4800.270280] [ 4800.273190] ? show_regs.cold.9+0x22/0x2f [ 4800.274598] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4800.279910] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4800.287369] ? watchdog+0x40/0x40 [ 4800.288613] ? hrtimer_interrupt+0x128/0x2c0 [ 4800.292393] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4800.298395] ? apic_timer_interrupt+0xf/0x20 [ 4800.303300] [ 4800.305813] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4800.310088] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4800.315979] do_raw_spin_lock+0x177/0x190 [ 4800.322154] _raw_spin_lock+0x21/0x30 [ 4800.325664] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4800.330045] process_one_work+0x2c8/0x700 [ 4800.334310] worker_thread+0x296/0x6e0 [ 4800.337839] ? rescuer_thread+0x570/0x570 [ 4800.339284] kthread+0x1d1/0x200 [ 4800.342394] ? set_kthread_struct+0x70/0x70 [ 4800.344509] ret_from_fork+0x1f/0x30 [ 4802.016242] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2843s! [ 4802.023684] Showing busy workqueues and worker pools: [ 4802.029617] workqueue events: flags=0x0 [ 4802.033638] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4802.042623] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 4802.053705] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4802.056007] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 4802.075128] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4802.086749] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 4802.096468] workqueue events_long: flags=0x0 [ 4802.098619] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4802.103518] pending: lu_objects_flush [obdclass] [ 4802.114317] workqueue events_unbound: flags=0x2 [ 4802.118497] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4802.124679] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4802.136477] workqueue events_power_efficient: flags=0x80 [ 4802.144550] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4802.148481] pending: neigh_periodic_work [ 4802.151105] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4802.154237] pending: do_cache_clean, check_lifetime [ 4802.158214] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4802.165027] pending: neigh_periodic_work [ 4802.167619] workqueue writeback: flags=0x4a [ 4802.171976] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4802.177539] pending: wb_workfn, wb_workfn [ 4802.179750] workqueue xprtiod: flags=0xa [ 4802.181968] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4802.185485] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 4802.189590] workqueue ipv6_addrconf: flags=0x40008 [ 4802.191893] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4802.195193] pending: addrconf_verify_work [ 4802.197242] workqueue knbd0-recv: flags=0x1a [ 4802.199447] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4802.203362] in-flight: 657:recv_work [ 4802.206464] workqueue ptlrpc_pinger: flags=0x2 [ 4802.211579] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4802.214830] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 4802.217659] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2843s workers=4 idle: 20831 12642 28859 [ 4802.224907] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4792s workers=2 idle: 659 [ 4804.062212] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [ 4804.066814] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4804.130275] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4804.139054] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4804.146522] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4804.149855] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4804.167221] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4804.169132] RAX: 0000000000006a86 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 4804.177153] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 4804.183019] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4804.186205] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 4804.189088] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 4804.193026] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 4804.199122] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4804.203156] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 4804.214902] Call Trace: [ 4804.216255] [ 4804.217253] ? show_regs.cold.9+0x22/0x2f [ 4804.221226] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4804.226894] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4804.230282] ? watchdog+0x40/0x40 [ 4804.235903] ? hrtimer_interrupt+0x128/0x2c0 [ 4804.248321] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4804.255178] ? apic_timer_interrupt+0xf/0x20 [ 4804.259260] [ 4804.259994] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4804.261326] do_raw_spin_lock+0x177/0x190 [ 4804.262972] _raw_spin_lock+0x21/0x30 [ 4804.264086] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4804.266821] ? do_raw_spin_unlock+0x75/0x190 [ 4804.268863] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4804.272056] ? do_wait_intr+0xf0/0xf0 [ 4804.274060] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4804.278281] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4804.281587] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4804.287396] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4804.293425] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4804.304393] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4804.313364] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4804.317389] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4804.324067] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4804.331706] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4804.337144] lmv_intent_open+0x338/0x1150 [lmv] [ 4804.340598] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4804.342583] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4804.352016] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4804.359136] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4804.360853] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4804.368919] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4804.371232] ? do_raw_spin_unlock+0x75/0x190 [ 4804.373843] ? _raw_spin_unlock+0x12/0x30 [ 4804.377471] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4804.382055] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4804.387991] ll_atomic_open+0x36a/0xd70 [lustre] [ 4804.389745] lookup_open+0x356/0xb40 [ 4804.393736] do_last+0x313/0xfc0 [ 4804.394801] ? nd_jump_root+0xe5/0x160 [ 4804.400841] ? path_init+0x437/0x520 [ 4804.407235] path_openat+0xf7/0x500 [ 4804.410358] ? do_raw_spin_unlock+0x75/0x190 [ 4804.413742] do_filp_open+0x99/0x140 [ 4804.420542] ? getname_flags+0x6e/0x330 [ 4804.422594] ? __check_object_size+0xff/0x256 [ 4804.426792] ? do_raw_spin_unlock+0x75/0x190 [ 4804.435936] ? _raw_spin_unlock+0x12/0x30 [ 4804.444824] do_sys_openat2+0x2b4/0x410 [ 4804.446043] do_sys_open+0x73/0xa0 [ 4804.447171] __x64_sys_openat+0x24/0x30 [ 4804.450403] do_syscall_64+0xc1/0x440 [ 4804.454260] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4804.457068] RIP: 0033:0x7ff54af45332 [ 4804.458589] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4804.471419] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4804.474951] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 4804.484177] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 4804.492393] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 4804.496192] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 4804.499636] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 4824.058549] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 4824.060899] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4824.092325] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4824.097113] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4824.100970] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4824.105575] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4824.137033] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4824.140761] RAX: 0000000000005d8f RBX: ffff96848811cae0 RCX: 0000000000000001 [ 4824.151180] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 4824.164693] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4824.178879] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 4824.191849] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 4824.205456] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 4824.222148] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4824.226578] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 4824.240057] Call Trace: [ 4824.251140] [ 4824.251931] ? show_regs.cold.9+0x22/0x2f [ 4824.253431] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4824.270387] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4824.275651] ? watchdog+0x40/0x40 [ 4824.282988] ? hrtimer_interrupt+0x128/0x2c0 [ 4824.288404] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4824.293521] ? apic_timer_interrupt+0xf/0x20 [ 4824.296758] [ 4824.297955] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4824.304538] do_raw_spin_lock+0x177/0x190 [ 4824.314036] _raw_spin_lock+0x21/0x30 [ 4824.316268] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4824.322957] ? do_raw_spin_unlock+0x75/0x190 [ 4824.329371] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4824.335590] ? do_wait_intr+0xf0/0xf0 [ 4824.340408] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4824.348210] mdc_close+0x53b/0xd70 [mdc] [ 4824.351261] lmv_close+0x18d/0x490 [lmv] [ 4824.360194] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 4824.367730] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4824.370580] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4824.380759] ll_md_close+0x274/0x800 [lustre] [ 4824.388482] ll_file_release+0x128/0xc50 [lustre] [ 4824.393705] __fput+0x10d/0x440 [ 4824.401164] ____fput+0x12/0x20 [ 4824.402210] task_work_run+0xc6/0x110 [ 4824.410155] exit_to_usermode_loop+0x1dd/0x1f0 [ 4824.413983] do_syscall_64+0x3d6/0x440 [ 4824.419584] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4824.425814] RIP: 0033:0x7ff54af45cd8 [ 4824.430323] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4824.455708] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4824.461872] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 4824.472057] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 4824.482978] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 4824.487382] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 4824.493916] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4828.056346] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 4828.060158] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 4828.076891] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4828.076973] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4828.076981] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4828.076984] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4828.076998] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4828.077002] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4828.077007] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 4828.077009] RDX: 0000000000006ab2 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 4828.077013] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 4828.077015] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 4828.077018] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4828.077022] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 4828.077025] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4828.077027] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 4828.077036] Call Trace: [ 4828.077042] [ 4828.077048] ? show_regs.cold.9+0x22/0x2f [ 4828.077259] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4828.077266] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4828.077271] ? watchdog+0x40/0x40 [ 4828.077275] ? hrtimer_interrupt+0x128/0x2c0 [ 4828.077279] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4828.077288] ? apic_timer_interrupt+0xf/0x20 [ 4828.077293] [ 4828.077296] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4828.077301] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4828.077306] do_raw_spin_lock+0x177/0x190 [ 4828.077310] _raw_spin_lock+0x21/0x30 [ 4828.077316] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4828.077872] obd_mod_free+0x46/0x140 [ptlrpc] [ 4828.078267] mdc_commit_open+0x91/0xb0 [mdc] [ 4828.078308] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4828.094048] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4828.097689] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4828.097700] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4828.097707] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4828.097971] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4828.097981] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4828.097986] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4828.097990] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 4828.097992] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 4828.097994] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 4828.097996] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 4828.097999] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 4828.098012] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 4828.098016] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4828.098018] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 4828.098026] Call Trace: [ 4828.098032] [ 4828.098038] ? show_regs.cold.9+0x22/0x2f [ 4828.098047] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4828.098054] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4828.098059] ? watchdog+0x40/0x40 [ 4828.098063] ? hrtimer_interrupt+0x128/0x2c0 [ 4828.098066] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4828.098073] ? apic_timer_interrupt+0xf/0x20 [ 4828.098077] [ 4828.098079] ? native_safe_halt+0x2e/0x50 [ 4828.098083] kvm_wait+0xcd/0xe0 [ 4828.098090] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4828.098097] do_raw_spin_lock+0x177/0x190 [ 4828.098102] _raw_spin_lock+0x21/0x30 [ 4828.098106] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4828.144771] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4828.145049] after_reply+0xe2c/0x1780 [ptlrpc] [ 4828.145265] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4828.145476] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4828.145689] ? do_wait_intr+0xf0/0xf0 [ 4828.145700] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4828.145916] kthread+0x1d1/0x200 [ 4828.145923] ? set_kthread_struct+0x70/0x70 [ 4828.145927] ret_from_fork+0x1f/0x30 [ 4828.735257] process_one_work+0x2c8/0x700 [ 4828.740370] worker_thread+0x296/0x6e0 [ 4828.746822] ? rescuer_thread+0x570/0x570 [ 4828.753548] kthread+0x1d1/0x200 [ 4828.755555] ? set_kthread_struct+0x70/0x70 [ 4828.760407] ret_from_fork+0x1f/0x30 [ 4832.060211] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [ 4832.068941] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4832.115939] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4832.121289] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4832.125107] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4832.128093] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4832.136309] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4832.139644] RAX: 0000000000001732 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 4832.142274] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 4832.145718] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4832.148706] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 4832.151939] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 4832.154684] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 4832.157340] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4832.161248] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 4832.165506] Call Trace: [ 4832.167322] [ 4832.168135] ? show_regs.cold.9+0x22/0x2f [ 4832.170012] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4832.174114] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4832.177993] ? watchdog+0x40/0x40 [ 4832.180298] ? hrtimer_interrupt+0x128/0x2c0 [ 4832.184504] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4832.186317] ? apic_timer_interrupt+0xf/0x20 [ 4832.188097] [ 4832.188693] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4832.190321] do_raw_spin_lock+0x177/0x190 [ 4832.192640] _raw_spin_lock+0x21/0x30 [ 4832.194208] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4832.197969] ? do_raw_spin_unlock+0x75/0x190 [ 4832.199296] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4832.203249] ? do_wait_intr+0xf0/0xf0 [ 4832.207052] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4832.214062] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4832.216898] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4832.219867] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4832.222411] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4832.226826] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4832.230906] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4832.233315] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4832.242662] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4832.246183] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4832.252063] lmv_intent_open+0x338/0x1150 [lmv] [ 4832.255607] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4832.258613] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4832.261798] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4832.263740] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4832.265793] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4832.268958] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4832.272035] ? do_raw_spin_unlock+0x75/0x190 [ 4832.274181] ? _raw_spin_unlock+0x12/0x30 [ 4832.276212] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4832.278740] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4832.281149] ll_atomic_open+0x36a/0xd70 [lustre] [ 4832.283713] lookup_open+0x356/0xb40 [ 4832.284943] do_last+0x313/0xfc0 [ 4832.286309] ? nd_jump_root+0xe5/0x160 [ 4832.289045] ? path_init+0x437/0x520 [ 4832.290783] path_openat+0xf7/0x500 [ 4832.292524] ? do_raw_spin_unlock+0x75/0x190 [ 4832.295550] do_filp_open+0x99/0x140 [ 4832.298243] ? getname_flags+0x6e/0x330 [ 4832.301287] ? __check_object_size+0xff/0x256 [ 4832.304118] ? do_raw_spin_unlock+0x75/0x190 [ 4832.309224] ? _raw_spin_unlock+0x12/0x30 [ 4832.312243] do_sys_openat2+0x2b4/0x410 [ 4832.316141] do_sys_open+0x73/0xa0 [ 4832.318707] __x64_sys_openat+0x24/0x30 [ 4832.320777] do_syscall_64+0xc1/0x440 [ 4832.322517] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4832.325455] RIP: 0033:0x7ff54af45332 [ 4832.327294] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4832.341156] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4832.349951] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 4832.355085] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 4832.361229] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 4832.365507] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 4832.369158] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 4832.738641] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2873s! [ 4832.757806] Showing busy workqueues and worker pools: [ 4832.760811] workqueue events: flags=0x0 [ 4832.765810] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4832.777351] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 4832.790660] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4832.796812] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 4832.809517] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4832.819428] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 4832.826604] workqueue events_long: flags=0x0 [ 4832.828960] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4832.838425] pending: lu_objects_flush [obdclass] [ 4832.843607] workqueue events_unbound: flags=0x2 [ 4832.849944] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4832.859882] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4832.871823] workqueue events_power_efficient: flags=0x80 [ 4832.881231] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4832.890463] pending: neigh_periodic_work [ 4832.895954] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4832.904729] pending: do_cache_clean, check_lifetime [ 4832.906501] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4832.912820] pending: neigh_periodic_work [ 4832.920272] workqueue writeback: flags=0x4a [ 4832.921407] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4832.922920] pending: wb_workfn, wb_workfn [ 4832.931461] workqueue xprtiod: flags=0xa [ 4832.935867] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4832.943984] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 4832.951446] workqueue ipv6_addrconf: flags=0x40008 [ 4832.955833] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4832.964838] pending: addrconf_verify_work [ 4832.969844] workqueue knbd0-recv: flags=0x1a [ 4832.971409] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4832.976939] in-flight: 657:recv_work [ 4832.981868] workqueue ptlrpc_pinger: flags=0x2 [ 4832.986813] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4832.998383] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 4833.011049] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2874s workers=4 idle: 20831 12642 28859 [ 4833.023507] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4823s workers=2 idle: 659 [ 4852.058227] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 4852.064962] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4852.117172] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4852.122944] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4852.133313] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4852.139008] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4852.165120] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4852.173596] RAX: 0000000000005710 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 4852.181531] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 4852.185773] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4852.192292] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 4852.202180] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 4852.210436] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 4852.217810] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4852.223269] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 4852.234045] Call Trace: [ 4852.235111] [ 4852.235953] ? show_regs.cold.9+0x22/0x2f [ 4852.242050] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4852.245676] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4852.249567] ? watchdog+0x40/0x40 [ 4852.253324] ? hrtimer_interrupt+0x128/0x2c0 [ 4852.259141] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4852.261891] ? apic_timer_interrupt+0xf/0x20 [ 4852.275907] [ 4852.276589] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4852.281611] do_raw_spin_lock+0x177/0x190 [ 4852.288943] _raw_spin_lock+0x21/0x30 [ 4852.294551] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4852.302769] ? do_raw_spin_unlock+0x75/0x190 [ 4852.310126] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4852.314342] ? do_wait_intr+0xf0/0xf0 [ 4852.315791] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4852.318306] mdc_close+0x53b/0xd70 [mdc] [ 4852.319838] lmv_close+0x18d/0x490 [lmv] [ 4852.322120] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 4852.324490] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4852.328623] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4852.331129] ll_md_close+0x274/0x800 [lustre] [ 4852.332830] ll_file_release+0x128/0xc50 [lustre] [ 4852.334921] __fput+0x10d/0x440 [ 4852.336340] ____fput+0x12/0x20 [ 4852.337683] task_work_run+0xc6/0x110 [ 4852.343282] exit_to_usermode_loop+0x1dd/0x1f0 [ 4852.348734] do_syscall_64+0x3d6/0x440 [ 4852.350298] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4852.355999] RIP: 0033:0x7ff54af45cd8 [ 4852.359181] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4852.371850] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4852.375067] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 4852.384572] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 4852.388067] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 4852.392921] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 4852.397494] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4856.059613] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 4856.068086] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4856.097985] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4856.105594] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4856.117398] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4856.126095] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4856.134052] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4856.154981] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4856.169728] RAX: 0000000000007f1c RBX: ffff96848811cae0 RCX: 0000000000000001 [ 4856.187395] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 4856.196633] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 4856.205816] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 4856.213954] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 4856.220982] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 4856.226661] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4856.231470] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 4856.237299] Call Trace: [ 4856.240931] [ 4856.242350] ? show_regs.cold.9+0x22/0x2f [ 4856.244159] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4856.247285] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4856.249980] ? watchdog+0x40/0x40 [ 4856.252160] ? hrtimer_interrupt+0x128/0x2c0 [ 4856.254882] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4856.259361] ? apic_timer_interrupt+0xf/0x20 [ 4856.261621] [ 4856.263032] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4856.266495] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4856.271474] do_raw_spin_lock+0x177/0x190 [ 4856.275785] _raw_spin_lock+0x21/0x30 [ 4856.278577] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4856.282238] process_one_work+0x2c8/0x700 [ 4856.286790] worker_thread+0x296/0x6e0 [ 4856.289659] ? rescuer_thread+0x570/0x570 [ 4856.291279] kthread+0x1d1/0x200 [ 4856.295509] ? set_kthread_struct+0x70/0x70 [ 4856.303706] ret_from_fork+0x1f/0x30 [ 4860.056412] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [ 4860.060531] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [ 4860.061611] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4860.061680] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4860.061686] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4860.061689] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4860.061702] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4860.061706] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4860.061710] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 4860.061712] RDX: 0000000000007ada RSI: 0000000000000003 RDI: ffff96848811cae0 [ 4860.061715] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 4860.061717] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 4860.061719] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4860.061722] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 4860.061725] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4860.061727] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 4860.061735] Call Trace: [ 4860.061740] [ 4860.061746] ? show_regs.cold.9+0x22/0x2f [ 4860.061755] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4860.061761] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4860.061765] ? watchdog+0x40/0x40 [ 4860.061769] ? hrtimer_interrupt+0x128/0x2c0 [ 4860.061773] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4860.061781] ? apic_timer_interrupt+0xf/0x20 [ 4860.061785] [ 4860.061787] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4860.061792] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4860.061796] do_raw_spin_lock+0x177/0x190 [ 4860.061800] _raw_spin_lock+0x21/0x30 [ 4860.061806] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4860.062102] obd_mod_free+0x46/0x140 [ptlrpc] [ 4860.062843] mdc_commit_open+0x91/0xb0 [mdc] [ 4860.062881] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4860.063204] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4860.063435] after_reply+0xe2c/0x1780 [ptlrpc] [ 4860.063633] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4860.063829] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4860.064030] ? do_wait_intr+0xf0/0xf0 [ 4860.064039] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4860.064239] kthread+0x1d1/0x200 [ 4860.064245] ? set_kthread_struct+0x70/0x70 [ 4860.064249] ret_from_fork+0x1f/0x30 [ 4860.277683] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4860.333660] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4860.342332] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4860.348771] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4860.354909] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4860.366070] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4860.370876] RAX: 000000000000711c RBX: ffff96848811cae0 RCX: 0000000000000001 [ 4860.378464] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 4860.383404] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4860.386816] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 4860.390640] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 4860.394531] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 4860.400422] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4860.405692] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 4860.407676] Call Trace: [ 4860.408883] [ 4860.409893] ? show_regs.cold.9+0x22/0x2f [ 4860.410908] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4860.414584] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4860.417785] ? watchdog+0x40/0x40 [ 4860.418961] ? hrtimer_interrupt+0x128/0x2c0 [ 4860.420061] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4860.421495] ? apic_timer_interrupt+0xf/0x20 [ 4860.423318] [ 4860.424032] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4860.428928] do_raw_spin_lock+0x177/0x190 [ 4860.431766] _raw_spin_lock+0x21/0x30 [ 4860.438542] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4860.443595] ? do_raw_spin_unlock+0x75/0x190 [ 4860.446064] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4860.451442] ? do_wait_intr+0xf0/0xf0 [ 4860.454294] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4860.460396] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4860.462574] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4860.466796] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4860.473607] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4860.475449] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4860.479698] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4860.483300] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4860.490469] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4860.498520] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4860.501035] lmv_intent_open+0x338/0x1150 [lmv] [ 4860.504466] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4860.512683] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4860.515872] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4860.518987] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4860.522090] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4860.524552] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4860.532671] ? do_raw_spin_unlock+0x75/0x190 [ 4860.536871] ? _raw_spin_unlock+0x12/0x30 [ 4860.539880] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4860.542131] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4860.545102] ll_atomic_open+0x36a/0xd70 [lustre] [ 4860.548801] lookup_open+0x356/0xb40 [ 4860.551143] do_last+0x313/0xfc0 [ 4860.559002] ? nd_jump_root+0xe5/0x160 [ 4860.561151] ? path_init+0x437/0x520 [ 4860.562359] path_openat+0xf7/0x500 [ 4860.568083] ? do_raw_spin_unlock+0x75/0x190 [ 4860.571738] do_filp_open+0x99/0x140 [ 4860.581576] ? getname_flags+0x6e/0x330 [ 4860.582836] ? __check_object_size+0xff/0x256 [ 4860.591436] ? do_raw_spin_unlock+0x75/0x190 [ 4860.598650] ? _raw_spin_unlock+0x12/0x30 [ 4860.605089] do_sys_openat2+0x2b4/0x410 [ 4860.609678] do_sys_open+0x73/0xa0 [ 4860.614723] __x64_sys_openat+0x24/0x30 [ 4860.621838] do_syscall_64+0xc1/0x440 [ 4860.627376] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4860.634232] RIP: 0033:0x7ff54af45332 [ 4860.640235] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4860.662836] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4860.670807] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 4860.676844] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 4860.688818] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 4860.695020] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 4860.704964] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 4863.457421] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2904s! [ 4863.462054] Showing busy workqueues and worker pools: [ 4863.465282] workqueue events: flags=0x0 [ 4863.466910] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4863.475244] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 4863.482749] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4863.486762] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 4863.494317] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4863.498656] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 4863.508582] workqueue events_long: flags=0x0 [ 4863.514047] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4863.518436] pending: lu_objects_flush [obdclass] [ 4863.521657] workqueue events_unbound: flags=0x2 [ 4863.523902] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4863.527204] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4863.532575] workqueue events_power_efficient: flags=0x80 [ 4863.535788] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4863.540621] pending: neigh_periodic_work [ 4863.542984] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4863.546621] pending: do_cache_clean, check_lifetime [ 4863.549480] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4863.552669] pending: neigh_periodic_work [ 4863.554829] workqueue writeback: flags=0x4a [ 4863.559069] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4863.563238] pending: wb_workfn, wb_workfn [ 4863.565033] workqueue xprtiod: flags=0xa [ 4863.567318] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4863.571798] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 4863.576476] workqueue ipv6_addrconf: flags=0x40008 [ 4863.579824] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4863.583735] pending: addrconf_verify_work [ 4863.585515] workqueue knbd0-recv: flags=0x1a [ 4863.586847] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4863.589379] in-flight: 657:recv_work [ 4863.593525] workqueue ptlrpc_pinger: flags=0x2 [ 4863.596808] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4863.600900] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 4863.604179] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2904s workers=4 idle: 20831 12642 28859 [ 4863.611423] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4853s workers=2 idle: 659 [ 4880.058011] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 4880.060376] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4880.079071] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4880.083121] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4880.086231] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4880.088161] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4880.094193] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4880.096666] RAX: 00000000000037c8 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 4880.099607] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 4880.102181] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4880.104836] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 4880.107734] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 4880.110422] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 4880.115314] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4880.118019] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 4880.121443] Call Trace: [ 4880.122404] [ 4880.123226] ? show_regs.cold.9+0x22/0x2f [ 4880.125371] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4880.127415] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4880.131256] ? watchdog+0x40/0x40 [ 4880.134439] ? hrtimer_interrupt+0x128/0x2c0 [ 4880.137200] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4880.139745] ? apic_timer_interrupt+0xf/0x20 [ 4880.141747] [ 4880.142754] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4880.146430] do_raw_spin_lock+0x177/0x190 [ 4880.149216] _raw_spin_lock+0x21/0x30 [ 4880.151135] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4880.157676] ? do_raw_spin_unlock+0x75/0x190 [ 4880.163418] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4880.167900] ? do_wait_intr+0xf0/0xf0 [ 4880.172386] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4880.178749] mdc_close+0x53b/0xd70 [mdc] [ 4880.182651] lmv_close+0x18d/0x490 [lmv] [ 4880.185210] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 4880.189027] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4880.192164] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4880.194581] ll_md_close+0x274/0x800 [lustre] [ 4880.197336] ll_file_release+0x128/0xc50 [lustre] [ 4880.208029] __fput+0x10d/0x440 [ 4880.210749] ____fput+0x12/0x20 [ 4880.214851] task_work_run+0xc6/0x110 [ 4880.222186] exit_to_usermode_loop+0x1dd/0x1f0 [ 4880.226727] do_syscall_64+0x3d6/0x440 [ 4880.228702] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4880.232411] RIP: 0033:0x7ff54af45cd8 [ 4880.234860] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4880.245402] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4880.249594] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 4880.253372] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 4880.257227] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 4880.264368] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 4880.266855] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4884.059072] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 4884.063204] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4884.090236] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4884.096226] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4884.101112] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4884.105119] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4884.108290] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4884.117723] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4884.121684] RAX: 0000000000000126 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 4884.125935] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 4884.130011] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 4884.133870] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 4884.137587] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 4884.141164] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 4884.145028] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4884.147456] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 4884.157042] Call Trace: [ 4884.158385] [ 4884.159496] ? show_regs.cold.9+0x22/0x2f [ 4884.165530] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4884.172202] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4884.174338] ? watchdog+0x40/0x40 [ 4884.180853] ? hrtimer_interrupt+0x128/0x2c0 [ 4884.182895] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4884.189784] ? apic_timer_interrupt+0xf/0x20 [ 4884.196046] [ 4884.197333] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4884.202164] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4884.205591] do_raw_spin_lock+0x177/0x190 [ 4884.208119] _raw_spin_lock+0x21/0x30 [ 4884.211355] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4884.213314] process_one_work+0x2c8/0x700 [ 4884.215589] worker_thread+0x296/0x6e0 [ 4884.217533] ? rescuer_thread+0x570/0x570 [ 4884.219753] kthread+0x1d1/0x200 [ 4884.222031] ? set_kthread_struct+0x70/0x70 [ 4884.223964] ret_from_fork+0x1f/0x30 [ 4888.055242] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 4888.059434] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4888.060335] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 4888.082355] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4888.082370] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4888.082373] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4888.082386] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4888.082390] RSP: 0000:ffffaf9fc1b97bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4888.082396] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 4888.082398] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 4888.082401] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 4888.082403] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 4888.082405] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4888.082408] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 4888.082411] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4888.082413] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 4888.082421] Call Trace: [ 4888.082427] [ 4888.082433] ? show_regs.cold.9+0x22/0x2f [ 4888.082443] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4888.082449] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4888.082453] ? watchdog+0x40/0x40 [ 4888.082458] ? hrtimer_interrupt+0x128/0x2c0 [ 4888.082462] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4888.082469] ? apic_timer_interrupt+0xf/0x20 [ 4888.082474] [ 4888.082476] ? native_safe_halt+0x2e/0x50 [ 4888.082480] kvm_wait+0xcd/0xe0 [ 4888.082488] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4888.082495] do_raw_spin_lock+0x177/0x190 [ 4888.082499] _raw_spin_lock+0x21/0x30 [ 4888.082506] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4888.082785] obd_mod_free+0x46/0x140 [ptlrpc] [ 4888.083132] mdc_commit_open+0x91/0xb0 [mdc] [ 4888.083170] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4888.083377] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4888.083581] after_reply+0xe2c/0x1780 [ptlrpc] [ 4888.083786] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4888.087617] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4888.087694] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4888.087702] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4888.087705] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4888.087718] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4888.087723] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4888.087726] RAX: 0000000000007d15 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 4888.087728] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 4888.087731] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4888.087732] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 4888.087735] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 4888.087738] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 4888.087741] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4888.087743] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 4888.087752] Call Trace: [ 4888.087757] [ 4888.087763] ? show_regs.cold.9+0x22/0x2f [ 4888.087772] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4888.087778] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4888.087782] ? watchdog+0x40/0x40 [ 4888.087787] ? hrtimer_interrupt+0x128/0x2c0 [ 4888.087791] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4888.087800] ? apic_timer_interrupt+0xf/0x20 [ 4888.087805] [ 4888.087808] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4888.087815] do_raw_spin_lock+0x177/0x190 [ 4888.087820] _raw_spin_lock+0x21/0x30 [ 4888.087825] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4888.089735] ? do_raw_spin_unlock+0x75/0x190 [ 4888.089745] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4888.090134] ? do_wait_intr+0xf0/0xf0 [ 4888.090144] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4888.090344] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4888.101850] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4888.102318] ? do_wait_intr+0xf0/0xf0 [ 4888.102329] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4888.102547] kthread+0x1d1/0x200 [ 4888.102555] ? set_kthread_struct+0x70/0x70 [ 4888.102559] ret_from_fork+0x1f/0x30 [ 4888.430454] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4888.433817] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4888.439081] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4888.443637] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4888.446242] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4888.451513] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4888.454572] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4888.461722] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4888.467540] lmv_intent_open+0x338/0x1150 [lmv] [ 4888.470512] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4888.475248] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4888.477803] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4888.480206] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4888.483323] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4888.488807] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4888.491386] ? do_raw_spin_unlock+0x75/0x190 [ 4888.494479] ? _raw_spin_unlock+0x12/0x30 [ 4888.498353] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4888.505211] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4888.508524] ll_atomic_open+0x36a/0xd70 [lustre] [ 4888.511672] lookup_open+0x356/0xb40 [ 4888.512827] do_last+0x313/0xfc0 [ 4888.513895] ? nd_jump_root+0xe5/0x160 [ 4888.527623] ? path_init+0x437/0x520 [ 4888.529296] path_openat+0xf7/0x500 [ 4888.530616] ? do_raw_spin_unlock+0x75/0x190 [ 4888.532434] do_filp_open+0x99/0x140 [ 4888.534516] ? getname_flags+0x6e/0x330 [ 4888.536519] ? __check_object_size+0xff/0x256 [ 4888.538608] ? do_raw_spin_unlock+0x75/0x190 [ 4888.541455] ? _raw_spin_unlock+0x12/0x30 [ 4888.543523] do_sys_openat2+0x2b4/0x410 [ 4888.546471] do_sys_open+0x73/0xa0 [ 4888.554295] __x64_sys_openat+0x24/0x30 [ 4888.556565] do_syscall_64+0xc1/0x440 [ 4888.557680] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4888.561333] RIP: 0033:0x7ff54af45332 [ 4888.564521] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4888.578306] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4888.584430] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 4888.587722] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 4888.597606] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 4888.603711] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 4888.611822] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 4893.663159] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2934s! [ 4893.670714] Showing busy workqueues and worker pools: [ 4893.675676] workqueue events: flags=0x0 [ 4893.678561] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4893.683667] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 4893.690328] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4893.694737] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 4893.702957] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4893.709176] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 4893.715136] workqueue events_long: flags=0x0 [ 4893.718549] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4893.725546] pending: lu_objects_flush [obdclass] [ 4893.729578] workqueue events_unbound: flags=0x2 [ 4893.733140] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4893.737713] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4893.743911] workqueue events_power_efficient: flags=0x80 [ 4893.748287] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4893.754464] pending: neigh_periodic_work [ 4893.758157] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4893.763361] pending: do_cache_clean, check_lifetime [ 4893.767981] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4893.773282] pending: neigh_periodic_work [ 4893.776319] workqueue writeback: flags=0x4a [ 4893.779301] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4893.783508] pending: wb_workfn, wb_workfn [ 4893.786486] workqueue xprtiod: flags=0xa [ 4893.789338] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4893.793571] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 4893.799014] workqueue ipv6_addrconf: flags=0x40008 [ 4893.804258] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4893.812284] pending: addrconf_verify_work [ 4893.817675] workqueue knbd0-recv: flags=0x1a [ 4893.822279] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4893.827313] in-flight: 657:recv_work [ 4893.830163] workqueue ptlrpc_pinger: flags=0x2 [ 4893.833249] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4893.837652] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 4893.841833] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2934s workers=4 idle: 20831 12642 28859 [ 4893.848715] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4884s workers=2 idle: 659 [ 4893.924294] rcu: INFO: rcu_sched self-detected stall on CPU [ 4893.935040] rcu: 0-...!: (1537758 ticks this GP) idle=98a/1/0x4000000000000002 softirq=125154/125154 fqs=959 [ 4893.948337] (t=2940084 jiffies g=253585 q=3719) [ 4893.953201] rcu: rcu_sched kthread starved for 2935044 jiffies! g253585 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=3 [ 4893.975640] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 4893.982803] rcu: RCU grace-period kthread stack dump: [ 4893.994132] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 4894.002639] Call Trace: [ 4894.015631] __schedule+0x351/0xcb0 [ 4894.017464] schedule+0xc0/0x180 [ 4894.019107] schedule_timeout+0xb4/0x190 [ 4894.030271] ? do_raw_spin_unlock+0x75/0x190 [ 4894.032470] ? __next_timer_interrupt+0x160/0x160 [ 4894.042462] ? prepare_to_swait_event+0xf3/0x140 [ 4894.046409] rcu_gp_kthread+0x8dd/0x15b0 [ 4894.048393] ? force_qs_rnp+0x2c0/0x2c0 [ 4894.068563] kthread+0x1d1/0x200 [ 4894.070194] ? set_kthread_struct+0x70/0x70 [ 4894.073266] ret_from_fork+0x1f/0x30 [ 4894.076615] rcu: Stack dump where RCU GP kthread last ran: [ 4894.083986] Sending NMI from CPU 0 to CPUs 3: [ 4894.093884] NMI backtrace for cpu 3 [ 4894.093890] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4894.093893] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4894.093894] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4894.093896] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4894.093898] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 [ 4894.093901] RAX: 0000000000007225 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 4894.093902] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 4894.093904] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4894.093905] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 4894.093906] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 4894.093908] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 4894.093909] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4894.093911] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 4894.093912] Call Trace: [ 4894.093913] [ 4894.093913] ? show_regs.cold.9+0x22/0x2f [ 4894.093914] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 4894.093915] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 4894.093916] ? nmi_handle+0xbd/0x210 [ 4894.093917] ? default_do_nmi+0x4b/0x1b0 [ 4894.093918] ? do_nmi+0x280/0x380 [ 4894.093918] ? end_repeat_nmi+0x16/0x1f [ 4894.093919] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4894.093920] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4894.093921] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4894.093923] [ 4894.093924] do_raw_spin_lock+0x177/0x190 [ 4894.093925] _raw_spin_lock+0x21/0x30 [ 4894.093926] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4894.093927] ? do_raw_spin_unlock+0x75/0x190 [ 4894.093928] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4894.093929] ? do_wait_intr+0xf0/0xf0 [ 4894.093930] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4894.093931] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4894.093933] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4894.093934] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4894.093935] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4894.093936] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4894.093937] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4894.093938] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4894.093939] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4894.093940] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4894.093942] lmv_intent_open+0x338/0x1150 [lmv] [ 4894.093943] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4894.093944] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4894.093945] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4894.093946] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4894.093947] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4894.093948] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4894.093949] ? do_raw_spin_unlock+0x75/0x190 [ 4894.093950] ? _raw_spin_unlock+0x12/0x30 [ 4894.093951] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4894.093952] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4894.093953] ll_atomic_open+0x36a/0xd70 [lustre] [ 4894.093954] lookup_open+0x356/0xb40 [ 4894.093968] do_last+0x313/0xfc0 [ 4894.093969] ? nd_jump_root+0xe5/0x160 [ 4894.093970] ? path_init+0x437/0x520 [ 4894.093971] path_openat+0xf7/0x500 [ 4894.093972] ? do_raw_spin_unlock+0x75/0x190 [ 4894.093973] do_filp_open+0x99/0x140 [ 4894.093975] ? getname_flags+0x6e/0x330 [ 4894.093976] ? __check_object_size+0xff/0x256 [ 4894.093977] ? do_raw_spin_unlock+0x75/0x190 [ 4894.093978] ? _raw_spin_unlock+0x12/0x30 [ 4894.093979] do_sys_openat2+0x2b4/0x410 [ 4894.093980] do_sys_open+0x73/0xa0 [ 4894.093981] __x64_sys_openat+0x24/0x30 [ 4894.093982] do_syscall_64+0xc1/0x440 [ 4894.093983] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4894.093984] RIP: 0033:0x7ff54af45332 [ 4894.093985] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4894.093988] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4894.093991] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 4894.093992] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 4894.093993] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 4894.093995] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 4894.093996] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 4894.094913] NMI backtrace for cpu 0 [ 4894.527476] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4894.534756] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4894.541267] Call Trace: [ 4894.543084] [ 4894.545232] ? dump_stack+0xbb/0x10e [ 4894.547522] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 4894.552258] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 4894.561794] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 4894.572662] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 4894.577057] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 4894.581859] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 4894.590273] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 4894.595487] ? update_process_times+0x6d/0xc0 [ 4894.599541] ? tick_sched_handle.isra.18+0x27/0x90 [ 4894.604171] ? tick_sched_timer+0x53/0xb0 [ 4894.607122] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4894.611281] ? tick_sched_handle.isra.18+0x90/0x90 [ 4894.615022] ? hrtimer_interrupt+0x128/0x2c0 [ 4894.616247] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4894.620274] ? apic_timer_interrupt+0xf/0x20 [ 4894.623659] [ 4894.625263] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4894.629243] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4894.633522] ? do_raw_spin_lock+0x177/0x190 [ 4894.639413] ? _raw_spin_lock+0x21/0x30 [ 4894.647853] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4894.654396] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 4894.659275] ? mdc_commit_open+0x91/0xb0 [mdc] [ 4894.663448] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4894.675850] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4894.684581] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 4894.691856] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4894.696476] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4894.704578] ? do_wait_intr+0xf0/0xf0 [ 4894.706676] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4894.712419] ? kthread+0x1d1/0x200 [ 4894.716673] ? set_kthread_struct+0x70/0x70 [ 4894.720576] ? ret_from_fork+0x1f/0x30 [ 4908.058601] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 4908.071369] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4908.145636] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4908.170858] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4908.184447] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4908.196195] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4908.221173] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4908.226564] RAX: 0000000000001f31 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 4908.236252] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 4908.251117] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4908.255284] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 4908.267779] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 4908.284118] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 4908.297625] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4908.304409] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 4908.319461] Call Trace: [ 4908.320470] [ 4908.321222] ? show_regs.cold.9+0x22/0x2f [ 4908.329495] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4908.338971] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4908.343346] ? watchdog+0x40/0x40 [ 4908.354639] ? hrtimer_interrupt+0x128/0x2c0 [ 4908.358338] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4908.360940] ? apic_timer_interrupt+0xf/0x20 [ 4908.371868] [ 4908.372582] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4908.384638] do_raw_spin_lock+0x177/0x190 [ 4908.390310] _raw_spin_lock+0x21/0x30 [ 4908.398896] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4908.408601] ? do_raw_spin_unlock+0x75/0x190 [ 4908.411182] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4908.420497] ? do_wait_intr+0xf0/0xf0 [ 4908.429038] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4908.442553] mdc_close+0x53b/0xd70 [mdc] [ 4908.444005] lmv_close+0x18d/0x490 [lmv] [ 4908.459490] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 4908.461487] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4908.477730] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4908.479361] ll_md_close+0x274/0x800 [lustre] [ 4908.493220] ll_file_release+0x128/0xc50 [lustre] [ 4908.505486] __fput+0x10d/0x440 [ 4908.506527] ____fput+0x12/0x20 [ 4908.507545] task_work_run+0xc6/0x110 [ 4908.508725] exit_to_usermode_loop+0x1dd/0x1f0 [ 4908.523603] do_syscall_64+0x3d6/0x440 [ 4908.528986] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4908.539453] RIP: 0033:0x7ff54af45cd8 [ 4908.547440] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4908.570314] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4908.578697] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 4908.595064] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 4908.597527] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 4908.607041] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 4908.618001] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4912.059708] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 4912.083370] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4912.166367] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4912.183290] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4912.201758] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4912.212312] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4912.229046] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4912.247733] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4912.250831] RAX: 0000000000003205 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 4912.273792] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 4912.286257] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 4912.298894] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 4912.312535] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 4912.327574] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 4912.342261] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4912.353932] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 4912.355730] Call Trace: [ 4912.356377] [ 4912.369897] ? show_regs.cold.9+0x22/0x2f [ 4912.374704] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4912.389495] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4912.390767] ? watchdog+0x40/0x40 [ 4912.391687] ? hrtimer_interrupt+0x128/0x2c0 [ 4912.407059] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4912.421922] ? apic_timer_interrupt+0xf/0x20 [ 4912.429466] [ 4912.437411] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4912.446418] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4912.470455] do_raw_spin_lock+0x177/0x190 [ 4912.484099] _raw_spin_lock+0x21/0x30 [ 4912.488194] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4912.498282] process_one_work+0x2c8/0x700 [ 4912.514858] worker_thread+0x296/0x6e0 [ 4912.516446] ? rescuer_thread+0x570/0x570 [ 4912.525319] kthread+0x1d1/0x200 [ 4912.533232] ? set_kthread_struct+0x70/0x70 [ 4912.537575] ret_from_fork+0x1f/0x30 [ 4920.055444] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [ 4920.061202] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [ 4920.064446] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4920.064524] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4920.064907] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4920.064911] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4920.064925] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4920.064929] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4920.064934] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 4920.064936] RDX: 00000000000000d5 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 4920.064939] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 4920.064941] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 4920.064943] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4920.064946] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 4920.064949] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4920.064950] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 4920.064959] Call Trace: [ 4920.064965] [ 4920.064971] ? show_regs.cold.9+0x22/0x2f [ 4920.064981] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4920.064987] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4920.064991] ? watchdog+0x40/0x40 [ 4920.064994] ? hrtimer_interrupt+0x128/0x2c0 [ 4920.064998] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4920.065005] ? apic_timer_interrupt+0xf/0x20 [ 4920.065009] [ 4920.065013] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4920.065017] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4920.065021] do_raw_spin_lock+0x177/0x190 [ 4920.065026] _raw_spin_lock+0x21/0x30 [ 4920.065031] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4920.065311] obd_mod_free+0x46/0x140 [ptlrpc] [ 4920.066143] mdc_commit_open+0x91/0xb0 [mdc] [ 4920.066185] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4920.066404] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4920.066610] after_reply+0xe2c/0x1780 [ptlrpc] [ 4920.071892] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4920.071978] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4920.071986] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4920.071989] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4920.072006] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4920.072011] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4920.072016] RAX: 0000000000007e83 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 4920.072018] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 4920.072020] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4920.072022] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 4920.072024] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 4920.072027] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 4920.072029] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4920.072032] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 4920.072041] Call Trace: [ 4920.072048] [ 4920.072053] ? show_regs.cold.9+0x22/0x2f [ 4920.072063] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4920.072068] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4920.072072] ? watchdog+0x40/0x40 [ 4920.072076] ? hrtimer_interrupt+0x128/0x2c0 [ 4920.072080] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4920.072089] ? apic_timer_interrupt+0xf/0x20 [ 4920.072094] [ 4920.072097] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4920.072103] do_raw_spin_lock+0x177/0x190 [ 4920.072108] _raw_spin_lock+0x21/0x30 [ 4920.072113] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4920.073054] ? do_raw_spin_unlock+0x75/0x190 [ 4920.073061] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4920.073425] ? do_wait_intr+0xf0/0xf0 [ 4920.073434] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4920.073879] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4920.074089] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4920.129010] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4920.129319] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4920.129867] ? do_wait_intr+0xf0/0xf0 [ 4920.129878] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4920.130094] kthread+0x1d1/0x200 [ 4920.130101] ? set_kthread_struct+0x70/0x70 [ 4920.130104] ret_from_fork+0x1f/0x30 [ 4920.384672] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4920.387021] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4920.389853] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4920.392348] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4920.394297] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4920.396645] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4920.399319] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4920.401590] lmv_intent_open+0x338/0x1150 [lmv] [ 4920.403688] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4920.406287] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4920.408634] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4920.410906] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4920.412988] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4920.415598] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4920.417820] ? do_raw_spin_unlock+0x75/0x190 [ 4920.420026] ? _raw_spin_unlock+0x12/0x30 [ 4920.421912] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4920.426618] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4920.429019] ll_atomic_open+0x36a/0xd70 [lustre] [ 4920.432010] lookup_open+0x356/0xb40 [ 4920.433694] do_last+0x313/0xfc0 [ 4920.435351] ? nd_jump_root+0xe5/0x160 [ 4920.437030] ? path_init+0x437/0x520 [ 4920.438948] path_openat+0xf7/0x500 [ 4920.440445] ? do_raw_spin_unlock+0x75/0x190 [ 4920.442105] do_filp_open+0x99/0x140 [ 4920.443936] ? getname_flags+0x6e/0x330 [ 4920.445582] ? __check_object_size+0xff/0x256 [ 4920.447370] ? do_raw_spin_unlock+0x75/0x190 [ 4920.448910] ? _raw_spin_unlock+0x12/0x30 [ 4920.450846] do_sys_openat2+0x2b4/0x410 [ 4920.452410] do_sys_open+0x73/0xa0 [ 4920.453841] __x64_sys_openat+0x24/0x30 [ 4920.455445] do_syscall_64+0xc1/0x440 [ 4920.457010] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4920.459391] RIP: 0033:0x7ff54af45332 [ 4920.461165] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4920.470445] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4920.474000] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 4920.477345] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 4920.480490] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 4920.483461] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 4920.486472] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 4923.875480] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2964s! [ 4923.882812] Showing busy workqueues and worker pools: [ 4923.886617] workqueue events: flags=0x0 [ 4923.888062] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4923.894680] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 4923.904815] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4923.917167] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 4923.929088] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4923.932358] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 4923.942050] workqueue events_long: flags=0x0 [ 4923.943925] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4923.955712] pending: lu_objects_flush [obdclass] [ 4923.964272] workqueue events_unbound: flags=0x2 [ 4923.966526] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4923.970062] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4923.981469] workqueue events_power_efficient: flags=0x80 [ 4923.987902] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4923.994185] pending: neigh_periodic_work [ 4923.998487] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4924.004966] pending: do_cache_clean, check_lifetime [ 4924.014957] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4924.019375] pending: neigh_periodic_work [ 4924.021117] workqueue writeback: flags=0x4a [ 4924.022962] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4924.026802] pending: wb_workfn, wb_workfn [ 4924.031896] workqueue xprtiod: flags=0xa [ 4924.034630] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4924.040507] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 4924.054371] workqueue ipv6_addrconf: flags=0x40008 [ 4924.065732] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4924.071472] pending: addrconf_verify_work [ 4924.074164] workqueue knbd0-recv: flags=0x1a [ 4924.082592] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4924.090230] in-flight: 657:recv_work [ 4924.101014] workqueue ptlrpc_pinger: flags=0x2 [ 4924.102605] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4924.112741] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 4924.123262] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2965s workers=4 idle: 20831 12642 28859 [ 4924.134728] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4914s workers=2 idle: 659 [ 4936.058411] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [ 4936.062363] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4936.125217] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4936.138282] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4936.149091] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4936.157640] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4936.199658] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4936.210682] RAX: 0000000000006343 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 4936.222279] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 4936.232131] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4936.237844] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 4936.241811] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 4936.254492] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 4936.257960] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4936.273126] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 4936.290162] Call Trace: [ 4936.295405] [ 4936.297022] ? show_regs.cold.9+0x22/0x2f [ 4936.304294] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4936.316956] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4936.329969] ? watchdog+0x40/0x40 [ 4936.340656] ? hrtimer_interrupt+0x128/0x2c0 [ 4936.351108] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4936.366854] ? apic_timer_interrupt+0xf/0x20 [ 4936.374558] [ 4936.377465] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4936.382984] do_raw_spin_lock+0x177/0x190 [ 4936.387787] _raw_spin_lock+0x21/0x30 [ 4936.391984] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4936.398834] ? do_raw_spin_unlock+0x75/0x190 [ 4936.404441] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4936.414968] ? do_wait_intr+0xf0/0xf0 [ 4936.420654] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4936.429620] mdc_close+0x53b/0xd70 [mdc] [ 4936.439845] lmv_close+0x18d/0x490 [lmv] [ 4936.441275] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 4936.452529] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4936.454581] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4936.465901] ll_md_close+0x274/0x800 [lustre] [ 4936.471623] ll_file_release+0x128/0xc50 [lustre] [ 4936.478705] __fput+0x10d/0x440 [ 4936.483715] ____fput+0x12/0x20 [ 4936.486723] task_work_run+0xc6/0x110 [ 4936.489934] exit_to_usermode_loop+0x1dd/0x1f0 [ 4936.496206] do_syscall_64+0x3d6/0x440 [ 4936.501733] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4936.505140] RIP: 0033:0x7ff54af45cd8 [ 4936.506328] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4936.527986] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4936.551945] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 4936.562904] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 4936.574034] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 4936.576438] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 4936.594902] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4940.060127] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 4940.067573] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4940.147897] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4940.166422] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4940.183610] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4940.190616] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4940.200290] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4940.223683] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4940.240987] RAX: 0000000000002086 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 4940.258022] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 4940.273975] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 4940.284940] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 4940.290244] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 4940.294559] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 4940.300826] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4940.304682] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 4940.316053] Call Trace: [ 4940.320504] [ 4940.324472] ? show_regs.cold.9+0x22/0x2f [ 4940.332376] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4940.341596] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4940.343654] ? watchdog+0x40/0x40 [ 4940.351027] ? hrtimer_interrupt+0x128/0x2c0 [ 4940.357863] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4940.368009] ? apic_timer_interrupt+0xf/0x20 [ 4940.375380] [ 4940.376856] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4940.390537] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4940.393003] do_raw_spin_lock+0x177/0x190 [ 4940.398897] _raw_spin_lock+0x21/0x30 [ 4940.404704] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4940.413114] process_one_work+0x2c8/0x700 [ 4940.418963] worker_thread+0x296/0x6e0 [ 4940.428284] ? rescuer_thread+0x570/0x570 [ 4940.432786] kthread+0x1d1/0x200 [ 4940.439378] ? set_kthread_struct+0x70/0x70 [ 4940.445811] ret_from_fork+0x1f/0x30 [ 4948.060563] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [ 4948.080029] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4948.145061] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4948.159847] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4948.171761] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4948.187819] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4948.222517] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4948.236965] RAX: 0000000000007d0f RBX: ffff96848811cae0 RCX: 0000000000000001 [ 4948.239367] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 4948.251623] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4948.257601] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 4948.261407] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 4948.269245] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 4948.290530] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4948.292423] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 4948.302552] Call Trace: [ 4948.304000] [ 4948.313211] ? show_regs.cold.9+0x22/0x2f [ 4948.316722] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4948.323903] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4948.330718] ? watchdog+0x40/0x40 [ 4948.339337] ? hrtimer_interrupt+0x128/0x2c0 [ 4948.341305] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4948.349763] ? apic_timer_interrupt+0xf/0x20 [ 4948.353602] [ 4948.356730] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4948.361859] do_raw_spin_lock+0x177/0x190 [ 4948.365741] _raw_spin_lock+0x21/0x30 [ 4948.368783] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4948.375937] ? do_raw_spin_unlock+0x75/0x190 [ 4948.377471] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4948.394372] ? do_wait_intr+0xf0/0xf0 [ 4948.400763] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4948.405667] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4948.416087] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4948.418123] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4948.431611] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4948.440850] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4948.448883] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4948.451523] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4948.458107] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4948.462128] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4948.464993] lmv_intent_open+0x338/0x1150 [lmv] [ 4948.473692] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4948.480548] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4948.490949] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4948.492410] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4948.502609] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4948.505733] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4948.508733] ? do_raw_spin_unlock+0x75/0x190 [ 4948.511365] ? _raw_spin_unlock+0x12/0x30 [ 4948.512698] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4948.517606] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4948.524952] ll_atomic_open+0x36a/0xd70 [lustre] [ 4948.527851] lookup_open+0x356/0xb40 [ 4948.532131] do_last+0x313/0xfc0 [ 4948.534786] ? nd_jump_root+0xe5/0x160 [ 4948.537468] ? path_init+0x437/0x520 [ 4948.539274] path_openat+0xf7/0x500 [ 4948.544060] ? do_raw_spin_unlock+0x75/0x190 [ 4948.547320] do_filp_open+0x99/0x140 [ 4948.548456] ? getname_flags+0x6e/0x330 [ 4948.549712] ? __check_object_size+0xff/0x256 [ 4948.553662] ? do_raw_spin_unlock+0x75/0x190 [ 4948.555886] ? _raw_spin_unlock+0x12/0x30 [ 4948.562545] do_sys_openat2+0x2b4/0x410 [ 4948.566337] do_sys_open+0x73/0xa0 [ 4948.568793] __x64_sys_openat+0x24/0x30 [ 4948.573242] do_syscall_64+0xc1/0x440 [ 4948.575800] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4948.579892] RIP: 0033:0x7ff54af45332 [ 4948.582609] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4948.596620] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4948.599871] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 4948.607513] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 4948.613078] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 4948.619032] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 4948.626824] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 4952.055056] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 4952.061806] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4952.107387] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4952.118454] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4952.124993] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4952.129858] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4952.142676] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4952.147673] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 4952.152074] RDX: 0000000000000f89 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 4952.156938] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 4952.163013] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 4952.167345] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4952.171803] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 4952.175747] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4952.177670] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 4952.181382] Call Trace: [ 4952.182881] [ 4952.184196] ? show_regs.cold.9+0x22/0x2f [ 4952.186347] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4952.188973] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4952.191339] ? watchdog+0x40/0x40 [ 4952.193006] ? hrtimer_interrupt+0x128/0x2c0 [ 4952.195199] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4952.197690] ? apic_timer_interrupt+0xf/0x20 [ 4952.199822] [ 4952.200835] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4952.203775] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4952.207952] do_raw_spin_lock+0x177/0x190 [ 4952.210177] _raw_spin_lock+0x21/0x30 [ 4952.212342] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4952.215800] obd_mod_free+0x46/0x140 [ptlrpc] [ 4952.218188] mdc_commit_open+0x91/0xb0 [mdc] [ 4952.220523] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4952.223508] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4952.226409] after_reply+0xe2c/0x1780 [ptlrpc] [ 4952.229328] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4952.233071] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4952.234758] ? do_wait_intr+0xf0/0xf0 [ 4952.236008] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4952.239759] kthread+0x1d1/0x200 [ 4952.241716] ? set_kthread_struct+0x70/0x70 [ 4952.243755] ret_from_fork+0x1f/0x30 [ 4954.591072] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2995s! [ 4954.601751] Showing busy workqueues and worker pools: [ 4954.604833] workqueue events: flags=0x0 [ 4954.606163] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4954.616045] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 4954.622845] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4954.625073] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 4954.629678] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4954.632842] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 4954.635823] workqueue events_long: flags=0x0 [ 4954.646701] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4954.654300] pending: lu_objects_flush [obdclass] [ 4954.658020] workqueue events_unbound: flags=0x2 [ 4954.665382] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4954.674609] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4954.682528] workqueue events_power_efficient: flags=0x80 [ 4954.689068] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4954.694094] pending: neigh_periodic_work [ 4954.705566] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4954.712507] pending: do_cache_clean, check_lifetime [ 4954.714131] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4954.724662] pending: neigh_periodic_work [ 4954.728074] workqueue writeback: flags=0x4a [ 4954.732189] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4954.744948] pending: wb_workfn, wb_workfn [ 4954.749101] workqueue xprtiod: flags=0xa [ 4954.754868] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4954.761854] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 4954.769150] workqueue ipv6_addrconf: flags=0x40008 [ 4954.776296] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4954.786317] pending: addrconf_verify_work [ 4954.787791] workqueue knbd0-recv: flags=0x1a [ 4954.789954] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4954.795866] in-flight: 657:recv_work [ 4954.800124] workqueue ptlrpc_pinger: flags=0x2 [ 4954.803356] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4954.811395] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 4954.813624] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2995s workers=4 idle: 20831 12642 28859 [ 4954.824029] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4945s workers=2 idle: 659 [ 4964.058713] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [ 4964.074926] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4964.122064] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4964.129784] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4964.135243] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4964.141827] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4964.157487] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 4964.160232] RAX: 0000000000007189 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 4964.167367] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 4964.171936] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4964.179824] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 4964.188924] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 4964.206008] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 4964.213675] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4964.222008] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 4964.235559] Call Trace: [ 4964.239970] [ 4964.243717] ? show_regs.cold.9+0x22/0x2f [ 4964.247427] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4964.253143] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4964.254793] ? watchdog+0x40/0x40 [ 4964.258548] ? hrtimer_interrupt+0x128/0x2c0 [ 4964.265629] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4964.271861] ? apic_timer_interrupt+0xf/0x20 [ 4964.278215] [ 4964.278988] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4964.280883] do_raw_spin_lock+0x177/0x190 [ 4964.290408] _raw_spin_lock+0x21/0x30 [ 4964.295397] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4964.303855] ? do_raw_spin_unlock+0x75/0x190 [ 4964.309409] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4964.319623] ? do_wait_intr+0xf0/0xf0 [ 4964.321936] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4964.327011] mdc_close+0x53b/0xd70 [mdc] [ 4964.333809] lmv_close+0x18d/0x490 [lmv] [ 4964.337002] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 4964.345611] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4964.352638] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4964.360716] ll_md_close+0x274/0x800 [lustre] [ 4964.366036] ll_file_release+0x128/0xc50 [lustre] [ 4964.370230] __fput+0x10d/0x440 [ 4964.371886] ____fput+0x12/0x20 [ 4964.374473] task_work_run+0xc6/0x110 [ 4964.376500] exit_to_usermode_loop+0x1dd/0x1f0 [ 4964.378857] do_syscall_64+0x3d6/0x440 [ 4964.380222] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4964.382936] RIP: 0033:0x7ff54af45cd8 [ 4964.384848] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4964.411131] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4964.422910] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 4964.432476] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 4964.444811] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 4964.452237] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 4964.457773] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4968.059656] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 4968.067780] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4968.115686] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4968.135603] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4968.147650] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4968.149985] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4968.162937] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4968.189789] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 4968.192368] RAX: 0000000000007f92 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 4968.199776] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 4968.208527] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 4968.212319] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 4968.223406] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 4968.231918] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 4968.249954] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4968.262583] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 4968.274468] Call Trace: [ 4968.279515] [ 4968.283527] ? show_regs.cold.9+0x22/0x2f [ 4968.295088] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4968.298491] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4968.309915] ? watchdog+0x40/0x40 [ 4968.318807] ? hrtimer_interrupt+0x128/0x2c0 [ 4968.324353] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4968.330487] ? apic_timer_interrupt+0xf/0x20 [ 4968.336583] [ 4968.340613] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4968.347039] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4968.349839] do_raw_spin_lock+0x177/0x190 [ 4968.352613] _raw_spin_lock+0x21/0x30 [ 4968.354618] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4968.357749] process_one_work+0x2c8/0x700 [ 4968.361406] worker_thread+0x296/0x6e0 [ 4968.363841] ? rescuer_thread+0x570/0x570 [ 4968.371061] kthread+0x1d1/0x200 [ 4968.374399] ? set_kthread_struct+0x70/0x70 [ 4968.375853] ret_from_fork+0x1f/0x30 [ 4976.060058] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [ 4976.070487] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4976.146142] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4976.162312] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4976.184141] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4976.192516] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4976.214906] RSP: 0000:ffffaf9fc15234f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4976.228948] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 4976.235135] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 4976.256613] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4976.258607] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 4976.284396] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 4976.302000] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 4976.310758] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4976.312438] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 4976.334551] Call Trace: [ 4976.336607] [ 4976.339784] ? show_regs.cold.9+0x22/0x2f [ 4976.341026] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4976.349898] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4976.351301] ? watchdog+0x40/0x40 [ 4976.352297] ? hrtimer_interrupt+0x128/0x2c0 [ 4976.360122] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4976.363210] ? apic_timer_interrupt+0xf/0x20 [ 4976.374945] [ 4976.375646] ? native_safe_halt+0x2e/0x50 [ 4976.376906] kvm_wait+0xcd/0xe0 [ 4976.377842] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4976.388074] do_raw_spin_lock+0x177/0x190 [ 4976.389316] _raw_spin_lock+0x21/0x30 [ 4976.397783] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4976.401658] ? do_raw_spin_unlock+0x75/0x190 [ 4976.406723] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4976.414657] ? do_wait_intr+0xf0/0xf0 [ 4976.418416] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4976.427330] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4976.441784] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4976.443876] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4976.460612] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4976.467980] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4976.469622] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4976.470859] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4976.485258] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4976.507200] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4976.514311] lmv_intent_open+0x338/0x1150 [lmv] [ 4976.515768] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4976.526191] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4976.528060] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4976.544426] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4976.553204] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4976.566972] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4976.577221] ? do_raw_spin_unlock+0x75/0x190 [ 4976.582193] ? _raw_spin_unlock+0x12/0x30 [ 4976.584173] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4976.592832] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4976.603882] ll_atomic_open+0x36a/0xd70 [lustre] [ 4976.609554] lookup_open+0x356/0xb40 [ 4976.618200] do_last+0x313/0xfc0 [ 4976.624840] ? nd_jump_root+0xe5/0x160 [ 4976.633828] ? path_init+0x437/0x520 [ 4976.636465] path_openat+0xf7/0x500 [ 4976.640426] ? do_raw_spin_unlock+0x75/0x190 [ 4976.650052] do_filp_open+0x99/0x140 [ 4976.653447] ? getname_flags+0x6e/0x330 [ 4976.659836] ? __check_object_size+0xff/0x256 [ 4976.671453] ? do_raw_spin_unlock+0x75/0x190 [ 4976.677820] ? _raw_spin_unlock+0x12/0x30 [ 4976.678951] do_sys_openat2+0x2b4/0x410 [ 4976.688724] do_sys_open+0x73/0xa0 [ 4976.691819] __x64_sys_openat+0x24/0x30 [ 4976.693324] do_syscall_64+0xc1/0x440 [ 4976.695205] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4976.699554] RIP: 0033:0x7ff54af45332 [ 4976.702263] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4976.715910] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4976.719183] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 4976.721720] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 4976.724814] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 4976.727393] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 4976.729735] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 4980.057427] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [ 4980.075139] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4980.105942] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4980.113030] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4980.118813] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4980.129358] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4980.157741] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4980.165074] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 4980.169488] RDX: 000000000000669c RSI: 0000000000000003 RDI: ffff96848811cae0 [ 4980.180487] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 4980.188279] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 4980.193402] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4980.204840] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 4980.214608] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4980.225552] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 4980.236264] Call Trace: [ 4980.240952] [ 4980.241822] ? show_regs.cold.9+0x22/0x2f [ 4980.247581] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4980.250741] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4980.254490] ? watchdog+0x40/0x40 [ 4980.256836] ? hrtimer_interrupt+0x128/0x2c0 [ 4980.258936] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4980.264610] ? apic_timer_interrupt+0xf/0x20 [ 4980.268916] [ 4980.270470] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4980.274777] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4980.276662] do_raw_spin_lock+0x177/0x190 [ 4980.283299] _raw_spin_lock+0x21/0x30 [ 4980.284952] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4980.290853] obd_mod_free+0x46/0x140 [ptlrpc] [ 4980.297870] mdc_commit_open+0x91/0xb0 [mdc] [ 4980.300303] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4980.303461] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4980.311446] after_reply+0xe2c/0x1780 [ptlrpc] [ 4980.317603] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4980.320765] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4980.327769] ? do_wait_intr+0xf0/0xf0 [ 4980.331226] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4980.339720] kthread+0x1d1/0x200 [ 4980.343730] ? set_kthread_struct+0x70/0x70 [ 4980.348958] ret_from_fork+0x1f/0x30 [ 4985.315310] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3026s! [ 4985.323702] Showing busy workqueues and worker pools: [ 4985.327470] workqueue events: flags=0x0 [ 4985.331143] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4985.335220] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 4985.341645] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4985.346244] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 4985.354583] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4985.360523] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 4985.368604] workqueue events_long: flags=0x0 [ 4985.373961] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4985.381774] pending: lu_objects_flush [obdclass] [ 4985.386190] workqueue events_unbound: flags=0x2 [ 4985.388746] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4985.392660] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4985.407178] workqueue events_power_efficient: flags=0x80 [ 4985.409230] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4985.415539] pending: neigh_periodic_work [ 4985.418057] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4985.421613] pending: do_cache_clean, check_lifetime [ 4985.424548] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4985.427842] pending: neigh_periodic_work [ 4985.430207] workqueue writeback: flags=0x4a [ 4985.431843] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4985.437387] pending: wb_workfn, wb_workfn [ 4985.439657] workqueue xprtiod: flags=0xa [ 4985.446720] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4985.451315] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 4985.462429] workqueue ipv6_addrconf: flags=0x40008 [ 4985.468857] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4985.475459] pending: addrconf_verify_work [ 4985.479566] workqueue knbd0-recv: flags=0x1a [ 4985.484755] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4985.491711] in-flight: 657:recv_work [ 4985.494276] workqueue ptlrpc_pinger: flags=0x2 [ 4985.502610] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4985.512302] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 4985.523203] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3026s workers=4 idle: 20831 12642 28859 [ 4985.533484] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4975s workers=2 idle: 659 [ 4992.062456] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [ 4992.076520] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4992.103285] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4992.115505] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4992.118264] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4992.121208] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4992.134854] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4992.139588] RAX: 0000000000007b0f RBX: ffff96848811cae0 RCX: 0000000000000001 [ 4992.142339] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 4992.146300] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4992.148790] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 4992.158784] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 4992.166125] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 4992.172271] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4992.180463] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 4992.188346] Call Trace: [ 4992.196505] [ 4992.197751] ? show_regs.cold.9+0x22/0x2f [ 4992.203323] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4992.206932] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4992.208446] ? watchdog+0x40/0x40 [ 4992.210596] ? hrtimer_interrupt+0x128/0x2c0 [ 4992.213280] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4992.215602] ? apic_timer_interrupt+0xf/0x20 [ 4992.219717] [ 4992.221105] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4992.228626] do_raw_spin_lock+0x177/0x190 [ 4992.230495] _raw_spin_lock+0x21/0x30 [ 4992.242495] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4992.244556] ? do_raw_spin_unlock+0x75/0x190 [ 4992.253056] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4992.258070] ? do_wait_intr+0xf0/0xf0 [ 4992.259309] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4992.265610] mdc_close+0x53b/0xd70 [mdc] [ 4992.267421] lmv_close+0x18d/0x490 [lmv] [ 4992.268788] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 4992.272442] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4992.276520] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4992.278425] ll_md_close+0x274/0x800 [lustre] [ 4992.283435] ll_file_release+0x128/0xc50 [lustre] [ 4992.287186] __fput+0x10d/0x440 [ 4992.288780] ____fput+0x12/0x20 [ 4992.293739] task_work_run+0xc6/0x110 [ 4992.294766] exit_to_usermode_loop+0x1dd/0x1f0 [ 4992.300214] do_syscall_64+0x3d6/0x440 [ 4992.302811] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4992.305122] RIP: 0033:0x7ff54af45cd8 [ 4992.306247] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4992.326678] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4992.332348] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 4992.339203] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 4992.345369] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 4992.348617] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 4992.359729] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4996.063104] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 4996.069242] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4996.110194] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4996.124728] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4996.132351] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4996.137566] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4996.140759] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4996.154238] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4996.158618] RAX: 0000000000005180 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 4996.162555] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 4996.165632] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 4996.175177] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 4996.190114] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 4996.197872] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 4996.207135] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4996.209439] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 4996.216521] Call Trace: [ 4996.222195] [ 4996.227194] ? show_regs.cold.9+0x22/0x2f [ 4996.233182] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4996.240185] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4996.246561] ? watchdog+0x40/0x40 [ 4996.253257] ? hrtimer_interrupt+0x128/0x2c0 [ 4996.257841] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4996.259458] ? apic_timer_interrupt+0xf/0x20 [ 4996.269270] [ 4996.272577] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4996.276153] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4996.283887] do_raw_spin_lock+0x177/0x190 [ 4996.285236] _raw_spin_lock+0x21/0x30 [ 4996.286471] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4996.299042] process_one_work+0x2c8/0x700 [ 4996.300442] worker_thread+0x296/0x6e0 [ 4996.309333] ? rescuer_thread+0x570/0x570 [ 4996.310761] kthread+0x1d1/0x200 [ 4996.311917] ? set_kthread_struct+0x70/0x70 [ 4996.328565] ret_from_fork+0x1f/0x30 [ 5004.062568] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 5004.072324] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5004.140228] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5004.157585] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5004.166836] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5004.174159] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5004.195012] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5004.204166] RAX: 00000000000066b2 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 5004.213183] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 5004.219846] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5004.225196] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 5004.232356] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 5004.241470] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 5004.250707] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5004.258984] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 5004.264817] Call Trace: [ 5004.267114] [ 5004.270990] ? show_regs.cold.9+0x22/0x2f [ 5004.275180] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5004.277641] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5004.279331] ? watchdog+0x40/0x40 [ 5004.280908] ? hrtimer_interrupt+0x128/0x2c0 [ 5004.286273] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5004.288093] ? apic_timer_interrupt+0xf/0x20 [ 5004.289577] [ 5004.295230] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5004.298063] do_raw_spin_lock+0x177/0x190 [ 5004.304662] _raw_spin_lock+0x21/0x30 [ 5004.305995] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5004.310312] ? do_raw_spin_unlock+0x75/0x190 [ 5004.316343] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5004.320110] ? do_wait_intr+0xf0/0xf0 [ 5004.323183] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5004.329421] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5004.331384] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5004.336511] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5004.339963] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5004.344808] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5004.349255] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5004.353345] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5004.364014] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5004.375228] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5004.377280] lmv_intent_open+0x338/0x1150 [lmv] [ 5004.398415] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5004.408036] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5004.420115] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5004.428204] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5004.432990] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5004.438583] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5004.443699] ? do_raw_spin_unlock+0x75/0x190 [ 5004.447766] ? _raw_spin_unlock+0x12/0x30 [ 5004.452327] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5004.454620] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5004.458586] ll_atomic_open+0x36a/0xd70 [lustre] [ 5004.461251] lookup_open+0x356/0xb40 [ 5004.463041] do_last+0x313/0xfc0 [ 5004.465391] ? nd_jump_root+0xe5/0x160 [ 5004.469297] ? path_init+0x437/0x520 [ 5004.472850] path_openat+0xf7/0x500 [ 5004.474687] ? do_raw_spin_unlock+0x75/0x190 [ 5004.476856] do_filp_open+0x99/0x140 [ 5004.481765] ? getname_flags+0x6e/0x330 [ 5004.483360] ? __check_object_size+0xff/0x256 [ 5004.485672] ? do_raw_spin_unlock+0x75/0x190 [ 5004.487409] ? _raw_spin_unlock+0x12/0x30 [ 5004.489905] do_sys_openat2+0x2b4/0x410 [ 5004.491367] do_sys_open+0x73/0xa0 [ 5004.492917] __x64_sys_openat+0x24/0x30 [ 5004.494266] do_syscall_64+0xc1/0x440 [ 5004.498152] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5004.501499] RIP: 0033:0x7ff54af45332 [ 5004.507142] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5004.515979] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5004.520513] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 5004.523868] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 5004.527845] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 5004.530833] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 5004.537141] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 5012.055136] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 5012.061315] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5012.095405] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5012.103797] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5012.113322] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5012.117182] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5012.130491] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5012.136475] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 5012.145555] RDX: 0000000000001e05 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 5012.150995] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 5012.157442] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 5012.162168] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5012.164403] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 5012.173552] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5012.178204] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 5012.183915] Call Trace: [ 5012.184890] [ 5012.185512] ? show_regs.cold.9+0x22/0x2f [ 5012.187276] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5012.189306] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5012.192607] ? watchdog+0x40/0x40 [ 5012.194205] ? hrtimer_interrupt+0x128/0x2c0 [ 5012.198127] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5012.199933] ? apic_timer_interrupt+0xf/0x20 [ 5012.202606] [ 5012.203879] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5012.209660] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5012.214405] do_raw_spin_lock+0x177/0x190 [ 5012.220934] _raw_spin_lock+0x21/0x30 [ 5012.225073] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5012.231898] obd_mod_free+0x46/0x140 [ptlrpc] [ 5012.234087] mdc_commit_open+0x91/0xb0 [mdc] [ 5012.236248] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5012.239177] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5012.242311] after_reply+0xe2c/0x1780 [ptlrpc] [ 5012.244993] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5012.247553] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5012.249491] ? do_wait_intr+0xf0/0xf0 [ 5012.251609] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5012.255395] kthread+0x1d1/0x200 [ 5012.261408] ? set_kthread_struct+0x70/0x70 [ 5012.263061] ret_from_fork+0x1f/0x30 [ 5016.036612] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3057s! [ 5016.044014] Showing busy workqueues and worker pools: [ 5016.048137] workqueue events: flags=0x0 [ 5016.049422] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5016.054011] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 5016.059079] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5016.062880] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 5016.068706] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5016.071596] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 5016.074946] workqueue events_long: flags=0x0 [ 5016.077455] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5016.080570] pending: lu_objects_flush [obdclass] [ 5016.083312] workqueue events_unbound: flags=0x2 [ 5016.085133] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5016.088257] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5016.093748] workqueue events_power_efficient: flags=0x80 [ 5016.096117] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5016.099481] pending: neigh_periodic_work [ 5016.101229] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5016.103573] pending: do_cache_clean, check_lifetime [ 5016.106325] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5016.109153] pending: neigh_periodic_work [ 5016.111085] workqueue writeback: flags=0x4a [ 5016.113069] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5016.115943] pending: wb_workfn, wb_workfn [ 5016.117931] workqueue xprtiod: flags=0xa [ 5016.119805] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5016.122918] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 5016.126747] workqueue ipv6_addrconf: flags=0x40008 [ 5016.128921] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5016.132389] pending: addrconf_verify_work [ 5016.134417] workqueue knbd0-recv: flags=0x1a [ 5016.137785] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5016.142108] in-flight: 657:recv_work [ 5016.144680] workqueue ptlrpc_pinger: flags=0x2 [ 5016.146861] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5016.150383] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 5016.156705] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3057s workers=4 idle: 20831 12642 28859 [ 5016.163172] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5006s workers=2 idle: 659 [ 5020.061764] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [ 5020.075487] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5020.149770] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5020.169115] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5020.185427] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5020.196908] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5020.226169] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5020.234448] RAX: 0000000000002913 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 5020.244290] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 5020.250421] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5020.278320] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 5020.303173] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 5020.313201] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 5020.324201] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5020.333333] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 5020.351898] Call Trace: [ 5020.355711] [ 5020.358646] ? show_regs.cold.9+0x22/0x2f [ 5020.366584] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5020.377152] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5020.384384] ? watchdog+0x40/0x40 [ 5020.396805] ? hrtimer_interrupt+0x128/0x2c0 [ 5020.406855] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5020.421627] ? apic_timer_interrupt+0xf/0x20 [ 5020.429423] [ 5020.430770] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5020.441783] do_raw_spin_lock+0x177/0x190 [ 5020.470191] _raw_spin_lock+0x21/0x30 [ 5020.479463] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5020.497881] ? do_raw_spin_unlock+0x75/0x190 [ 5020.507760] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5020.522509] ? do_wait_intr+0xf0/0xf0 [ 5020.523666] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5020.536011] mdc_close+0x53b/0xd70 [mdc] [ 5020.542371] lmv_close+0x18d/0x490 [lmv] [ 5020.550506] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 5020.553085] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5020.568099] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5020.578753] ll_md_close+0x274/0x800 [lustre] [ 5020.585077] ll_file_release+0x128/0xc50 [lustre] [ 5020.595050] __fput+0x10d/0x440 [ 5020.598613] ____fput+0x12/0x20 [ 5020.603073] task_work_run+0xc6/0x110 [ 5020.611916] exit_to_usermode_loop+0x1dd/0x1f0 [ 5020.622458] do_syscall_64+0x3d6/0x440 [ 5020.632927] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5020.634656] RIP: 0033:0x7ff54af45cd8 [ 5020.648178] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5020.687329] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5020.699556] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 5020.707399] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 5020.716410] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 5020.725668] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 5020.730302] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5024.059784] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 5024.067260] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5024.137077] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5024.147490] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5024.159295] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5024.168081] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5024.171108] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5024.190487] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5024.199467] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 5024.208013] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 5024.218029] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 5024.225678] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 5024.232250] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 5024.240633] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 5024.245549] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5024.249639] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 5024.255583] Call Trace: [ 5024.258078] [ 5024.264178] ? show_regs.cold.9+0x22/0x2f [ 5024.269101] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5024.271048] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5024.275884] ? watchdog+0x40/0x40 [ 5024.279294] ? hrtimer_interrupt+0x128/0x2c0 [ 5024.285816] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5024.288477] ? apic_timer_interrupt+0xf/0x20 [ 5024.291078] [ 5024.292484] ? native_safe_halt+0x2e/0x50 [ 5024.297124] kvm_wait+0xcd/0xe0 [ 5024.301027] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5024.307599] do_raw_spin_lock+0x177/0x190 [ 5024.309947] _raw_spin_lock+0x21/0x30 [ 5024.313088] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5024.317795] process_one_work+0x2c8/0x700 [ 5024.324146] worker_thread+0x296/0x6e0 [ 5024.328677] ? rescuer_thread+0x570/0x570 [ 5024.332501] kthread+0x1d1/0x200 [ 5024.334148] ? set_kthread_struct+0x70/0x70 [ 5024.342357] ret_from_fork+0x1f/0x30 [ 5032.099735] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 5032.111896] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5032.145207] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5032.160077] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5032.167876] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5032.169814] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5032.177733] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5032.182044] RAX: 0000000000002453 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 5032.187251] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 5032.190520] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5032.196425] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 5032.201942] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 5032.205337] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 5032.208032] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5032.212562] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 5032.218662] Call Trace: [ 5032.221195] [ 5032.222506] ? show_regs.cold.9+0x22/0x2f [ 5032.225024] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5032.227182] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5032.228832] ? watchdog+0x40/0x40 [ 5032.231971] ? hrtimer_interrupt+0x128/0x2c0 [ 5032.233913] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5032.236653] ? apic_timer_interrupt+0xf/0x20 [ 5032.238244] [ 5032.240370] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5032.242308] do_raw_spin_lock+0x177/0x190 [ 5032.244633] _raw_spin_lock+0x21/0x30 [ 5032.246268] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5032.249361] ? do_raw_spin_unlock+0x75/0x190 [ 5032.251300] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5032.253695] ? do_wait_intr+0xf0/0xf0 [ 5032.255137] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5032.257166] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5032.260941] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5032.263227] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5032.265609] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5032.267987] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5032.269747] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5032.272348] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5032.274732] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5032.278822] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5032.282922] lmv_intent_open+0x338/0x1150 [lmv] [ 5032.286346] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5032.288784] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5032.292356] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5032.295569] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5032.298248] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5032.303264] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5032.304953] ? do_raw_spin_unlock+0x75/0x190 [ 5032.308066] ? _raw_spin_unlock+0x12/0x30 [ 5032.309791] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5032.314606] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5032.316425] ll_atomic_open+0x36a/0xd70 [lustre] [ 5032.318265] lookup_open+0x356/0xb40 [ 5032.320665] do_last+0x313/0xfc0 [ 5032.322009] ? nd_jump_root+0xe5/0x160 [ 5032.324917] ? path_init+0x437/0x520 [ 5032.327857] path_openat+0xf7/0x500 [ 5032.329555] ? do_raw_spin_unlock+0x75/0x190 [ 5032.331108] do_filp_open+0x99/0x140 [ 5032.332682] ? getname_flags+0x6e/0x330 [ 5032.335427] ? __check_object_size+0xff/0x256 [ 5032.337057] ? do_raw_spin_unlock+0x75/0x190 [ 5032.339955] ? _raw_spin_unlock+0x12/0x30 [ 5032.341788] do_sys_openat2+0x2b4/0x410 [ 5032.344297] do_sys_open+0x73/0xa0 [ 5032.345610] __x64_sys_openat+0x24/0x30 [ 5032.348151] do_syscall_64+0xc1/0x440 [ 5032.349724] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5032.354681] RIP: 0033:0x7ff54af45332 [ 5032.356170] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5032.365501] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5032.368265] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 5032.370806] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 5032.374653] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 5032.380857] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 5032.384279] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 5044.055057] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 5044.063944] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5044.095722] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5044.102482] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5044.124407] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5044.130931] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5044.150914] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5044.163645] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 5044.173819] RDX: 00000000000025f1 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 5044.177439] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 5044.186341] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 5044.196644] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5044.204720] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 5044.215560] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5044.218926] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 5044.233451] Call Trace: [ 5044.235478] [ 5044.236982] ? show_regs.cold.9+0x22/0x2f [ 5044.243783] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5044.248388] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5044.258423] ? watchdog+0x40/0x40 [ 5044.262053] ? hrtimer_interrupt+0x128/0x2c0 [ 5044.267283] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5044.283412] ? apic_timer_interrupt+0xf/0x20 [ 5044.289100] [ 5044.297869] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5044.299841] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5044.313218] do_raw_spin_lock+0x177/0x190 [ 5044.327649] _raw_spin_lock+0x21/0x30 [ 5044.329774] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5044.336769] obd_mod_free+0x46/0x140 [ptlrpc] [ 5044.344425] mdc_commit_open+0x91/0xb0 [mdc] [ 5044.348092] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5044.352876] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5044.359317] after_reply+0xe2c/0x1780 [ptlrpc] [ 5044.361957] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5044.363969] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5044.366983] ? do_wait_intr+0xf0/0xf0 [ 5044.369240] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5044.373190] kthread+0x1d1/0x200 [ 5044.376911] ? set_kthread_struct+0x70/0x70 [ 5044.378618] ret_from_fork+0x1f/0x30 [ 5046.251042] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3087s! [ 5046.280714] Showing busy workqueues and worker pools: [ 5046.282627] workqueue events: flags=0x0 [ 5046.301112] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5046.324107] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 5046.351680] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5046.354162] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 5046.385768] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5046.402732] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 5046.418562] workqueue events_long: flags=0x0 [ 5046.427197] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5046.442411] pending: lu_objects_flush [obdclass] [ 5046.447345] workqueue events_unbound: flags=0x2 [ 5046.457853] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5046.464851] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5046.477258] workqueue events_power_efficient: flags=0x80 [ 5046.482336] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5046.498431] pending: neigh_periodic_work [ 5046.513122] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5046.535658] pending: do_cache_clean, check_lifetime [ 5046.537660] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5046.551137] pending: neigh_periodic_work [ 5046.557304] workqueue writeback: flags=0x4a [ 5046.559412] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5046.565748] pending: wb_workfn, wb_workfn [ 5046.568210] workqueue xprtiod: flags=0xa [ 5046.569956] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5046.578780] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 5046.584780] workqueue ipv6_addrconf: flags=0x40008 [ 5046.609306] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5046.621082] pending: addrconf_verify_work [ 5046.625157] workqueue knbd0-recv: flags=0x1a [ 5046.631918] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5046.638128] in-flight: 657:recv_work [ 5046.644466] workqueue ptlrpc_pinger: flags=0x2 [ 5046.648240] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5046.659710] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 5046.671903] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3087s workers=4 idle: 20831 12642 28859 [ 5046.695019] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5036s workers=2 idle: 659 [ 5048.059890] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 5048.072613] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5048.168125] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5048.192112] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5048.218806] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5048.220292] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5048.272087] RSP: 0018:ffffaf9fc1543ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5048.305265] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 5048.315926] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 5048.336619] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5048.357837] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 5048.369503] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 5048.382073] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 5048.398923] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5048.410853] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 5048.421945] Call Trace: [ 5048.433623] [ 5048.435684] ? show_regs.cold.9+0x22/0x2f [ 5048.445234] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5048.476729] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5048.490716] ? watchdog+0x40/0x40 [ 5048.491688] ? hrtimer_interrupt+0x128/0x2c0 [ 5048.493633] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5048.495095] ? apic_timer_interrupt+0xf/0x20 [ 5048.496485] [ 5048.497185] ? native_safe_halt+0x2e/0x50 [ 5048.498411] kvm_wait+0xcd/0xe0 [ 5048.499375] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5048.500773] do_raw_spin_lock+0x177/0x190 [ 5048.502163] _raw_spin_lock+0x21/0x30 [ 5048.503235] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5048.504989] ? do_raw_spin_unlock+0x75/0x190 [ 5048.506746] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5048.509645] ? do_wait_intr+0xf0/0xf0 [ 5048.511569] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5048.514263] mdc_close+0x53b/0xd70 [mdc] [ 5048.516194] lmv_close+0x18d/0x490 [lmv] [ 5048.518645] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 5048.521043] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5048.523751] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5048.526524] ll_md_close+0x274/0x800 [lustre] [ 5048.528805] ll_file_release+0x128/0xc50 [lustre] [ 5048.531585] __fput+0x10d/0x440 [ 5048.532860] ____fput+0x12/0x20 [ 5048.534613] task_work_run+0xc6/0x110 [ 5048.536269] exit_to_usermode_loop+0x1dd/0x1f0 [ 5048.538626] do_syscall_64+0x3d6/0x440 [ 5048.540570] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5048.542916] RIP: 0033:0x7ff54af45cd8 [ 5048.544816] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5048.559518] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5048.569162] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 5048.575730] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 5048.585781] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 5048.589704] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 5048.605716] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5052.059261] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 5052.063922] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5052.095553] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5052.104869] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5052.110624] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5052.112397] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5052.118013] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5052.141147] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5052.149013] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 5052.153953] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 5052.157671] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 5052.160846] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 5052.166083] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 5052.171581] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 5052.179084] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5052.181660] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 5052.184949] Call Trace: [ 5052.189141] [ 5052.191139] ? show_regs.cold.9+0x22/0x2f [ 5052.197499] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5052.198819] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5052.199998] ? watchdog+0x40/0x40 [ 5052.203783] ? hrtimer_interrupt+0x128/0x2c0 [ 5052.207985] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5052.212986] ? apic_timer_interrupt+0xf/0x20 [ 5052.216969] [ 5052.219576] ? native_safe_halt+0x2e/0x50 [ 5052.224687] kvm_wait+0xcd/0xe0 [ 5052.227637] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5052.232569] do_raw_spin_lock+0x177/0x190 [ 5052.233870] _raw_spin_lock+0x21/0x30 [ 5052.238669] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5052.243072] process_one_work+0x2c8/0x700 [ 5052.247379] worker_thread+0x296/0x6e0 [ 5052.251604] ? rescuer_thread+0x570/0x570 [ 5052.255650] kthread+0x1d1/0x200 [ 5052.258956] ? set_kthread_struct+0x70/0x70 [ 5052.261262] ret_from_fork+0x1f/0x30 [ 5060.061304] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 5060.077449] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5060.123377] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5060.127303] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5060.130668] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5060.133287] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5060.146894] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5060.152041] RAX: 0000000000004082 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 5060.155021] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 5060.165331] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5060.171488] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 5060.176673] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 5060.178891] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 5060.182923] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5060.194932] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 5060.199333] Call Trace: [ 5060.200884] [ 5060.201914] ? show_regs.cold.9+0x22/0x2f [ 5060.203672] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5060.206811] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5060.211816] ? watchdog+0x40/0x40 [ 5060.215831] ? hrtimer_interrupt+0x128/0x2c0 [ 5060.217269] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5060.222673] ? apic_timer_interrupt+0xf/0x20 [ 5060.225953] [ 5060.227019] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5060.229832] do_raw_spin_lock+0x177/0x190 [ 5060.231983] _raw_spin_lock+0x21/0x30 [ 5060.235775] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5060.243141] ? do_raw_spin_unlock+0x75/0x190 [ 5060.245158] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5060.253406] ? do_wait_intr+0xf0/0xf0 [ 5060.255116] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5060.257773] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5060.260857] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5060.265827] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5060.267701] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5060.271973] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5060.274159] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5060.279195] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5060.281070] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5060.289803] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5060.293800] lmv_intent_open+0x338/0x1150 [lmv] [ 5060.296567] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5060.299064] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5060.302433] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5060.306661] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5060.309578] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5060.311821] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5060.315926] ? do_raw_spin_unlock+0x75/0x190 [ 5060.317465] ? _raw_spin_unlock+0x12/0x30 [ 5060.320540] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5060.323863] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5060.326039] ll_atomic_open+0x36a/0xd70 [lustre] [ 5060.330677] lookup_open+0x356/0xb40 [ 5060.332496] do_last+0x313/0xfc0 [ 5060.336531] ? nd_jump_root+0xe5/0x160 [ 5060.339799] ? path_init+0x437/0x520 [ 5060.353535] path_openat+0xf7/0x500 [ 5060.358101] ? do_raw_spin_unlock+0x75/0x190 [ 5060.363162] do_filp_open+0x99/0x140 [ 5060.365088] ? getname_flags+0x6e/0x330 [ 5060.371847] ? __check_object_size+0xff/0x256 [ 5060.376272] ? do_raw_spin_unlock+0x75/0x190 [ 5060.378735] ? _raw_spin_unlock+0x12/0x30 [ 5060.383752] do_sys_openat2+0x2b4/0x410 [ 5060.385767] do_sys_open+0x73/0xa0 [ 5060.387939] __x64_sys_openat+0x24/0x30 [ 5060.390296] do_syscall_64+0xc1/0x440 [ 5060.392236] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5060.395002] RIP: 0033:0x7ff54af45332 [ 5060.397941] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5060.407633] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5060.411035] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 5060.413747] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 5060.419425] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 5060.423486] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 5060.427592] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 5072.057638] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 5072.062927] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5072.117494] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5072.123986] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5072.130364] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5072.137811] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5072.167023] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5072.173840] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 5072.180209] RDX: 00000000000068ed RSI: 0000000000000003 RDI: ffff96848811cae0 [ 5072.186370] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 5072.194040] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 5072.201745] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5072.209129] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 5072.212720] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5072.225979] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 5072.237368] Call Trace: [ 5072.239709] [ 5072.243885] ? show_regs.cold.9+0x22/0x2f [ 5072.245562] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5072.250776] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5072.256917] ? watchdog+0x40/0x40 [ 5072.263350] ? hrtimer_interrupt+0x128/0x2c0 [ 5072.268802] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5072.273315] ? apic_timer_interrupt+0xf/0x20 [ 5072.275798] [ 5072.278309] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5072.284944] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5072.291750] do_raw_spin_lock+0x177/0x190 [ 5072.293420] _raw_spin_lock+0x21/0x30 [ 5072.297753] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5072.300942] obd_mod_free+0x46/0x140 [ptlrpc] [ 5072.303902] mdc_commit_open+0x91/0xb0 [mdc] [ 5072.312023] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5072.317932] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5072.333617] after_reply+0xe2c/0x1780 [ptlrpc] [ 5072.346456] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5072.356971] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5072.367139] ? do_wait_intr+0xf0/0xf0 [ 5072.375641] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5072.387650] kthread+0x1d1/0x200 [ 5072.391393] ? set_kthread_struct+0x70/0x70 [ 5072.397944] ret_from_fork+0x1f/0x30 [ 5073.933794] rcu: INFO: rcu_sched self-detected stall on CPU [ 5073.942924] rcu: 0-...!: (1617933 ticks this GP) idle=98a/1/0x4000000000000002 softirq=125154/125154 fqs=959 [ 5073.947098] (t=3120093 jiffies g=253585 q=3799) [ 5073.953364] rcu: rcu_sched kthread starved for 3115053 jiffies! g253585 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [ 5073.957534] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 5073.961177] rcu: RCU grace-period kthread stack dump: [ 5073.963010] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 5073.966594] Call Trace: [ 5073.967628] __schedule+0x351/0xcb0 [ 5073.968951] schedule+0xc0/0x180 [ 5073.970022] schedule_timeout+0xb4/0x190 [ 5073.971457] ? do_raw_spin_unlock+0x75/0x190 [ 5073.972997] ? __next_timer_interrupt+0x160/0x160 [ 5073.974884] ? prepare_to_swait_event+0xf3/0x140 [ 5073.977020] rcu_gp_kthread+0x8dd/0x15b0 [ 5073.978579] ? force_qs_rnp+0x2c0/0x2c0 [ 5073.985232] kthread+0x1d1/0x200 [ 5073.986497] ? set_kthread_struct+0x70/0x70 [ 5073.989204] ret_from_fork+0x1f/0x30 [ 5073.990867] rcu: Stack dump where RCU GP kthread last ran: [ 5073.994715] Sending NMI from CPU 0 to CPUs 1: [ 5073.998410] NMI backtrace for cpu 1 [ 5073.998417] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5073.998419] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5073.998422] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5073.998423] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5073.998426] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 [ 5073.998429] RAX: 0000000000003885 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 5073.998430] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 5073.998432] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5073.998433] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 5073.998434] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 5073.998436] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 5073.998437] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5073.998438] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 5073.998439] Call Trace: [ 5073.998440] [ 5073.998441] ? show_regs.cold.9+0x22/0x2f [ 5073.998442] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 5073.998443] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 5073.998444] ? nmi_handle+0xbd/0x210 [ 5073.998445] ? default_do_nmi+0x4b/0x1b0 [ 5073.998446] ? do_nmi+0x280/0x380 [ 5073.998447] ? end_repeat_nmi+0x16/0x1f [ 5073.998448] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5073.998449] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5073.998450] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5073.998451] [ 5073.998452] do_raw_spin_lock+0x177/0x190 [ 5073.998453] _raw_spin_lock+0x21/0x30 [ 5073.998454] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5073.998455] ? do_raw_spin_unlock+0x75/0x190 [ 5073.998456] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5073.998457] ? do_wait_intr+0xf0/0xf0 [ 5073.998458] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5073.998459] mdc_close+0x53b/0xd70 [mdc] [ 5073.998460] lmv_close+0x18d/0x490 [lmv] [ 5073.998460] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 5073.998462] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5073.998463] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5073.998464] ll_md_close+0x274/0x800 [lustre] [ 5073.998465] ll_file_release+0x128/0xc50 [lustre] [ 5073.998466] __fput+0x10d/0x440 [ 5073.998467] ____fput+0x12/0x20 [ 5073.998467] task_work_run+0xc6/0x110 [ 5073.998468] exit_to_usermode_loop+0x1dd/0x1f0 [ 5073.998470] do_syscall_64+0x3d6/0x440 [ 5073.998471] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5073.998472] RIP: 0033:0x7ff54af45cd8 [ 5073.998473] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5073.998475] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5073.998478] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 5073.998479] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 5073.998480] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 5073.998482] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 5073.998484] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5073.998884] NMI backtrace for cpu 0 [ 5074.277352] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5074.283774] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5074.293321] Call Trace: [ 5074.295451] [ 5074.296648] ? dump_stack+0xbb/0x10e [ 5074.297907] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 5074.301202] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 5074.303885] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 5074.306318] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 5074.308753] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 5074.311919] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 5074.313822] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 5074.315912] ? update_process_times+0x6d/0xc0 [ 5074.317694] ? tick_sched_handle.isra.18+0x27/0x90 [ 5074.321316] ? tick_sched_timer+0x53/0xb0 [ 5074.323676] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5074.327016] ? tick_sched_handle.isra.18+0x90/0x90 [ 5074.329199] ? hrtimer_interrupt+0x128/0x2c0 [ 5074.331131] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5074.336929] ? apic_timer_interrupt+0xf/0x20 [ 5074.341259] [ 5074.341982] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5074.350750] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5074.352120] ? do_raw_spin_lock+0x177/0x190 [ 5074.354673] ? _raw_spin_lock+0x21/0x30 [ 5074.357165] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5074.360421] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 5074.362576] ? mdc_commit_open+0x91/0xb0 [mdc] [ 5074.364683] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5074.367181] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5074.376098] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 5074.378160] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5074.389912] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5074.395540] ? do_wait_intr+0xf0/0xf0 [ 5074.398608] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5074.409340] ? kthread+0x1d1/0x200 [ 5074.411010] ? set_kthread_struct+0x70/0x70 [ 5074.416869] ? ret_from_fork+0x1f/0x30 [ 5076.961990] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3118s! [ 5076.976286] Showing busy workqueues and worker pools: [ 5076.983373] workqueue events: flags=0x0 [ 5076.987961] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5077.002519] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 5077.013524] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5077.021983] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 5077.035473] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5077.042142] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 5077.056297] workqueue events_long: flags=0x0 [ 5077.060382] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5077.072434] pending: lu_objects_flush [obdclass] [ 5077.076968] workqueue events_unbound: flags=0x2 [ 5077.097353] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5077.099492] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5077.105996] workqueue events_power_efficient: flags=0x80 [ 5077.109893] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5077.116998] pending: neigh_periodic_work [ 5077.125964] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5077.133989] pending: do_cache_clean, check_lifetime [ 5077.136725] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5077.143043] pending: neigh_periodic_work [ 5077.149393] workqueue writeback: flags=0x4a [ 5077.156108] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5077.158319] pending: wb_workfn, wb_workfn [ 5077.163174] workqueue xprtiod: flags=0xa [ 5077.165801] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5077.172580] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 5077.176681] workqueue ipv6_addrconf: flags=0x40008 [ 5077.182897] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5077.187879] pending: addrconf_verify_work [ 5077.190090] workqueue knbd0-recv: flags=0x1a [ 5077.191901] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5077.194498] in-flight: 657:recv_work [ 5077.196131] workqueue ptlrpc_pinger: flags=0x2 [ 5077.198147] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5077.200349] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 5077.202913] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3118s workers=4 idle: 20831 12642 28859 [ 5077.206076] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5067s workers=2 idle: 659 [ 5080.059107] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 5080.061490] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5080.082703] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5080.090420] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5080.098944] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5080.102490] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5080.105127] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5080.114348] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5080.119061] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 5080.124468] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 5080.128366] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 5080.131942] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 5080.138818] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 5080.141779] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 5080.148421] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5080.151415] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 5080.153795] Call Trace: [ 5080.154806] [ 5080.159857] ? show_regs.cold.9+0x22/0x2f [ 5080.162957] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5080.167117] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5080.169094] ? watchdog+0x40/0x40 [ 5080.172349] ? hrtimer_interrupt+0x128/0x2c0 [ 5080.173942] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5080.175112] ? apic_timer_interrupt+0xf/0x20 [ 5080.178779] [ 5080.181113] ? native_safe_halt+0x2e/0x50 [ 5080.183106] kvm_wait+0xcd/0xe0 [ 5080.184757] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5080.186868] do_raw_spin_lock+0x177/0x190 [ 5080.188771] _raw_spin_lock+0x21/0x30 [ 5080.192866] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5080.196901] process_one_work+0x2c8/0x700 [ 5080.201745] worker_thread+0x296/0x6e0 [ 5080.207267] ? rescuer_thread+0x570/0x570 [ 5080.212261] kthread+0x1d1/0x200 [ 5080.216097] ? set_kthread_struct+0x70/0x70 [ 5080.220778] ret_from_fork+0x1f/0x30 [ 5088.065020] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 5088.070271] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5088.126213] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5088.130766] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5088.140762] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5088.148817] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5088.166743] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5088.184244] RAX: 0000000000003b1a RBX: ffff96848811cae0 RCX: 0000000000000001 [ 5088.196421] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 5088.207818] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5088.212556] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 5088.216016] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 5088.221173] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 5088.223946] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5088.235029] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 5088.242852] Call Trace: [ 5088.244982] [ 5088.245775] ? show_regs.cold.9+0x22/0x2f [ 5088.250332] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5088.253288] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5088.256626] ? watchdog+0x40/0x40 [ 5088.261821] ? hrtimer_interrupt+0x128/0x2c0 [ 5088.268857] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5088.276866] ? apic_timer_interrupt+0xf/0x20 [ 5088.282581] [ 5088.291479] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5088.293443] do_raw_spin_lock+0x177/0x190 [ 5088.300706] _raw_spin_lock+0x21/0x30 [ 5088.303483] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5088.307734] ? do_raw_spin_unlock+0x75/0x190 [ 5088.311807] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5088.314583] ? do_wait_intr+0xf0/0xf0 [ 5088.317207] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5088.319819] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5088.322974] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5088.328069] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5088.334611] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5088.341825] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5088.347224] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5088.355934] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5088.357887] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5088.374571] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5088.381807] lmv_intent_open+0x338/0x1150 [lmv] [ 5088.384597] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5088.392842] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5088.398602] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5088.401381] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5088.406873] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5088.414424] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5088.428153] ? do_raw_spin_unlock+0x75/0x190 [ 5088.434188] ? _raw_spin_unlock+0x12/0x30 [ 5088.440972] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5088.459643] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5088.461486] ll_atomic_open+0x36a/0xd70 [lustre] [ 5088.469792] lookup_open+0x356/0xb40 [ 5088.472442] do_last+0x313/0xfc0 [ 5088.478912] ? nd_jump_root+0xe5/0x160 [ 5088.484184] ? path_init+0x437/0x520 [ 5088.488020] path_openat+0xf7/0x500 [ 5088.494014] ? do_raw_spin_unlock+0x75/0x190 [ 5088.498491] do_filp_open+0x99/0x140 [ 5088.503182] ? getname_flags+0x6e/0x330 [ 5088.515958] ? __check_object_size+0xff/0x256 [ 5088.522017] ? do_raw_spin_unlock+0x75/0x190 [ 5088.531028] ? _raw_spin_unlock+0x12/0x30 [ 5088.534054] do_sys_openat2+0x2b4/0x410 [ 5088.541100] do_sys_open+0x73/0xa0 [ 5088.548732] __x64_sys_openat+0x24/0x30 [ 5088.554358] do_syscall_64+0xc1/0x440 [ 5088.561052] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5088.562840] RIP: 0033:0x7ff54af45332 [ 5088.571247] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5088.598887] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5088.619581] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 5088.635825] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 5088.639523] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 5088.647491] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 5088.663662] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 5100.058025] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 5100.071708] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5100.110606] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5100.119505] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5100.132421] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5100.139428] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5100.161565] RSP: 0018:ffffaf9fc1543ae0 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 5100.175316] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 5100.180919] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 5100.187559] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5100.193306] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 5100.197570] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 5100.202341] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 5100.208209] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5100.214578] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 5100.220847] Call Trace: [ 5100.222736] [ 5100.224015] ? show_regs.cold.9+0x22/0x2f [ 5100.228415] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5100.230091] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5100.236171] ? watchdog+0x40/0x40 [ 5100.237428] ? hrtimer_interrupt+0x128/0x2c0 [ 5100.239506] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5100.244381] ? apic_timer_interrupt+0xf/0x20 [ 5100.245658] [ 5100.250578] ? native_safe_halt+0x2e/0x50 [ 5100.252532] kvm_wait+0xcd/0xe0 [ 5100.256092] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5100.260502] do_raw_spin_lock+0x177/0x190 [ 5100.265964] _raw_spin_lock+0x21/0x30 [ 5100.268480] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5100.275419] ? do_raw_spin_unlock+0x75/0x190 [ 5100.287724] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5100.293480] ? do_wait_intr+0xf0/0xf0 [ 5100.296281] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5100.299693] mdc_close+0x53b/0xd70 [mdc] [ 5100.302586] lmv_close+0x18d/0x490 [lmv] [ 5100.305980] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 5100.309283] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5100.312445] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5100.316617] ll_md_close+0x274/0x800 [lustre] [ 5100.319735] ll_file_release+0x128/0xc50 [lustre] [ 5100.322355] __fput+0x10d/0x440 [ 5100.327974] ____fput+0x12/0x20 [ 5100.334586] task_work_run+0xc6/0x110 [ 5100.342774] exit_to_usermode_loop+0x1dd/0x1f0 [ 5100.346525] do_syscall_64+0x3d6/0x440 [ 5100.353370] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5100.359409] RIP: 0033:0x7ff54af45cd8 [ 5100.361495] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5100.385497] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5100.395731] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 5100.399545] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 5100.409069] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 5100.417513] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 5100.429079] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5104.055826] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 5104.069256] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5104.112902] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5104.119201] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5104.124625] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5104.127308] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5104.136330] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5104.140555] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 5104.143809] RDX: 0000000000006144 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 5104.147252] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 5104.151823] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 5104.161184] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5104.171259] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 5104.180415] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5104.182372] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 5104.192688] Call Trace: [ 5104.198218] [ 5104.199216] ? show_regs.cold.9+0x22/0x2f [ 5104.204789] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5104.209965] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5104.214010] ? watchdog+0x40/0x40 [ 5104.216580] ? hrtimer_interrupt+0x128/0x2c0 [ 5104.221583] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5104.223074] ? apic_timer_interrupt+0xf/0x20 [ 5104.225138] [ 5104.226165] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5104.230174] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5104.232569] do_raw_spin_lock+0x177/0x190 [ 5104.234678] _raw_spin_lock+0x21/0x30 [ 5104.237317] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5104.239980] obd_mod_free+0x46/0x140 [ptlrpc] [ 5104.244186] mdc_commit_open+0x91/0xb0 [mdc] [ 5104.245675] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5104.248904] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5104.250670] after_reply+0xe2c/0x1780 [ptlrpc] [ 5104.256495] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5104.263859] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5104.267423] ? do_wait_intr+0xf0/0xf0 [ 5104.269625] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5104.272294] kthread+0x1d1/0x200 [ 5104.273750] ? set_kthread_struct+0x70/0x70 [ 5104.275319] ret_from_fork+0x1f/0x30 [ 5107.679177] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3148s! [ 5107.686200] Showing busy workqueues and worker pools: [ 5107.688481] workqueue events: flags=0x0 [ 5107.690044] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5107.693140] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 5107.699105] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5107.702606] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 5107.725728] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5107.731714] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 5107.740583] workqueue events_long: flags=0x0 [ 5107.745651] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5107.749437] pending: lu_objects_flush [obdclass] [ 5107.753574] workqueue events_unbound: flags=0x2 [ 5107.756033] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5107.761438] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5107.767429] workqueue events_power_efficient: flags=0x80 [ 5107.780751] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5107.788276] pending: neigh_periodic_work [ 5107.794648] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5107.804627] pending: do_cache_clean, check_lifetime [ 5107.810270] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5107.814160] pending: neigh_periodic_work [ 5107.816693] workqueue writeback: flags=0x4a [ 5107.819611] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5107.822864] pending: wb_workfn, wb_workfn [ 5107.828860] workqueue xprtiod: flags=0xa [ 5107.832874] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5107.836520] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 5107.844255] workqueue ipv6_addrconf: flags=0x40008 [ 5107.851802] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5107.858737] pending: addrconf_verify_work [ 5107.860389] workqueue knbd0-recv: flags=0x1a [ 5107.863549] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5107.870631] in-flight: 657:recv_work [ 5107.874042] workqueue ptlrpc_pinger: flags=0x2 [ 5107.876246] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5107.879512] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 5107.882971] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3149s workers=4 idle: 20831 12642 28859 [ 5107.890118] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5098s workers=2 idle: 659 [ 5108.059009] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 5108.071100] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5108.091288] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5108.094855] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5108.097304] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5108.099506] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5108.101423] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5108.107946] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5108.111159] RAX: 0000000000006609 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 5108.113597] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 5108.116388] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 5108.118829] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 5108.122017] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 5108.124738] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 5108.128891] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5108.133415] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 5108.137524] Call Trace: [ 5108.139173] [ 5108.140089] ? show_regs.cold.9+0x22/0x2f [ 5108.142118] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5108.144254] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5108.146271] ? watchdog+0x40/0x40 [ 5108.148062] ? hrtimer_interrupt+0x128/0x2c0 [ 5108.150293] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5108.152442] ? apic_timer_interrupt+0xf/0x20 [ 5108.154057] [ 5108.155038] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5108.157125] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5108.159978] do_raw_spin_lock+0x177/0x190 [ 5108.162033] _raw_spin_lock+0x21/0x30 [ 5108.163586] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5108.168071] process_one_work+0x2c8/0x700 [ 5108.170268] worker_thread+0x296/0x6e0 [ 5108.172367] ? rescuer_thread+0x570/0x570 [ 5108.174371] kthread+0x1d1/0x200 [ 5108.176403] ? set_kthread_struct+0x70/0x70 [ 5108.179104] ret_from_fork+0x1f/0x30 [ 5116.062379] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 5116.070377] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5116.094531] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5116.102239] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5116.109956] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5116.116380] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5116.130953] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5116.137856] RAX: 0000000000007d0b RBX: ffff96848811cae0 RCX: 0000000000000001 [ 5116.148571] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 5116.156854] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5116.169957] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 5116.178440] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 5116.182442] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 5116.189103] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5116.195371] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 5116.198511] Call Trace: [ 5116.201734] [ 5116.203541] ? show_regs.cold.9+0x22/0x2f [ 5116.209110] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5116.211502] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5116.217957] ? watchdog+0x40/0x40 [ 5116.221235] ? hrtimer_interrupt+0x128/0x2c0 [ 5116.224160] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5116.227240] ? apic_timer_interrupt+0xf/0x20 [ 5116.232400] [ 5116.233510] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5116.234954] do_raw_spin_lock+0x177/0x190 [ 5116.236104] _raw_spin_lock+0x21/0x30 [ 5116.239889] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5116.243502] ? do_raw_spin_unlock+0x75/0x190 [ 5116.248091] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5116.250528] ? do_wait_intr+0xf0/0xf0 [ 5116.251471] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5116.253108] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5116.255003] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5116.257606] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5116.259661] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5116.262572] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5116.263740] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5116.267968] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5116.270834] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5116.287059] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5116.292242] lmv_intent_open+0x338/0x1150 [lmv] [ 5116.300254] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5116.310864] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5116.313938] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5116.317527] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5116.319719] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5116.323357] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5116.329119] ? do_raw_spin_unlock+0x75/0x190 [ 5116.331916] ? _raw_spin_unlock+0x12/0x30 [ 5116.340060] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5116.343925] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5116.350720] ll_atomic_open+0x36a/0xd70 [lustre] [ 5116.356778] lookup_open+0x356/0xb40 [ 5116.359535] do_last+0x313/0xfc0 [ 5116.362549] ? nd_jump_root+0xe5/0x160 [ 5116.363939] ? path_init+0x437/0x520 [ 5116.366005] path_openat+0xf7/0x500 [ 5116.367439] ? do_raw_spin_unlock+0x75/0x190 [ 5116.369399] do_filp_open+0x99/0x140 [ 5116.370623] ? getname_flags+0x6e/0x330 [ 5116.372903] ? __check_object_size+0xff/0x256 [ 5116.375373] ? do_raw_spin_unlock+0x75/0x190 [ 5116.377938] ? _raw_spin_unlock+0x12/0x30 [ 5116.380941] do_sys_openat2+0x2b4/0x410 [ 5116.382780] do_sys_open+0x73/0xa0 [ 5116.384255] __x64_sys_openat+0x24/0x30 [ 5116.385688] do_syscall_64+0xc1/0x440 [ 5116.387446] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5116.389560] RIP: 0033:0x7ff54af45332 [ 5116.391317] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5116.399065] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5116.401838] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 5116.404403] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 5116.407732] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 5116.410738] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 5116.414466] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 5128.058754] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 5128.068063] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5128.118917] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5128.135860] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5128.146814] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5128.160242] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5128.180474] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 5128.191444] RAX: 0000000000007818 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 5128.196209] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 5128.201086] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5128.205381] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 5128.211942] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 5128.226038] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 5128.236338] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5128.247994] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 5128.262119] Call Trace: [ 5128.263073] [ 5128.263739] ? show_regs.cold.9+0x22/0x2f [ 5128.266232] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5128.267974] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5128.275734] ? watchdog+0x40/0x40 [ 5128.280902] ? hrtimer_interrupt+0x128/0x2c0 [ 5128.286563] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5128.296728] ? apic_timer_interrupt+0xf/0x20 [ 5128.300972] [ 5128.304855] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5128.315853] do_raw_spin_lock+0x177/0x190 [ 5128.321983] _raw_spin_lock+0x21/0x30 [ 5128.326543] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5128.337215] ? do_raw_spin_unlock+0x75/0x190 [ 5128.340636] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5128.344534] ? do_wait_intr+0xf0/0xf0 [ 5128.347845] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5128.350525] mdc_close+0x53b/0xd70 [mdc] [ 5128.354343] lmv_close+0x18d/0x490 [lmv] [ 5128.359374] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 5128.364061] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5128.368117] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5128.372000] ll_md_close+0x274/0x800 [lustre] [ 5128.375339] ll_file_release+0x128/0xc50 [lustre] [ 5128.379232] __fput+0x10d/0x440 [ 5128.381382] ____fput+0x12/0x20 [ 5128.383881] task_work_run+0xc6/0x110 [ 5128.387301] exit_to_usermode_loop+0x1dd/0x1f0 [ 5128.390722] do_syscall_64+0x3d6/0x440 [ 5128.392561] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5128.399149] RIP: 0033:0x7ff54af45cd8 [ 5128.401900] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5128.412913] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5128.419078] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 5128.427550] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 5128.433229] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 5128.442714] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 5128.447377] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5132.058186] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 5132.064536] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5132.162467] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5132.187817] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5132.212161] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5132.214331] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5132.249487] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5132.256941] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 5132.276351] RDX: 0000000000007d1c RSI: 0000000000000003 RDI: ffff96848811cae0 [ 5132.293173] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 5132.301822] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 5132.314371] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5132.328193] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 5132.341380] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5132.345435] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 5132.361237] Call Trace: [ 5132.364599] [ 5132.365570] ? show_regs.cold.9+0x22/0x2f [ 5132.367534] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5132.373421] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5132.380072] ? watchdog+0x40/0x40 [ 5132.382395] ? hrtimer_interrupt+0x128/0x2c0 [ 5132.387296] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5132.395782] ? apic_timer_interrupt+0xf/0x20 [ 5132.404446] [ 5132.405227] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5132.407182] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5132.409713] do_raw_spin_lock+0x177/0x190 [ 5132.411050] _raw_spin_lock+0x21/0x30 [ 5132.415294] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5132.416843] obd_mod_free+0x46/0x140 [ptlrpc] [ 5132.426638] mdc_commit_open+0x91/0xb0 [mdc] [ 5132.428399] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5132.433093] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5132.439960] after_reply+0xe2c/0x1780 [ptlrpc] [ 5132.446220] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5132.449399] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5132.456434] ? do_wait_intr+0xf0/0xf0 [ 5132.457511] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5132.464844] kthread+0x1d1/0x200 [ 5132.465946] ? set_kthread_struct+0x70/0x70 [ 5132.472726] ret_from_fork+0x1f/0x30 [ 5136.059170] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 5136.096276] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5136.176116] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5136.184669] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5136.190643] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5136.197672] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5136.201213] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5136.214771] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5136.219594] RAX: 0000000000006d4a RBX: ffff96848811cae0 RCX: 0000000000000001 [ 5136.225218] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 5136.228907] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 5136.233959] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 5136.238712] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 5136.250046] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 5136.260380] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5136.272410] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 5136.279014] Call Trace: [ 5136.285884] [ 5136.289960] ? show_regs.cold.9+0x22/0x2f [ 5136.294250] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5136.302995] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5136.306914] ? watchdog+0x40/0x40 [ 5136.309391] ? hrtimer_interrupt+0x128/0x2c0 [ 5136.312174] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5136.318009] ? apic_timer_interrupt+0xf/0x20 [ 5136.328761] [ 5136.329813] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5136.334729] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5136.340422] do_raw_spin_lock+0x177/0x190 [ 5136.342089] _raw_spin_lock+0x21/0x30 [ 5136.343656] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5136.346366] process_one_work+0x2c8/0x700 [ 5136.348717] worker_thread+0x296/0x6e0 [ 5136.351000] ? rescuer_thread+0x570/0x570 [ 5136.353418] kthread+0x1d1/0x200 [ 5136.355490] ? set_kthread_struct+0x70/0x70 [ 5136.358289] ret_from_fork+0x1f/0x30 [ 5138.407651] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3179s! [ 5138.416948] Showing busy workqueues and worker pools: [ 5138.434147] workqueue events: flags=0x0 [ 5138.439367] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5138.452284] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 5138.466117] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5138.470981] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 5138.490069] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5138.503691] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 5138.511013] workqueue events_long: flags=0x0 [ 5138.518071] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5138.524770] pending: lu_objects_flush [obdclass] [ 5138.531667] workqueue events_unbound: flags=0x2 [ 5138.534014] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5138.540255] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5138.561911] workqueue events_power_efficient: flags=0x80 [ 5138.571223] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5138.585654] pending: neigh_periodic_work [ 5138.592270] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5138.601968] pending: do_cache_clean, check_lifetime [ 5138.613604] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5138.626694] pending: neigh_periodic_work [ 5138.636993] workqueue writeback: flags=0x4a [ 5138.639058] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5138.654300] pending: wb_workfn, wb_workfn [ 5138.670093] workqueue xprtiod: flags=0xa [ 5138.686424] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5138.697159] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 5138.714247] workqueue ipv6_addrconf: flags=0x40008 [ 5138.716725] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5138.730839] pending: addrconf_verify_work [ 5138.733506] workqueue knbd0-recv: flags=0x1a [ 5138.738818] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5138.750426] in-flight: 657:recv_work [ 5138.758732] workqueue ptlrpc_pinger: flags=0x2 [ 5138.767835] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5138.780703] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 5138.790150] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3179s workers=4 idle: 20831 12642 28859 [ 5138.803170] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5129s workers=2 idle: 659 [ 5144.071502] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 5144.084041] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5144.134510] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5144.154838] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5144.167497] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5144.176303] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5144.190332] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5144.203548] RAX: 0000000000002286 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 5144.213822] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 5144.223383] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5144.232545] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 5144.240901] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 5144.247482] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 5144.263481] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5144.272926] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 5144.295900] Call Trace: [ 5144.301554] [ 5144.305010] ? show_regs.cold.9+0x22/0x2f [ 5144.316430] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5144.329428] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5144.334285] ? watchdog+0x40/0x40 [ 5144.336524] ? hrtimer_interrupt+0x128/0x2c0 [ 5144.338835] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5144.347338] ? apic_timer_interrupt+0xf/0x20 [ 5144.351554] [ 5144.355308] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5144.364479] do_raw_spin_lock+0x177/0x190 [ 5144.372483] _raw_spin_lock+0x21/0x30 [ 5144.375193] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5144.388193] ? do_raw_spin_unlock+0x75/0x190 [ 5144.389796] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5144.407966] ? do_wait_intr+0xf0/0xf0 [ 5144.414347] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5144.430747] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5144.440810] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5144.462454] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5144.473274] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5144.484092] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5144.494503] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5144.502882] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5144.510003] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5144.519887] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5144.530718] lmv_intent_open+0x338/0x1150 [lmv] [ 5144.538098] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5144.551858] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5144.567447] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5144.568934] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5144.570488] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5144.588956] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5144.590503] ? do_raw_spin_unlock+0x75/0x190 [ 5144.607120] ? _raw_spin_unlock+0x12/0x30 [ 5144.615474] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5144.623142] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5144.635170] ll_atomic_open+0x36a/0xd70 [lustre] [ 5144.641532] lookup_open+0x356/0xb40 [ 5144.647323] do_last+0x313/0xfc0 [ 5144.650024] ? nd_jump_root+0xe5/0x160 [ 5144.654511] ? path_init+0x437/0x520 [ 5144.659744] path_openat+0xf7/0x500 [ 5144.669268] ? do_raw_spin_unlock+0x75/0x190 [ 5144.680968] do_filp_open+0x99/0x140 [ 5144.693705] ? getname_flags+0x6e/0x330 [ 5144.698149] ? __check_object_size+0xff/0x256 [ 5144.708029] ? do_raw_spin_unlock+0x75/0x190 [ 5144.718023] ? _raw_spin_unlock+0x12/0x30 [ 5144.724225] do_sys_openat2+0x2b4/0x410 [ 5144.727218] do_sys_open+0x73/0xa0 [ 5144.736910] __x64_sys_openat+0x24/0x30 [ 5144.747387] do_syscall_64+0xc1/0x440 [ 5144.759022] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5144.775941] RIP: 0033:0x7ff54af45332 [ 5144.777450] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5144.817599] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5144.837903] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 5144.855267] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 5144.879298] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 5144.901176] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 5144.906988] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 5156.058022] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 5156.065516] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5156.133302] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5156.149210] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5156.162255] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5156.164088] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5156.173152] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5156.177143] RAX: 00000000000068ff RBX: ffff96848811cae0 RCX: 0000000000000001 [ 5156.180958] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 5156.187567] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5156.193481] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 5156.199556] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 5156.209081] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 5156.213566] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5156.219081] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 5156.234022] Call Trace: [ 5156.236489] [ 5156.246322] ? show_regs.cold.9+0x22/0x2f [ 5156.257666] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5156.260450] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5156.270604] ? watchdog+0x40/0x40 [ 5156.274271] ? hrtimer_interrupt+0x128/0x2c0 [ 5156.277129] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5156.282476] ? apic_timer_interrupt+0xf/0x20 [ 5156.289170] [ 5156.290214] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5156.296519] do_raw_spin_lock+0x177/0x190 [ 5156.301879] _raw_spin_lock+0x21/0x30 [ 5156.304920] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5156.311473] ? do_raw_spin_unlock+0x75/0x190 [ 5156.313552] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5156.316558] ? do_wait_intr+0xf0/0xf0 [ 5156.318365] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5156.321400] mdc_close+0x53b/0xd70 [mdc] [ 5156.323235] lmv_close+0x18d/0x490 [lmv] [ 5156.325025] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 5156.328121] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5156.331178] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5156.333309] ll_md_close+0x274/0x800 [lustre] [ 5156.335578] ll_file_release+0x128/0xc50 [lustre] [ 5156.337719] __fput+0x10d/0x440 [ 5156.339478] ____fput+0x12/0x20 [ 5156.340933] task_work_run+0xc6/0x110 [ 5156.342741] exit_to_usermode_loop+0x1dd/0x1f0 [ 5156.345214] do_syscall_64+0x3d6/0x440 [ 5156.346901] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5156.349353] RIP: 0033:0x7ff54af45cd8 [ 5156.351388] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5156.360510] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5156.363755] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 5156.367373] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 5156.370986] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 5156.374365] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 5156.378666] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5164.059309] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 5164.064069] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 5164.067974] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5164.068054] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5164.068061] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5164.068064] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5164.068078] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5164.068082] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5164.068086] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 5164.068088] RDX: 0000000000001aa1 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 5164.068091] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 5164.068092] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 5164.068094] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5164.068097] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 5164.068100] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5164.068103] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 5164.068112] Call Trace: [ 5164.068117] [ 5164.068123] ? show_regs.cold.9+0x22/0x2f [ 5164.068132] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5164.068139] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5164.068144] ? watchdog+0x40/0x40 [ 5164.068148] ? hrtimer_interrupt+0x128/0x2c0 [ 5164.068152] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5164.068161] ? apic_timer_interrupt+0xf/0x20 [ 5164.068165] [ 5164.068168] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5164.068173] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5164.068178] do_raw_spin_lock+0x177/0x190 [ 5164.068182] _raw_spin_lock+0x21/0x30 [ 5164.068187] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5164.069608] obd_mod_free+0x46/0x140 [ptlrpc] [ 5164.082802] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5164.082878] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5164.082885] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5164.082892] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5164.117792] mdc_commit_open+0x91/0xb0 [mdc] [ 5164.117843] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5164.118118] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5164.118921] after_reply+0xe2c/0x1780 [ptlrpc] [ 5164.119116] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5164.119766] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5164.119976] ? do_wait_intr+0xf0/0xf0 [ 5164.119987] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5164.120204] kthread+0x1d1/0x200 [ 5164.120210] ? set_kthread_struct+0x70/0x70 [ 5164.120215] ret_from_fork+0x1f/0x30 [ 5164.528606] [ 5164.529018] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5164.538067] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5164.553102] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5164.556608] RAX: 0000000000003e88 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 5164.559742] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 5164.566724] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 5164.570801] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 5164.581654] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 5164.587244] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 5164.596559] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5164.598907] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 5164.609819] Call Trace: [ 5164.610755] [ 5164.614037] ? show_regs.cold.9+0x22/0x2f [ 5164.617895] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5164.622077] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5164.626673] ? watchdog+0x40/0x40 [ 5164.628375] ? hrtimer_interrupt+0x128/0x2c0 [ 5164.630054] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5164.633853] ? apic_timer_interrupt+0xf/0x20 [ 5164.648269] [ 5164.653849] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5164.657709] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5164.664112] do_raw_spin_lock+0x177/0x190 [ 5164.672032] _raw_spin_lock+0x21/0x30 [ 5164.678118] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5164.681537] process_one_work+0x2c8/0x700 [ 5164.683963] worker_thread+0x296/0x6e0 [ 5164.686773] ? rescuer_thread+0x570/0x570 [ 5164.690251] kthread+0x1d1/0x200 [ 5164.691408] ? set_kthread_struct+0x70/0x70 [ 5164.694561] ret_from_fork+0x1f/0x30 [ 5169.119933] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3210s! [ 5169.131882] Showing busy workqueues and worker pools: [ 5169.138316] workqueue events: flags=0x0 [ 5169.143718] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5169.156504] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 5169.165920] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5169.172900] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 5169.185684] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5169.192808] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 5169.199212] workqueue events_long: flags=0x0 [ 5169.206830] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5169.220389] pending: lu_objects_flush [obdclass] [ 5169.223100] workqueue events_unbound: flags=0x2 [ 5169.227451] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5169.232398] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5169.239678] workqueue events_power_efficient: flags=0x80 [ 5169.249717] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5169.254945] pending: neigh_periodic_work [ 5169.260076] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5169.263030] pending: do_cache_clean, check_lifetime [ 5169.267322] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5169.269749] pending: neigh_periodic_work [ 5169.277588] workqueue writeback: flags=0x4a [ 5169.279183] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5169.286556] pending: wb_workfn, wb_workfn [ 5169.294737] workqueue xprtiod: flags=0xa [ 5169.307453] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5169.310572] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 5169.314224] workqueue ipv6_addrconf: flags=0x40008 [ 5169.327675] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5169.334754] pending: addrconf_verify_work [ 5169.340055] workqueue knbd0-recv: flags=0x1a [ 5169.344533] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5169.350790] in-flight: 657:recv_work [ 5169.352273] workqueue ptlrpc_pinger: flags=0x2 [ 5169.356509] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5169.369096] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 5169.376511] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3210s workers=4 idle: 20831 12642 28859 [ 5169.380328] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5159s workers=2 idle: 659 [ 5172.060021] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 5172.063520] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5172.082001] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5172.088447] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5172.097032] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5172.098358] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5172.105032] RSP: 0000:ffffaf9fc15234f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5172.108500] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 5172.111628] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 5172.114154] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5172.116604] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 5172.119218] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 5172.121799] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 5172.125764] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5172.128545] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 5172.131313] Call Trace: [ 5172.132427] [ 5172.133340] ? show_regs.cold.9+0x22/0x2f [ 5172.135291] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5172.137352] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5172.139248] ? watchdog+0x40/0x40 [ 5172.141550] ? hrtimer_interrupt+0x128/0x2c0 [ 5172.144526] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5172.146859] ? apic_timer_interrupt+0xf/0x20 [ 5172.149178] [ 5172.150101] ? native_safe_halt+0x2e/0x50 [ 5172.152356] kvm_wait+0xcd/0xe0 [ 5172.154515] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5172.158096] do_raw_spin_lock+0x177/0x190 [ 5172.161207] _raw_spin_lock+0x21/0x30 [ 5172.163658] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5172.166244] ? do_raw_spin_unlock+0x75/0x190 [ 5172.168561] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5172.171006] ? do_wait_intr+0xf0/0xf0 [ 5172.173365] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5172.176862] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5172.179855] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5172.184145] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5172.186623] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5172.189837] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5172.192588] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5172.195315] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5172.197904] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5172.202171] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5172.204326] lmv_intent_open+0x338/0x1150 [lmv] [ 5172.206775] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5172.209694] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5172.213245] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5172.215000] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5172.217290] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5172.220052] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5172.222217] ? do_raw_spin_unlock+0x75/0x190 [ 5172.223766] ? _raw_spin_unlock+0x12/0x30 [ 5172.225738] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5172.227968] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5172.230231] ll_atomic_open+0x36a/0xd70 [lustre] [ 5172.232548] lookup_open+0x356/0xb40 [ 5172.234486] do_last+0x313/0xfc0 [ 5172.236046] ? nd_jump_root+0xe5/0x160 [ 5172.237748] ? path_init+0x437/0x520 [ 5172.239561] path_openat+0xf7/0x500 [ 5172.240938] ? do_raw_spin_unlock+0x75/0x190 [ 5172.243186] do_filp_open+0x99/0x140 [ 5172.244729] ? getname_flags+0x6e/0x330 [ 5172.246452] ? __check_object_size+0xff/0x256 [ 5172.248031] ? do_raw_spin_unlock+0x75/0x190 [ 5172.249514] ? _raw_spin_unlock+0x12/0x30 [ 5172.251234] do_sys_openat2+0x2b4/0x410 [ 5172.252503] do_sys_open+0x73/0xa0 [ 5172.253621] __x64_sys_openat+0x24/0x30 [ 5172.255207] do_syscall_64+0xc1/0x440 [ 5172.256355] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5172.257652] RIP: 0033:0x7ff54af45332 [ 5172.258761] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5172.266364] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5172.269838] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 5172.273061] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 5172.276434] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 5172.279165] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 5172.281768] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 5184.062074] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [ 5184.072338] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5184.145963] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5184.163323] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5184.180744] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5184.190651] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5184.212389] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5184.220544] RAX: 0000000000007c0d RBX: ffff96848811cae0 RCX: 0000000000000001 [ 5184.229730] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 5184.237566] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5184.244084] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 5184.259273] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 5184.269497] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 5184.285928] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5184.288479] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 5184.306525] Call Trace: [ 5184.309486] [ 5184.313559] ? show_regs.cold.9+0x22/0x2f [ 5184.323924] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5184.329914] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5184.331503] ? watchdog+0x40/0x40 [ 5184.332616] ? hrtimer_interrupt+0x128/0x2c0 [ 5184.344399] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5184.359855] ? apic_timer_interrupt+0xf/0x20 [ 5184.363435] [ 5184.364277] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5184.371962] do_raw_spin_lock+0x177/0x190 [ 5184.374958] _raw_spin_lock+0x21/0x30 [ 5184.376342] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5184.397148] ? do_raw_spin_unlock+0x75/0x190 [ 5184.398765] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5184.413376] ? do_wait_intr+0xf0/0xf0 [ 5184.419308] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5184.425269] mdc_close+0x53b/0xd70 [mdc] [ 5184.427326] lmv_close+0x18d/0x490 [lmv] [ 5184.440240] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 5184.444256] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5184.451338] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5184.458330] ll_md_close+0x274/0x800 [lustre] [ 5184.461452] ll_file_release+0x128/0xc50 [lustre] [ 5184.467235] __fput+0x10d/0x440 [ 5184.470926] ____fput+0x12/0x20 [ 5184.473261] task_work_run+0xc6/0x110 [ 5184.475300] exit_to_usermode_loop+0x1dd/0x1f0 [ 5184.479305] do_syscall_64+0x3d6/0x440 [ 5184.484340] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5184.491302] RIP: 0033:0x7ff54af45cd8 [ 5184.496889] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5184.508840] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5184.512325] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 5184.515835] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 5184.519175] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 5184.522242] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 5184.526830] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5192.060093] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 5192.068448] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5192.089747] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5192.094034] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5192.114454] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5192.118262] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5192.120710] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5192.128337] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5192.130796] RAX: 0000000000007443 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 5192.133395] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 5192.136129] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 5192.138318] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 5192.140900] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 5192.143551] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 5192.146496] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5192.148388] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 5192.150945] Call Trace: [ 5192.151819] [ 5192.152445] ? show_regs.cold.9+0x22/0x2f [ 5192.153867] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5192.155719] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5192.157412] ? watchdog+0x40/0x40 [ 5192.158682] ? hrtimer_interrupt+0x128/0x2c0 [ 5192.160431] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5192.164492] ? apic_timer_interrupt+0xf/0x20 [ 5192.211258] [ 5192.212974] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5192.216733] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5192.220264] do_raw_spin_lock+0x177/0x190 [ 5192.224788] _raw_spin_lock+0x21/0x30 [ 5192.227838] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5192.229966] process_one_work+0x2c8/0x700 [ 5192.233059] worker_thread+0x296/0x6e0 [ 5192.235845] ? rescuer_thread+0x570/0x570 [ 5192.271723] kthread+0x1d1/0x200 [ 5192.284842] ? set_kthread_struct+0x70/0x70 [ 5192.287519] ret_from_fork+0x1f/0x30 [ 5196.056669] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [ 5196.073847] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5196.156574] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5196.185930] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5196.192919] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5196.206935] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5196.255762] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5196.276646] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 5196.293032] RDX: 00000000000075eb RSI: 0000000000000003 RDI: ffff96848811cae0 [ 5196.310616] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 5196.332847] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 5196.350595] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5196.376315] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 5196.389056] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5196.407173] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 5196.409880] Call Trace: [ 5196.434662] [ 5196.435981] ? show_regs.cold.9+0x22/0x2f [ 5196.441633] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5196.456805] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5196.472548] ? watchdog+0x40/0x40 [ 5196.473778] ? hrtimer_interrupt+0x128/0x2c0 [ 5196.481838] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5196.493616] ? apic_timer_interrupt+0xf/0x20 [ 5196.502448] [ 5196.503168] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5196.519545] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5196.534382] do_raw_spin_lock+0x177/0x190 [ 5196.541592] _raw_spin_lock+0x21/0x30 [ 5196.558440] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5196.568675] obd_mod_free+0x46/0x140 [ptlrpc] [ 5196.584484] mdc_commit_open+0x91/0xb0 [mdc] [ 5196.595601] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5196.611692] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5196.617921] after_reply+0xe2c/0x1780 [ptlrpc] [ 5196.622792] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5196.633835] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5196.641623] ? do_wait_intr+0xf0/0xf0 [ 5196.650127] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5196.660248] kthread+0x1d1/0x200 [ 5196.671296] ? set_kthread_struct+0x70/0x70 [ 5196.677464] ret_from_fork+0x1f/0x30 [ 5199.840378] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3240s! [ 5199.850411] Showing busy workqueues and worker pools: [ 5199.851911] workqueue events: flags=0x0 [ 5199.868333] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5199.871490] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 5199.879789] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5199.884753] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 5199.894105] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5199.900632] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 5199.902944] workqueue events_long: flags=0x0 [ 5199.908130] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5199.914738] pending: lu_objects_flush [obdclass] [ 5199.920872] workqueue events_unbound: flags=0x2 [ 5199.924105] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5199.927973] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5199.938718] workqueue events_power_efficient: flags=0x80 [ 5199.942109] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5199.950152] pending: neigh_periodic_work [ 5199.960569] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5199.962893] pending: do_cache_clean, check_lifetime [ 5199.968935] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5199.978714] pending: neigh_periodic_work [ 5199.980216] workqueue writeback: flags=0x4a [ 5199.988146] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5199.990232] pending: wb_workfn, wb_workfn [ 5199.997290] workqueue xprtiod: flags=0xa [ 5199.998688] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5200.013154] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 5200.029395] workqueue ipv6_addrconf: flags=0x40008 [ 5200.032512] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5200.035596] pending: addrconf_verify_work [ 5200.039804] workqueue knbd0-recv: flags=0x1a [ 5200.042901] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5200.045325] in-flight: 657:recv_work [ 5200.050790] workqueue ptlrpc_pinger: flags=0x2 [ 5200.054136] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5200.057726] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 5200.065757] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3241s workers=4 idle: 20831 12642 28859 [ 5200.070002] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 5200.072331] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5190s workers=2 idle: 659 [ 5200.084770] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5200.143099] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5200.156745] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5200.165711] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5200.173455] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5200.185455] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5200.187896] RAX: 00000000000030b2 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 5200.198075] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 5200.200663] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5200.202615] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 5200.209129] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 5200.215197] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 5200.220834] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5200.225327] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 5200.227853] Call Trace: [ 5200.236038] [ 5200.240376] ? show_regs.cold.9+0x22/0x2f [ 5200.247441] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5200.249973] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5200.261527] ? watchdog+0x40/0x40 [ 5200.262483] ? hrtimer_interrupt+0x128/0x2c0 [ 5200.270760] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5200.277219] ? apic_timer_interrupt+0xf/0x20 [ 5200.278685] [ 5200.281210] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5200.291427] do_raw_spin_lock+0x177/0x190 [ 5200.292805] _raw_spin_lock+0x21/0x30 [ 5200.298220] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5200.306462] ? do_raw_spin_unlock+0x75/0x190 [ 5200.308658] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5200.312349] ? do_wait_intr+0xf0/0xf0 [ 5200.315801] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5200.319234] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5200.324092] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5200.336164] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5200.341756] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5200.350726] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5200.357138] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5200.358338] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5200.364360] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5200.370937] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5200.373640] lmv_intent_open+0x338/0x1150 [lmv] [ 5200.375976] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5200.378929] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5200.382268] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5200.384785] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5200.389752] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5200.395389] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5200.405017] ? do_raw_spin_unlock+0x75/0x190 [ 5200.407221] ? _raw_spin_unlock+0x12/0x30 [ 5200.409168] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5200.421015] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5200.424954] ll_atomic_open+0x36a/0xd70 [lustre] [ 5200.427711] lookup_open+0x356/0xb40 [ 5200.430402] do_last+0x313/0xfc0 [ 5200.435759] ? nd_jump_root+0xe5/0x160 [ 5200.437564] ? path_init+0x437/0x520 [ 5200.443371] path_openat+0xf7/0x500 [ 5200.444627] ? do_raw_spin_unlock+0x75/0x190 [ 5200.445875] do_filp_open+0x99/0x140 [ 5200.456285] ? getname_flags+0x6e/0x330 [ 5200.457699] ? __check_object_size+0xff/0x256 [ 5200.469846] ? do_raw_spin_unlock+0x75/0x190 [ 5200.471116] ? _raw_spin_unlock+0x12/0x30 [ 5200.480129] do_sys_openat2+0x2b4/0x410 [ 5200.482738] do_sys_open+0x73/0xa0 [ 5200.491712] __x64_sys_openat+0x24/0x30 [ 5200.492993] do_syscall_64+0xc1/0x440 [ 5200.496099] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5200.510011] RIP: 0033:0x7ff54af45332 [ 5200.511611] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5200.540262] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5200.542864] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 5200.552954] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 5200.556603] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 5200.562220] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 5200.570925] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 5212.073532] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [ 5212.102659] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5212.209573] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5212.226780] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5212.267644] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5212.277086] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5212.319170] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5212.329668] RAX: 000000000000468f RBX: ffff96848811cae0 RCX: 0000000000000001 [ 5212.344963] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 5212.347510] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5212.366360] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 5212.386630] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 5212.400341] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 5212.417079] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5212.431385] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 5212.450898] Call Trace: [ 5212.454308] [ 5212.456986] ? show_regs.cold.9+0x22/0x2f [ 5212.472625] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5212.474401] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5212.496862] ? watchdog+0x40/0x40 [ 5212.507990] ? hrtimer_interrupt+0x128/0x2c0 [ 5212.513029] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5212.523993] ? apic_timer_interrupt+0xf/0x20 [ 5212.534221] [ 5212.538198] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5212.545329] do_raw_spin_lock+0x177/0x190 [ 5212.557089] _raw_spin_lock+0x21/0x30 [ 5212.562302] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5212.567070] ? do_raw_spin_unlock+0x75/0x190 [ 5212.572113] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5212.579287] ? do_wait_intr+0xf0/0xf0 [ 5212.583374] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5212.594236] mdc_close+0x53b/0xd70 [mdc] [ 5212.595237] lmv_close+0x18d/0x490 [lmv] [ 5212.596183] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 5212.613935] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5212.629809] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5212.637997] ll_md_close+0x274/0x800 [lustre] [ 5212.648066] ll_file_release+0x128/0xc50 [lustre] [ 5212.657092] __fput+0x10d/0x440 [ 5212.666236] ____fput+0x12/0x20 [ 5212.671155] task_work_run+0xc6/0x110 [ 5212.680365] exit_to_usermode_loop+0x1dd/0x1f0 [ 5212.693065] do_syscall_64+0x3d6/0x440 [ 5212.703009] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5212.719570] RIP: 0033:0x7ff54af45cd8 [ 5212.726928] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5212.775170] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5212.782481] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 5212.791986] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 5212.794258] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 5212.805389] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 5212.807722] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5220.071865] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 5220.089985] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5220.126368] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5220.132158] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5220.137269] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5220.141385] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5220.144868] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5220.153682] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5220.158946] RAX: 0000000000005030 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 5220.164140] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 5220.168402] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 5220.177427] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 5220.185405] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 5220.192934] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 5220.202597] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5220.209597] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 5220.216613] Call Trace: [ 5220.217730] [ 5220.219738] ? show_regs.cold.9+0x22/0x2f [ 5220.225587] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5220.229062] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5220.231562] ? watchdog+0x40/0x40 [ 5220.233068] ? hrtimer_interrupt+0x128/0x2c0 [ 5220.235056] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5220.237435] ? apic_timer_interrupt+0xf/0x20 [ 5220.239875] [ 5220.240770] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5220.243600] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5220.246472] do_raw_spin_lock+0x177/0x190 [ 5220.248657] _raw_spin_lock+0x21/0x30 [ 5220.250453] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5220.253545] process_one_work+0x2c8/0x700 [ 5220.257726] worker_thread+0x296/0x6e0 [ 5220.259397] ? rescuer_thread+0x570/0x570 [ 5220.261216] kthread+0x1d1/0x200 [ 5220.262583] ? set_kthread_struct+0x70/0x70 [ 5220.264615] ret_from_fork+0x1f/0x30 [ 5228.058323] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [ 5228.067116] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [ 5228.068196] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common [ 5228.083117] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5228.083206] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5228.083213] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5228.083215] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5228.083230] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5228.083235] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5228.083239] RAX: 0000000000003c15 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 5228.083241] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 5228.083244] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5228.083246] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 5228.083248] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 5228.083471] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 5228.083477] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5228.083479] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 5228.083595] Call Trace: [ 5228.083603] [ 5228.083608] ? show_regs.cold.9+0x22/0x2f [ 5228.083618] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5228.083624] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5228.083627] ? watchdog+0x40/0x40 [ 5228.083632] ? hrtimer_interrupt+0x128/0x2c0 [ 5228.083636] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5228.083643] ? apic_timer_interrupt+0xf/0x20 [ 5228.083648] [ 5228.083650] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5228.083657] do_raw_spin_lock+0x177/0x190 [ 5228.083662] _raw_spin_lock+0x21/0x30 [ 5228.083667] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5228.084017] ? do_raw_spin_unlock+0x75/0x190 [ 5228.084023] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5228.084448] ? do_wait_intr+0xf0/0xf0 [ 5228.084457] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5228.085740] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5228.085962] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5228.086174] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5228.112372] sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5228.112408] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5228.112416] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5228.112419] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5228.112433] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5228.112438] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5228.112442] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 5228.112445] RDX: 0000000000007609 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 5228.112447] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 5228.112449] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 5228.112452] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5228.112455] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 5228.112458] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5228.112461] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 5228.112469] Call Trace: [ 5228.112474] [ 5228.112479] ? show_regs.cold.9+0x22/0x2f [ 5228.112599] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5228.112606] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5228.112610] ? watchdog+0x40/0x40 [ 5228.112614] ? hrtimer_interrupt+0x128/0x2c0 [ 5228.112618] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5228.112626] ? apic_timer_interrupt+0xf/0x20 [ 5228.112632] [ 5228.112635] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5228.112640] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5228.112646] do_raw_spin_lock+0x177/0x190 [ 5228.112651] _raw_spin_lock+0x21/0x30 [ 5228.112658] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5228.112943] obd_mod_free+0x46/0x140 [ptlrpc] [ 5228.113143] mdc_commit_open+0x91/0xb0 [mdc] [ 5228.113177] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5228.114240] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5228.114442] after_reply+0xe2c/0x1780 [ptlrpc] [ 5228.114865] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5228.115060] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5228.115276] ? do_wait_intr+0xf0/0xf0 [ 5228.115295] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5228.115845] kthread+0x1d1/0x200 [ 5228.115852] ? set_kthread_struct+0x70/0x70 [ 5228.115855] ret_from_fork+0x1f/0x30 [ 5228.673784] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5228.679141] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5228.684347] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5228.693696] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5228.699061] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5228.705985] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5228.716657] lmv_intent_open+0x338/0x1150 [lmv] [ 5228.722591] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5228.726778] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5228.732609] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5228.735003] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5228.745683] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5228.752103] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5228.758851] ? do_raw_spin_unlock+0x75/0x190 [ 5228.761103] ? _raw_spin_unlock+0x12/0x30 [ 5228.764209] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5228.774432] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5228.783783] ll_atomic_open+0x36a/0xd70 [lustre] [ 5228.795015] lookup_open+0x356/0xb40 [ 5228.798757] do_last+0x313/0xfc0 [ 5228.800521] ? nd_jump_root+0xe5/0x160 [ 5228.806600] ? path_init+0x437/0x520 [ 5228.808977] path_openat+0xf7/0x500 [ 5228.811332] ? do_raw_spin_unlock+0x75/0x190 [ 5228.818826] do_filp_open+0x99/0x140 [ 5228.823402] ? getname_flags+0x6e/0x330 [ 5228.824844] ? __check_object_size+0xff/0x256 [ 5228.826343] ? do_raw_spin_unlock+0x75/0x190 [ 5228.833340] ? _raw_spin_unlock+0x12/0x30 [ 5228.836361] do_sys_openat2+0x2b4/0x410 [ 5228.843975] do_sys_open+0x73/0xa0 [ 5228.851297] __x64_sys_openat+0x24/0x30 [ 5228.854255] do_syscall_64+0xc1/0x440 [ 5228.855509] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5228.864340] RIP: 0033:0x7ff54af45332 [ 5228.870423] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5228.889419] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5228.900869] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 5228.912312] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 5228.927402] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 5228.938502] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 5228.954792] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 5230.560472] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3271s! [ 5230.567042] Showing busy workqueues and worker pools: [ 5230.569995] workqueue events: flags=0x0 [ 5230.575785] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5230.582010] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 5230.587945] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5230.593613] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 5230.602418] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5230.612189] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 5230.619265] workqueue events_long: flags=0x0 [ 5230.622265] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5230.627239] pending: lu_objects_flush [obdclass] [ 5230.633373] workqueue events_unbound: flags=0x2 [ 5230.635471] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5230.641180] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5230.650390] workqueue events_power_efficient: flags=0x80 [ 5230.656430] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5230.659644] pending: neigh_periodic_work [ 5230.663544] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5230.670002] pending: do_cache_clean, check_lifetime [ 5230.677486] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5230.681293] pending: neigh_periodic_work [ 5230.689119] workqueue writeback: flags=0x4a [ 5230.692126] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5230.698338] pending: wb_workfn, wb_workfn [ 5230.701122] workqueue xprtiod: flags=0xa [ 5230.708659] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5230.712567] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 5230.719343] workqueue ipv6_addrconf: flags=0x40008 [ 5230.724210] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5230.731408] pending: addrconf_verify_work [ 5230.734338] workqueue knbd0-recv: flags=0x1a [ 5230.736190] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5230.744618] in-flight: 657:recv_work [ 5230.751348] workqueue ptlrpc_pinger: flags=0x2 [ 5230.758970] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5230.768836] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 5230.772764] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3271s workers=4 idle: 20831 12642 28859 [ 5230.781193] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5221s workers=2 idle: 659 [ 5240.058034] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [ 5240.068591] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5240.133970] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5240.148545] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5240.171207] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5240.177365] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5240.195152] RSP: 0018:ffffaf9fc1543ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5240.208895] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 5240.212743] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 5240.216271] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5240.219531] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 5240.221913] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 5240.225272] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 5240.230856] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5240.242613] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 5240.247855] Call Trace: [ 5240.253219] [ 5240.255492] ? show_regs.cold.9+0x22/0x2f [ 5240.259785] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5240.262525] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5240.264412] ? watchdog+0x40/0x40 [ 5240.265846] ? hrtimer_interrupt+0x128/0x2c0 [ 5240.267390] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5240.270545] ? apic_timer_interrupt+0xf/0x20 [ 5240.271968] [ 5240.273405] ? native_safe_halt+0x2e/0x50 [ 5240.275301] kvm_wait+0xcd/0xe0 [ 5240.278089] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5240.281467] do_raw_spin_lock+0x177/0x190 [ 5240.283863] _raw_spin_lock+0x21/0x30 [ 5240.285720] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5240.290057] ? do_raw_spin_unlock+0x75/0x190 [ 5240.292870] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5240.297159] ? do_wait_intr+0xf0/0xf0 [ 5240.302543] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5240.308810] mdc_close+0x53b/0xd70 [mdc] [ 5240.318340] lmv_close+0x18d/0x490 [lmv] [ 5240.323822] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 5240.327188] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5240.331407] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5240.333191] ll_md_close+0x274/0x800 [lustre] [ 5240.336671] ll_file_release+0x128/0xc50 [lustre] [ 5240.339868] __fput+0x10d/0x440 [ 5240.344035] ____fput+0x12/0x20 [ 5240.348095] task_work_run+0xc6/0x110 [ 5240.355893] exit_to_usermode_loop+0x1dd/0x1f0 [ 5240.357521] do_syscall_64+0x3d6/0x440 [ 5240.366252] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5240.368292] RIP: 0033:0x7ff54af45cd8 [ 5240.375823] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5240.392866] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5240.403862] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 5240.414351] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 5240.431757] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 5240.444305] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 5240.459084] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5248.063082] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 5248.066930] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5248.153019] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5248.181508] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5248.189468] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5248.201628] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5248.205092] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5248.230075] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5248.244664] RAX: 0000000000004889 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 5248.259598] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 5248.274926] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 5248.282770] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 5248.289639] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 5248.305211] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 5248.319590] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5248.323570] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 5248.329103] Call Trace: [ 5248.331503] [ 5248.332634] ? show_regs.cold.9+0x22/0x2f [ 5248.341534] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5248.349336] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5248.353250] ? watchdog+0x40/0x40 [ 5248.357680] ? hrtimer_interrupt+0x128/0x2c0 [ 5248.358813] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5248.370469] ? apic_timer_interrupt+0xf/0x20 [ 5248.372286] [ 5248.373364] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5248.382082] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5248.392943] do_raw_spin_lock+0x177/0x190 [ 5248.397013] _raw_spin_lock+0x21/0x30 [ 5248.398937] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5248.401122] process_one_work+0x2c8/0x700 [ 5248.404744] worker_thread+0x296/0x6e0 [ 5248.407235] ? rescuer_thread+0x570/0x570 [ 5248.410153] kthread+0x1d1/0x200 [ 5248.412456] ? set_kthread_struct+0x70/0x70 [ 5248.414731] ret_from_fork+0x1f/0x30 [ 5253.936640] rcu: INFO: rcu_sched self-detected stall on CPU [ 5253.939107] rcu: 0-...!: (1708804 ticks this GP) idle=98a/1/0x4000000000000002 softirq=125154/125154 fqs=959 [ 5253.943848] (t=3300096 jiffies g=253585 q=3862) [ 5253.946042] rcu: rcu_sched kthread starved for 3295056 jiffies! g253585 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [ 5253.951277] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 5253.956227] rcu: RCU grace-period kthread stack dump: [ 5253.959830] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 5253.967007] Call Trace: [ 5253.968991] __schedule+0x351/0xcb0 [ 5253.971317] schedule+0xc0/0x180 [ 5253.974060] schedule_timeout+0xb4/0x190 [ 5253.976329] ? do_raw_spin_unlock+0x75/0x190 [ 5253.978722] ? __next_timer_interrupt+0x160/0x160 [ 5253.981219] ? prepare_to_swait_event+0xf3/0x140 [ 5253.983432] rcu_gp_kthread+0x8dd/0x15b0 [ 5253.984964] ? force_qs_rnp+0x2c0/0x2c0 [ 5253.986643] kthread+0x1d1/0x200 [ 5253.987881] ? set_kthread_struct+0x70/0x70 [ 5253.990416] ret_from_fork+0x1f/0x30 [ 5253.992654] rcu: Stack dump where RCU GP kthread last ran: [ 5253.995436] NMI backtrace for cpu 0 [ 5253.997439] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5254.003357] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5254.008642] Call Trace: [ 5254.010056] [ 5254.011037] ? dump_stack+0xbb/0x10e [ 5254.013352] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 5254.016114] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 5254.018843] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 5254.021654] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 5254.024848] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 5254.027824] ? rcu_check_gp_kthread_starvation+0x16f/0x188 [ 5254.030260] ? rcu_sched_clock_irq.cold.51+0xce/0x307 [ 5254.033042] ? update_process_times+0x6d/0xc0 [ 5254.035146] ? tick_sched_handle.isra.18+0x27/0x90 [ 5254.037237] ? tick_sched_timer+0x53/0xb0 [ 5254.039268] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5254.041338] ? tick_sched_handle.isra.18+0x90/0x90 [ 5254.043484] ? hrtimer_interrupt+0x128/0x2c0 [ 5254.044937] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5254.047503] ? apic_timer_interrupt+0xf/0x20 [ 5254.055171] [ 5254.061227] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5254.069278] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5254.081044] ? do_raw_spin_lock+0x177/0x190 [ 5254.082690] ? _raw_spin_lock+0x21/0x30 [ 5254.088674] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5254.092973] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 5254.096153] ? mdc_commit_open+0x91/0xb0 [mdc] [ 5254.098915] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5254.101972] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5254.105177] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 5254.108152] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5254.111408] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5254.114209] ? do_wait_intr+0xf0/0xf0 [ 5254.116824] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5254.120385] ? kthread+0x1d1/0x200 [ 5254.122519] ? set_kthread_struct+0x70/0x70 [ 5254.125905] ? ret_from_fork+0x1f/0x30 [ 5254.128166] NMI backtrace for cpu 0 [ 5254.129871] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5254.135096] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5254.140846] Call Trace: [ 5254.142036] [ 5254.144196] ? dump_stack+0xbb/0x10e [ 5254.149002] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 5254.155894] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 5254.162120] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 5254.167691] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 5254.173518] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 5254.177519] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 5254.181832] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 5254.185389] ? update_process_times+0x6d/0xc0 [ 5254.187821] ? tick_sched_handle.isra.18+0x27/0x90 [ 5254.190579] ? tick_sched_timer+0x53/0xb0 [ 5254.193264] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5254.195831] ? tick_sched_handle.isra.18+0x90/0x90 [ 5254.198421] ? hrtimer_interrupt+0x128/0x2c0 [ 5254.200892] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5254.203633] ? apic_timer_interrupt+0xf/0x20 [ 5254.206214] [ 5254.207470] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5254.211534] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5254.215386] ? do_raw_spin_lock+0x177/0x190 [ 5254.219008] ? _raw_spin_lock+0x21/0x30 [ 5254.222124] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5254.225823] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 5254.227843] ? mdc_commit_open+0x91/0xb0 [mdc] [ 5254.229499] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5254.232858] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5254.236051] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 5254.239143] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5254.247919] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5254.252191] ? do_wait_intr+0xf0/0xf0 [ 5254.254869] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5254.258135] ? kthread+0x1d1/0x200 [ 5254.261020] ? set_kthread_struct+0x70/0x70 [ 5254.264255] ? ret_from_fork+0x1f/0x30 [ 5256.062650] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [ 5256.073036] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5256.115941] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5256.125057] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5256.133496] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5256.142842] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5256.159144] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5256.167756] RAX: 0000000000003b1c RBX: ffff96848811cae0 RCX: 0000000000000001 [ 5256.176688] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 5256.184965] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5256.188801] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 5256.194162] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 5256.199470] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 5256.202655] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5256.205510] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 5256.210122] Call Trace: [ 5256.211207] [ 5256.212532] ? show_regs.cold.9+0x22/0x2f [ 5256.214181] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5256.216409] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5256.218504] ? watchdog+0x40/0x40 [ 5256.220086] ? hrtimer_interrupt+0x128/0x2c0 [ 5256.221860] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5256.223617] ? apic_timer_interrupt+0xf/0x20 [ 5256.225680] [ 5256.226468] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5256.228134] do_raw_spin_lock+0x177/0x190 [ 5256.229319] _raw_spin_lock+0x21/0x30 [ 5256.230327] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5256.232615] ? do_raw_spin_unlock+0x75/0x190 [ 5256.234047] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5256.235966] ? do_wait_intr+0xf0/0xf0 [ 5256.237501] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5256.240169] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5256.243006] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5256.246043] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5256.248973] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5256.251330] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5256.253387] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5256.254775] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5256.256477] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5256.258524] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5256.261049] lmv_intent_open+0x338/0x1150 [lmv] [ 5256.264159] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5256.268091] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5256.271166] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5256.273105] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5256.275123] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5256.277227] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5256.279108] ? do_raw_spin_unlock+0x75/0x190 [ 5256.280758] ? _raw_spin_unlock+0x12/0x30 [ 5256.282230] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5256.285017] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5256.286873] ll_atomic_open+0x36a/0xd70 [lustre] [ 5256.288680] lookup_open+0x356/0xb40 [ 5256.290165] do_last+0x313/0xfc0 [ 5256.291260] ? nd_jump_root+0xe5/0x160 [ 5256.292649] ? path_init+0x437/0x520 [ 5256.293524] path_openat+0xf7/0x500 [ 5256.294207] ? do_raw_spin_unlock+0x75/0x190 [ 5256.295517] do_filp_open+0x99/0x140 [ 5256.296176] ? getname_flags+0x6e/0x330 [ 5256.297457] ? __check_object_size+0xff/0x256 [ 5256.298513] ? do_raw_spin_unlock+0x75/0x190 [ 5256.299597] ? _raw_spin_unlock+0x12/0x30 [ 5256.303821] do_sys_openat2+0x2b4/0x410 [ 5256.304819] do_sys_open+0x73/0xa0 [ 5256.305754] __x64_sys_openat+0x24/0x30 [ 5256.306849] do_syscall_64+0xc1/0x440 [ 5256.308088] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5256.310107] RIP: 0033:0x7ff54af45332 [ 5256.312036] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5256.322080] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5256.325974] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 5256.329340] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 5256.333369] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 5256.338016] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 5256.341259] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 5261.279179] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3302s! [ 5261.283855] Showing busy workqueues and worker pools: [ 5261.287349] workqueue events: flags=0x0 [ 5261.289637] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5261.294306] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 5261.304443] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5261.314860] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 5261.327907] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5261.337113] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 5261.343030] workqueue events_long: flags=0x0 [ 5261.346365] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5261.349451] pending: lu_objects_flush [obdclass] [ 5261.354247] workqueue events_unbound: flags=0x2 [ 5261.356996] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5261.360655] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5261.366960] workqueue events_power_efficient: flags=0x80 [ 5261.370270] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5261.378638] pending: neigh_periodic_work [ 5261.380322] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5261.385586] pending: do_cache_clean, check_lifetime [ 5261.395873] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5261.403988] pending: neigh_periodic_work [ 5261.406537] workqueue writeback: flags=0x4a [ 5261.410438] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5261.412848] pending: wb_workfn, wb_workfn [ 5261.420419] workqueue xprtiod: flags=0xa [ 5261.425197] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5261.430226] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 5261.448577] workqueue ipv6_addrconf: flags=0x40008 [ 5261.451352] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5261.455209] pending: addrconf_verify_work [ 5261.460277] workqueue knbd0-recv: flags=0x1a [ 5261.462231] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5261.470247] in-flight: 657:recv_work [ 5261.471771] workqueue ptlrpc_pinger: flags=0x2 [ 5261.476516] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5261.483567] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 5261.489154] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3302s workers=4 idle: 20831 12642 28859 [ 5261.491684] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5251s workers=2 idle: 659 [ 5268.062947] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [ 5268.067976] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5268.113756] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5268.120760] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5268.125519] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5268.127743] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5268.135911] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5268.139434] RAX: 0000000000003185 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 5268.142935] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 5268.146568] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5268.150530] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 5268.154520] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 5268.158435] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 5268.162671] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5268.165817] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 5268.169743] Call Trace: [ 5268.170677] [ 5268.171348] ? show_regs.cold.9+0x22/0x2f [ 5268.173095] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5268.176051] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5268.178538] ? watchdog+0x40/0x40 [ 5268.180447] ? hrtimer_interrupt+0x128/0x2c0 [ 5268.182057] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5268.184888] ? apic_timer_interrupt+0xf/0x20 [ 5268.187469] [ 5268.188671] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5268.191680] do_raw_spin_lock+0x177/0x190 [ 5268.194287] _raw_spin_lock+0x21/0x30 [ 5268.196417] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5268.199973] ? do_raw_spin_unlock+0x75/0x190 [ 5268.202942] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5268.206605] ? do_wait_intr+0xf0/0xf0 [ 5268.208715] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5268.212496] mdc_close+0x53b/0xd70 [mdc] [ 5268.214886] lmv_close+0x18d/0x490 [lmv] [ 5268.217432] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 5268.220864] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5268.224080] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5268.227513] ll_md_close+0x274/0x800 [lustre] [ 5268.230440] ll_file_release+0x128/0xc50 [lustre] [ 5268.233066] __fput+0x10d/0x440 [ 5268.235394] ____fput+0x12/0x20 [ 5268.236951] task_work_run+0xc6/0x110 [ 5268.239093] exit_to_usermode_loop+0x1dd/0x1f0 [ 5268.241986] do_syscall_64+0x3d6/0x440 [ 5268.244485] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5268.247463] RIP: 0033:0x7ff54af45cd8 [ 5268.249845] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5268.259859] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5268.265585] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 5268.269867] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 5268.274110] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 5268.278940] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 5268.283148] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5276.059296] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 5276.063151] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5276.093948] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5276.099364] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5276.103946] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5276.107001] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5276.109554] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5276.119002] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5276.123032] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 5276.126750] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 5276.129588] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 5276.132355] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 5276.135734] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 5276.142074] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 5276.148549] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5276.154457] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 5276.160709] Call Trace: [ 5276.161831] [ 5276.162998] ? show_regs.cold.9+0x22/0x2f [ 5276.164964] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5276.167702] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5276.169904] ? watchdog+0x40/0x40 [ 5276.171689] ? hrtimer_interrupt+0x128/0x2c0 [ 5276.173726] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5276.176045] ? apic_timer_interrupt+0xf/0x20 [ 5276.177948] [ 5276.178953] ? native_safe_halt+0x2e/0x50 [ 5276.180605] kvm_wait+0xcd/0xe0 [ 5276.182000] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5276.184281] do_raw_spin_lock+0x177/0x190 [ 5276.185899] _raw_spin_lock+0x21/0x30 [ 5276.187572] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5276.189918] process_one_work+0x2c8/0x700 [ 5276.191920] worker_thread+0x296/0x6e0 [ 5276.193683] ? rescuer_thread+0x570/0x570 [ 5276.195604] kthread+0x1d1/0x200 [ 5276.197003] ? set_kthread_struct+0x70/0x70 [ 5276.199005] ret_from_fork+0x1f/0x30 [ 5284.060670] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [ 5284.068897] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5284.096982] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5284.101885] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5284.105867] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5284.107975] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5284.120514] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 5284.125347] RAX: 0000000000007f90 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 5284.132626] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 5284.136851] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5284.139776] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 5284.142867] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 5284.145562] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 5284.149756] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5284.154804] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 5284.162680] Call Trace: [ 5284.163932] [ 5284.165525] ? show_regs.cold.9+0x22/0x2f [ 5284.168607] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5284.171560] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5284.180516] ? watchdog+0x40/0x40 [ 5284.187190] ? hrtimer_interrupt+0x128/0x2c0 [ 5284.190829] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5284.193622] ? apic_timer_interrupt+0xf/0x20 [ 5284.196330] [ 5284.197285] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5284.200385] do_raw_spin_lock+0x177/0x190 [ 5284.203120] _raw_spin_lock+0x21/0x30 [ 5284.205105] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5284.208664] ? do_raw_spin_unlock+0x75/0x190 [ 5284.211728] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5284.215223] ? do_wait_intr+0xf0/0xf0 [ 5284.217052] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5284.220315] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5284.223045] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5284.226015] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5284.229356] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5284.232635] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5284.235267] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5284.237511] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5284.240221] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5284.243658] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5284.246621] lmv_intent_open+0x338/0x1150 [lmv] [ 5284.249005] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5284.251910] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5284.254359] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5284.256833] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5284.259681] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5284.263719] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5284.266200] ? do_raw_spin_unlock+0x75/0x190 [ 5284.268840] ? _raw_spin_unlock+0x12/0x30 [ 5284.271360] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5284.274719] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5284.277787] ll_atomic_open+0x36a/0xd70 [lustre] [ 5284.280836] lookup_open+0x356/0xb40 [ 5284.283618] do_last+0x313/0xfc0 [ 5284.285366] ? nd_jump_root+0xe5/0x160 [ 5284.287696] ? path_init+0x437/0x520 [ 5284.289656] path_openat+0xf7/0x500 [ 5284.291768] ? do_raw_spin_unlock+0x75/0x190 [ 5284.295264] do_filp_open+0x99/0x140 [ 5284.298094] ? getname_flags+0x6e/0x330 [ 5284.301092] ? __check_object_size+0xff/0x256 [ 5284.305758] ? do_raw_spin_unlock+0x75/0x190 [ 5284.312790] ? _raw_spin_unlock+0x12/0x30 [ 5284.314239] do_sys_openat2+0x2b4/0x410 [ 5284.317834] do_sys_open+0x73/0xa0 [ 5284.321816] __x64_sys_openat+0x24/0x30 [ 5284.324293] do_syscall_64+0xc1/0x440 [ 5284.326866] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5284.329699] RIP: 0033:0x7ff54af45332 [ 5284.336709] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5284.360105] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5284.367309] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 5284.371864] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 5284.376308] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 5284.379822] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 5284.382344] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 5288.055960] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 5288.068197] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5288.146441] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5288.154823] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5288.175011] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5288.181897] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5288.205102] RSP: 0000:ffffaf9fc1b97bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5288.212155] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 5288.220604] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 5288.226904] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 5288.232707] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 5288.237472] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5288.243226] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 5288.248707] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5288.253322] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 5288.258455] Call Trace: [ 5288.260388] [ 5288.261125] ? show_regs.cold.9+0x22/0x2f [ 5288.262726] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5288.271808] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5288.276201] ? watchdog+0x40/0x40 [ 5288.277627] ? hrtimer_interrupt+0x128/0x2c0 [ 5288.280879] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5288.286358] ? apic_timer_interrupt+0xf/0x20 [ 5288.289412] [ 5288.291327] ? native_safe_halt+0x2e/0x50 [ 5288.294301] kvm_wait+0xcd/0xe0 [ 5288.296424] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5288.303586] do_raw_spin_lock+0x177/0x190 [ 5288.313216] _raw_spin_lock+0x21/0x30 [ 5288.314648] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5288.318974] obd_mod_free+0x46/0x140 [ptlrpc] [ 5288.322859] mdc_commit_open+0x91/0xb0 [mdc] [ 5288.325875] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5288.333327] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5288.340490] after_reply+0xe2c/0x1780 [ptlrpc] [ 5288.344804] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5288.348463] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5288.352148] ? do_wait_intr+0xf0/0xf0 [ 5288.355313] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5288.359560] kthread+0x1d1/0x200 [ 5288.362308] ? set_kthread_struct+0x70/0x70 [ 5288.365288] ret_from_fork+0x1f/0x30 [ 5291.999128] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3333s! [ 5292.003587] Showing busy workqueues and worker pools: [ 5292.006584] workqueue events: flags=0x0 [ 5292.008795] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5292.013126] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 5292.018513] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5292.023874] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 5292.033236] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5292.038477] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 5292.042952] workqueue events_long: flags=0x0 [ 5292.045559] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5292.049177] pending: lu_objects_flush [obdclass] [ 5292.052439] workqueue events_unbound: flags=0x2 [ 5292.054620] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5292.058556] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5292.067309] workqueue events_power_efficient: flags=0x80 [ 5292.071365] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5292.075767] pending: neigh_periodic_work [ 5292.082061] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5292.089450] pending: do_cache_clean, check_lifetime [ 5292.092229] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5292.096538] pending: neigh_periodic_work [ 5292.099799] workqueue writeback: flags=0x4a [ 5292.103985] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5292.110076] pending: wb_workfn, wb_workfn [ 5292.112557] workqueue xprtiod: flags=0xa [ 5292.115893] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5292.119968] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 5292.125817] workqueue ipv6_addrconf: flags=0x40008 [ 5292.129280] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5292.132842] pending: addrconf_verify_work [ 5292.135371] workqueue knbd0-recv: flags=0x1a [ 5292.147397] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5292.154728] in-flight: 657:recv_work [ 5292.159146] workqueue ptlrpc_pinger: flags=0x2 [ 5292.162973] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5292.167391] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 5292.174625] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3333s workers=4 idle: 20831 12642 28859 [ 5292.178884] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5282s workers=2 idle: 659 [ 5296.060429] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [ 5296.073074] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5296.117071] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5296.129855] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5296.141563] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5296.150115] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5296.173657] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 5296.178140] RAX: 0000000000007a87 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 5296.182208] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 5296.189563] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5296.194261] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 5296.207707] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 5296.218637] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 5296.227269] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5296.235082] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 5296.240445] Call Trace: [ 5296.242227] [ 5296.243505] ? show_regs.cold.9+0x22/0x2f [ 5296.245473] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5296.248647] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5296.251187] ? watchdog+0x40/0x40 [ 5296.253424] ? hrtimer_interrupt+0x128/0x2c0 [ 5296.259545] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5296.264665] ? apic_timer_interrupt+0xf/0x20 [ 5296.268406] [ 5296.270006] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5296.274397] do_raw_spin_lock+0x177/0x190 [ 5296.277241] _raw_spin_lock+0x21/0x30 [ 5296.278346] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5296.283431] ? do_raw_spin_unlock+0x75/0x190 [ 5296.288339] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5296.291072] ? do_wait_intr+0xf0/0xf0 [ 5296.294245] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5296.299630] mdc_close+0x53b/0xd70 [mdc] [ 5296.302619] lmv_close+0x18d/0x490 [lmv] [ 5296.305217] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 5296.309025] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5296.312195] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5296.314994] ll_md_close+0x274/0x800 [lustre] [ 5296.317568] ll_file_release+0x128/0xc50 [lustre] [ 5296.320205] __fput+0x10d/0x440 [ 5296.321662] ____fput+0x12/0x20 [ 5296.323533] task_work_run+0xc6/0x110 [ 5296.325568] exit_to_usermode_loop+0x1dd/0x1f0 [ 5296.327855] do_syscall_64+0x3d6/0x440 [ 5296.330275] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5296.333255] RIP: 0033:0x7ff54af45cd8 [ 5296.335466] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5296.346874] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5296.350806] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 5296.355218] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 5296.359569] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 5296.364428] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 5296.372618] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5304.060771] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 5304.067355] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5304.139889] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5304.152956] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5304.162968] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5304.168796] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5304.175685] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5304.206922] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5304.209937] RAX: 000000000000708a RBX: ffff96848811cae0 RCX: 0000000000000001 [ 5304.217750] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 5304.221796] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 5304.236256] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 5304.247431] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 5304.253256] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 5304.260525] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5304.266040] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 5304.274923] Call Trace: [ 5304.280886] [ 5304.284755] ? show_regs.cold.9+0x22/0x2f [ 5304.298083] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5304.312104] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5304.316802] ? watchdog+0x40/0x40 [ 5304.322660] ? hrtimer_interrupt+0x128/0x2c0 [ 5304.332751] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5304.343154] ? apic_timer_interrupt+0xf/0x20 [ 5304.344287] [ 5304.348741] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5304.357552] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5304.361453] do_raw_spin_lock+0x177/0x190 [ 5304.368995] _raw_spin_lock+0x21/0x30 [ 5304.379000] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5304.380847] process_one_work+0x2c8/0x700 [ 5304.386260] worker_thread+0x296/0x6e0 [ 5304.394779] ? rescuer_thread+0x570/0x570 [ 5304.401579] kthread+0x1d1/0x200 [ 5304.404486] ? set_kthread_struct+0x70/0x70 [ 5304.408882] ret_from_fork+0x1f/0x30 [ 5312.061336] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [ 5312.072411] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5312.127808] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5312.146823] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5312.158702] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5312.167863] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5312.197697] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5312.209617] RAX: 0000000000004584 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 5312.213824] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 5312.221549] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5312.227089] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 5312.234531] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 5312.241764] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 5312.248045] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5312.253732] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 5312.261432] Call Trace: [ 5312.263760] [ 5312.265510] ? show_regs.cold.9+0x22/0x2f [ 5312.268218] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5312.272004] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5312.275545] ? watchdog+0x40/0x40 [ 5312.276781] ? hrtimer_interrupt+0x128/0x2c0 [ 5312.280651] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5312.283297] ? apic_timer_interrupt+0xf/0x20 [ 5312.286110] [ 5312.287040] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5312.291962] do_raw_spin_lock+0x177/0x190 [ 5312.293315] _raw_spin_lock+0x21/0x30 [ 5312.297053] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5312.303739] ? do_raw_spin_unlock+0x75/0x190 [ 5312.307102] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5312.311076] ? do_wait_intr+0xf0/0xf0 [ 5312.315969] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5312.323100] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5312.326276] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5312.333809] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5312.341556] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5312.354465] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5312.356145] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5312.364819] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5312.374127] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5312.378614] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5312.383464] lmv_intent_open+0x338/0x1150 [lmv] [ 5312.386703] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5312.393095] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5312.398004] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5312.400798] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5312.407715] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5312.412696] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5312.418598] ? do_raw_spin_unlock+0x75/0x190 [ 5312.423108] ? _raw_spin_unlock+0x12/0x30 [ 5312.425313] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5312.432484] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5312.434349] ll_atomic_open+0x36a/0xd70 [lustre] [ 5312.444609] lookup_open+0x356/0xb40 [ 5312.447788] do_last+0x313/0xfc0 [ 5312.449868] ? nd_jump_root+0xe5/0x160 [ 5312.452291] ? path_init+0x437/0x520 [ 5312.463699] path_openat+0xf7/0x500 [ 5312.468143] ? do_raw_spin_unlock+0x75/0x190 [ 5312.474038] do_filp_open+0x99/0x140 [ 5312.477692] ? getname_flags+0x6e/0x330 [ 5312.481790] ? __check_object_size+0xff/0x256 [ 5312.486529] ? do_raw_spin_unlock+0x75/0x190 [ 5312.490029] ? _raw_spin_unlock+0x12/0x30 [ 5312.493888] do_sys_openat2+0x2b4/0x410 [ 5312.497029] do_sys_open+0x73/0xa0 [ 5312.501241] __x64_sys_openat+0x24/0x30 [ 5312.502618] do_syscall_64+0xc1/0x440 [ 5312.511958] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5312.517518] RIP: 0033:0x7ff54af45332 [ 5312.524109] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5312.535810] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5312.541894] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 5312.547813] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 5312.552825] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 5312.556985] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 5312.560950] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 5320.055038] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 5320.066308] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5320.125395] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5320.132510] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5320.136721] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5320.140612] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5320.155376] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5320.164510] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 5320.170390] RDX: 0000000000000aa0 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 5320.175048] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 5320.181460] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 5320.185063] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5320.189081] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 5320.196201] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5320.203488] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 5320.209141] Call Trace: [ 5320.210458] [ 5320.212115] ? show_regs.cold.9+0x22/0x2f [ 5320.215032] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5320.217768] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5320.221274] ? watchdog+0x40/0x40 [ 5320.223584] ? hrtimer_interrupt+0x128/0x2c0 [ 5320.225489] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5320.229908] ? apic_timer_interrupt+0xf/0x20 [ 5320.236294] [ 5320.237617] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5320.245097] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5320.254536] do_raw_spin_lock+0x177/0x190 [ 5320.260581] _raw_spin_lock+0x21/0x30 [ 5320.263504] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5320.270699] obd_mod_free+0x46/0x140 [ptlrpc] [ 5320.276185] mdc_commit_open+0x91/0xb0 [mdc] [ 5320.286157] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5320.303437] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5320.314359] after_reply+0xe2c/0x1780 [ptlrpc] [ 5320.320416] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5320.322807] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5320.334952] ? do_wait_intr+0xf0/0xf0 [ 5320.336530] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5320.359188] kthread+0x1d1/0x200 [ 5320.360385] ? set_kthread_struct+0x70/0x70 [ 5320.363295] ret_from_fork+0x1f/0x30 [ 5322.212202] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3363s! [ 5322.218517] Showing busy workqueues and worker pools: [ 5322.220090] workqueue events: flags=0x0 [ 5322.227229] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5322.233363] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 5322.241177] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5322.254156] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 5322.274085] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5322.288432] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 5322.299903] workqueue events_long: flags=0x0 [ 5322.305387] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5322.311934] pending: lu_objects_flush [obdclass] [ 5322.320186] workqueue events_unbound: flags=0x2 [ 5322.324508] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5322.329142] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5322.341042] workqueue events_power_efficient: flags=0x80 [ 5322.345966] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5322.350037] pending: neigh_periodic_work [ 5322.354225] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5322.365022] pending: do_cache_clean, check_lifetime [ 5322.369442] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5322.374702] pending: neigh_periodic_work [ 5322.377383] workqueue writeback: flags=0x4a [ 5322.383442] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5322.388278] pending: wb_workfn, wb_workfn [ 5322.391327] workqueue xprtiod: flags=0xa [ 5322.394331] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5322.398141] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 5322.405559] workqueue ipv6_addrconf: flags=0x40008 [ 5322.407531] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5322.416160] pending: addrconf_verify_work [ 5322.420396] workqueue knbd0-recv: flags=0x1a [ 5322.426397] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5322.437340] in-flight: 657:recv_work [ 5322.440387] workqueue ptlrpc_pinger: flags=0x2 [ 5322.448174] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5322.454253] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 5322.462651] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3363s workers=4 idle: 20831 12642 28859 [ 5322.472048] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5312s workers=2 idle: 659 [ 5324.059040] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 5324.063319] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5324.119350] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5324.132556] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5324.141037] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5324.146939] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5324.166242] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5324.174972] RAX: 0000000000000412 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 5324.182336] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 5324.189506] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5324.197880] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 5324.203414] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 5324.209100] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 5324.217125] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5324.221309] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 5324.226512] Call Trace: [ 5324.228216] [ 5324.229193] ? show_regs.cold.9+0x22/0x2f [ 5324.231520] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5324.233945] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5324.237238] ? watchdog+0x40/0x40 [ 5324.240373] ? hrtimer_interrupt+0x128/0x2c0 [ 5324.243523] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5324.246363] ? apic_timer_interrupt+0xf/0x20 [ 5324.249584] [ 5324.252559] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5324.257492] do_raw_spin_lock+0x177/0x190 [ 5324.260972] _raw_spin_lock+0x21/0x30 [ 5324.263881] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5324.266768] ? do_raw_spin_unlock+0x75/0x190 [ 5324.268336] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5324.272257] ? do_wait_intr+0xf0/0xf0 [ 5324.274367] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5324.278432] mdc_close+0x53b/0xd70 [mdc] [ 5324.281303] lmv_close+0x18d/0x490 [lmv] [ 5324.284426] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 5324.287554] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5324.291877] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5324.297513] ll_md_close+0x274/0x800 [lustre] [ 5324.301954] ll_file_release+0x128/0xc50 [lustre] [ 5324.308538] __fput+0x10d/0x440 [ 5324.312437] ____fput+0x12/0x20 [ 5324.313508] task_work_run+0xc6/0x110 [ 5324.314709] exit_to_usermode_loop+0x1dd/0x1f0 [ 5324.317890] do_syscall_64+0x3d6/0x440 [ 5324.325541] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5324.330939] RIP: 0033:0x7ff54af45cd8 [ 5324.333192] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5324.349823] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5324.355266] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 5324.360531] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 5324.363587] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 5324.368913] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 5324.381020] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5332.062239] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 5332.075886] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5332.126551] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5332.141799] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5332.150151] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5332.157059] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5332.168345] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5332.197015] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 5332.204749] RAX: 0000000000002478 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 5332.208963] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 5332.215977] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 5332.224093] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 5332.231653] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 5332.237212] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 5332.244044] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5332.249728] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 5332.258549] Call Trace: [ 5332.259853] [ 5332.261636] ? show_regs.cold.9+0x22/0x2f [ 5332.265460] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5332.271803] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5332.280160] ? watchdog+0x40/0x40 [ 5332.285491] ? hrtimer_interrupt+0x128/0x2c0 [ 5332.292700] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5332.298651] ? apic_timer_interrupt+0xf/0x20 [ 5332.304915] [ 5332.306877] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5332.312900] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5332.318259] do_raw_spin_lock+0x177/0x190 [ 5332.321413] _raw_spin_lock+0x21/0x30 [ 5332.326188] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5332.332961] process_one_work+0x2c8/0x700 [ 5332.337183] worker_thread+0x296/0x6e0 [ 5332.342615] ? rescuer_thread+0x570/0x570 [ 5332.344689] kthread+0x1d1/0x200 [ 5332.350427] ? set_kthread_struct+0x70/0x70 [ 5332.354963] ret_from_fork+0x1f/0x30 [ 5340.060052] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [ 5340.063949] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5340.116163] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5340.122299] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5340.126938] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5340.137553] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5340.154246] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5340.158343] RAX: 0000000000005422 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 5340.161847] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 5340.165756] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5340.170053] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 5340.177561] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 5340.182424] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 5340.187611] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5340.195724] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 5340.202588] Call Trace: [ 5340.206271] [ 5340.207824] ? show_regs.cold.9+0x22/0x2f [ 5340.210760] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5340.217956] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5340.223771] ? watchdog+0x40/0x40 [ 5340.226344] ? hrtimer_interrupt+0x128/0x2c0 [ 5340.231342] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5340.236009] ? apic_timer_interrupt+0xf/0x20 [ 5340.239193] [ 5340.240681] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5340.244938] do_raw_spin_lock+0x177/0x190 [ 5340.249376] _raw_spin_lock+0x21/0x30 [ 5340.251969] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5340.255936] ? do_raw_spin_unlock+0x75/0x190 [ 5340.260838] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5340.265607] ? do_wait_intr+0xf0/0xf0 [ 5340.268131] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5340.275954] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5340.280068] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5340.285036] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5340.289436] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5340.293559] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5340.297923] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5340.303990] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5340.307751] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5340.315520] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5340.320255] lmv_intent_open+0x338/0x1150 [lmv] [ 5340.323992] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5340.331039] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5340.338890] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5340.344250] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5340.349737] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5340.354619] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5340.358308] ? do_raw_spin_unlock+0x75/0x190 [ 5340.361249] ? _raw_spin_unlock+0x12/0x30 [ 5340.362597] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5340.366138] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5340.370397] ll_atomic_open+0x36a/0xd70 [lustre] [ 5340.374686] lookup_open+0x356/0xb40 [ 5340.377185] do_last+0x313/0xfc0 [ 5340.379704] ? nd_jump_root+0xe5/0x160 [ 5340.383466] ? path_init+0x437/0x520 [ 5340.387412] path_openat+0xf7/0x500 [ 5340.393840] ? do_raw_spin_unlock+0x75/0x190 [ 5340.397355] do_filp_open+0x99/0x140 [ 5340.402409] ? getname_flags+0x6e/0x330 [ 5340.406155] ? __check_object_size+0xff/0x256 [ 5340.408872] ? do_raw_spin_unlock+0x75/0x190 [ 5340.410986] ? _raw_spin_unlock+0x12/0x30 [ 5340.413127] do_sys_openat2+0x2b4/0x410 [ 5340.414737] do_sys_open+0x73/0xa0 [ 5340.416800] __x64_sys_openat+0x24/0x30 [ 5340.418446] do_syscall_64+0xc1/0x440 [ 5340.421044] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5340.423102] RIP: 0033:0x7ff54af45332 [ 5340.424172] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5340.431305] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5340.434911] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 5340.438194] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 5340.445745] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 5340.447929] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 5340.450707] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 5348.057476] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 5348.062344] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5348.106916] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5348.113350] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5348.118680] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5348.122602] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5348.144618] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5348.157574] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 5348.168310] RDX: 0000000000000f10 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 5348.179973] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 5348.192541] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 5348.198031] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5348.203324] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 5348.211830] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5348.214950] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 5348.221866] Call Trace: [ 5348.224794] [ 5348.225724] ? show_regs.cold.9+0x22/0x2f [ 5348.230634] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5348.234636] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5348.236066] ? watchdog+0x40/0x40 [ 5348.238074] ? hrtimer_interrupt+0x128/0x2c0 [ 5348.240315] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5348.242853] ? apic_timer_interrupt+0xf/0x20 [ 5348.244656] [ 5348.245211] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5348.246742] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5348.248474] do_raw_spin_lock+0x177/0x190 [ 5348.253775] _raw_spin_lock+0x21/0x30 [ 5348.257839] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5348.266123] obd_mod_free+0x46/0x140 [ptlrpc] [ 5348.269033] mdc_commit_open+0x91/0xb0 [mdc] [ 5348.271327] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5348.273885] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5348.278054] after_reply+0xe2c/0x1780 [ptlrpc] [ 5348.289627] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5348.291503] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5348.297616] ? do_wait_intr+0xf0/0xf0 [ 5348.298385] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5348.300212] kthread+0x1d1/0x200 [ 5348.305145] ? set_kthread_struct+0x70/0x70 [ 5348.308255] ret_from_fork+0x1f/0x30 [ 5352.058587] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 5352.062227] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5352.104330] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5352.113493] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5352.121164] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5352.128292] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5352.145170] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5352.152862] RAX: 0000000000007ea8 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 5352.159483] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 5352.166268] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5352.173991] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 5352.181589] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 5352.187709] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 5352.193775] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5352.197600] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 5352.204917] Call Trace: [ 5352.206175] [ 5352.207187] ? show_regs.cold.9+0x22/0x2f [ 5352.213196] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5352.218884] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5352.220846] ? watchdog+0x40/0x40 [ 5352.222406] ? hrtimer_interrupt+0x128/0x2c0 [ 5352.227943] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5352.230876] ? apic_timer_interrupt+0xf/0x20 [ 5352.235526] [ 5352.236929] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5352.239589] do_raw_spin_lock+0x177/0x190 [ 5352.245003] _raw_spin_lock+0x21/0x30 [ 5352.246952] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5352.251845] ? do_raw_spin_unlock+0x75/0x190 [ 5352.259037] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5352.262909] ? do_wait_intr+0xf0/0xf0 [ 5352.265768] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5352.268525] mdc_close+0x53b/0xd70 [mdc] [ 5352.270719] lmv_close+0x18d/0x490 [lmv] [ 5352.272673] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 5352.275307] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5352.278646] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5352.282067] ll_md_close+0x274/0x800 [lustre] [ 5352.286447] ll_file_release+0x128/0xc50 [lustre] [ 5352.290930] __fput+0x10d/0x440 [ 5352.293829] ____fput+0x12/0x20 [ 5352.295925] task_work_run+0xc6/0x110 [ 5352.298243] exit_to_usermode_loop+0x1dd/0x1f0 [ 5352.300863] do_syscall_64+0x3d6/0x440 [ 5352.303171] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5352.305885] RIP: 0033:0x7ff54af45cd8 [ 5352.307722] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5352.321928] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5352.326141] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 5352.331183] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 5352.336726] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 5352.340784] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 5352.345477] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5352.927636] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3394s! [ 5352.932181] Showing busy workqueues and worker pools: [ 5352.934549] workqueue events: flags=0x0 [ 5352.936491] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5352.939952] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 5352.944489] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5352.947734] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 5352.953045] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5352.956292] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 5352.959449] workqueue events_long: flags=0x0 [ 5352.961436] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5352.964555] pending: lu_objects_flush [obdclass] [ 5352.967084] workqueue events_unbound: flags=0x2 [ 5352.969152] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5352.972109] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5352.976969] workqueue events_power_efficient: flags=0x80 [ 5352.979428] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5352.982526] pending: neigh_periodic_work [ 5352.984551] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5352.987764] pending: do_cache_clean, check_lifetime [ 5352.990695] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5352.993876] pending: neigh_periodic_work [ 5352.996039] workqueue writeback: flags=0x4a [ 5352.998024] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5353.000920] pending: wb_workfn, wb_workfn [ 5353.002954] workqueue xprtiod: flags=0xa [ 5353.004812] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5353.007785] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 5353.011792] workqueue ipv6_addrconf: flags=0x40008 [ 5353.014153] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5353.017418] pending: addrconf_verify_work [ 5353.021180] workqueue knbd0-recv: flags=0x1a [ 5353.024302] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5353.031535] in-flight: 657:recv_work [ 5353.033744] workqueue ptlrpc_pinger: flags=0x2 [ 5353.035987] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5353.038851] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 5353.042264] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3394s workers=4 idle: 20831 12642 28859 [ 5353.046744] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5343s workers=2 idle: 659 [ 5360.061641] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 5360.071916] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5360.100482] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5360.109843] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5360.116193] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5360.123933] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5360.125920] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5360.140203] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5360.149636] RAX: 0000000000005a8c RBX: ffff96848811cae0 RCX: 0000000000000001 [ 5360.154209] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 5360.161503] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 5360.163942] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 5360.171649] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 5360.175196] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 5360.182617] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5360.186084] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 5360.192059] Call Trace: [ 5360.193757] [ 5360.195122] ? show_regs.cold.9+0x22/0x2f [ 5360.197675] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5360.199473] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5360.201491] ? watchdog+0x40/0x40 [ 5360.205014] ? hrtimer_interrupt+0x128/0x2c0 [ 5360.208040] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5360.210313] ? apic_timer_interrupt+0xf/0x20 [ 5360.213449] [ 5360.214482] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5360.217057] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5360.220601] do_raw_spin_lock+0x177/0x190 [ 5360.226707] _raw_spin_lock+0x21/0x30 [ 5360.232020] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5360.236167] process_one_work+0x2c8/0x700 [ 5360.240822] worker_thread+0x296/0x6e0 [ 5360.242781] ? rescuer_thread+0x570/0x570 [ 5360.244826] kthread+0x1d1/0x200 [ 5360.246837] ? set_kthread_struct+0x70/0x70 [ 5360.250164] ret_from_fork+0x1f/0x30 [ 5368.060065] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 5368.064467] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5368.098863] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5368.108468] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5368.113054] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5368.116719] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5368.129658] RSP: 0000:ffffaf9fc15234f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5368.134703] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 5368.142452] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 5368.144818] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5368.154991] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 5368.159480] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 5368.164897] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 5368.172206] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5368.174187] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 5368.180818] Call Trace: [ 5368.182029] [ 5368.182861] ? show_regs.cold.9+0x22/0x2f [ 5368.189381] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5368.191813] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5368.197950] ? watchdog+0x40/0x40 [ 5368.199113] ? hrtimer_interrupt+0x128/0x2c0 [ 5368.202090] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5368.205169] ? apic_timer_interrupt+0xf/0x20 [ 5368.208940] [ 5368.210024] ? native_safe_halt+0x2e/0x50 [ 5368.211974] kvm_wait+0xcd/0xe0 [ 5368.213110] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5368.219988] do_raw_spin_lock+0x177/0x190 [ 5368.222727] _raw_spin_lock+0x21/0x30 [ 5368.225125] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5368.229143] ? do_raw_spin_unlock+0x75/0x190 [ 5368.234599] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5368.238671] ? do_wait_intr+0xf0/0xf0 [ 5368.244687] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5368.250430] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5368.253989] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5368.260695] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5368.268632] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5368.276934] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5368.279942] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5368.282428] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5368.285975] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5368.298763] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5368.301947] lmv_intent_open+0x338/0x1150 [lmv] [ 5368.306077] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5368.308078] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5368.313976] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5368.316538] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5368.322705] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5368.328483] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5368.332931] ? do_raw_spin_unlock+0x75/0x190 [ 5368.335641] ? _raw_spin_unlock+0x12/0x30 [ 5368.339008] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5368.341967] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5368.344211] ll_atomic_open+0x36a/0xd70 [lustre] [ 5368.346960] lookup_open+0x356/0xb40 [ 5368.351688] do_last+0x313/0xfc0 [ 5368.355982] ? nd_jump_root+0xe5/0x160 [ 5368.359736] ? path_init+0x437/0x520 [ 5368.361013] path_openat+0xf7/0x500 [ 5368.369262] ? do_raw_spin_unlock+0x75/0x190 [ 5368.372548] do_filp_open+0x99/0x140 [ 5368.375104] ? getname_flags+0x6e/0x330 [ 5368.377386] ? __check_object_size+0xff/0x256 [ 5368.380378] ? do_raw_spin_unlock+0x75/0x190 [ 5368.382759] ? _raw_spin_unlock+0x12/0x30 [ 5368.384949] do_sys_openat2+0x2b4/0x410 [ 5368.392042] do_sys_open+0x73/0xa0 [ 5368.393879] __x64_sys_openat+0x24/0x30 [ 5368.396277] do_syscall_64+0xc1/0x440 [ 5368.398401] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5368.400423] RIP: 0033:0x7ff54af45332 [ 5368.402590] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5368.423152] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5368.430323] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 5368.437642] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 5368.442091] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 5368.451156] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 5368.462495] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 5380.058080] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 5380.060848] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 5380.064528] Modules linked in: lustre(O) [ 5380.074269] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5380.074348] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5380.074356] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5380.074359] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5380.074373] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5380.074378] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5380.074382] RAX: 0000000000006a91 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 5380.074384] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 5380.074387] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5380.074389] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 5380.074391] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 5380.074394] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 5380.074397] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5380.074399] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 5380.074844] Call Trace: [ 5380.074852] [ 5380.074869] ? show_regs.cold.9+0x22/0x2f [ 5380.074881] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5380.074888] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5380.074893] ? watchdog+0x40/0x40 [ 5380.074898] ? hrtimer_interrupt+0x128/0x2c0 [ 5380.074902] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5380.074912] ? apic_timer_interrupt+0xf/0x20 [ 5380.074916] [ 5380.074920] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5380.074927] do_raw_spin_lock+0x177/0x190 [ 5380.074932] _raw_spin_lock+0x21/0x30 [ 5380.074937] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5380.076362] ? do_raw_spin_unlock+0x75/0x190 [ 5380.076371] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5380.076592] ? do_wait_intr+0xf0/0xf0 [ 5380.076601] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5380.076931] mdc_close+0x53b/0xd70 [mdc] [ 5380.076975] lmv_close+0x18d/0x490 [lmv] [ 5380.077184] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 5380.077364] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5380.077479] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5380.077559] ll_md_close+0x274/0x800 [lustre] [ 5380.077640] ll_file_release+0x128/0xc50 [lustre] [ 5380.077721] __fput+0x10d/0x440 [ 5380.077730] ____fput+0x12/0x20 [ 5380.077735] task_work_run+0xc6/0x110 [ 5380.077741] exit_to_usermode_loop+0x1dd/0x1f0 [ 5380.077748] do_syscall_64+0x3d6/0x440 [ 5380.077753] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5380.080609] RIP: 0033:0x7ff54af45cd8 [ 5380.082292] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5380.082368] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5380.082376] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5380.082379] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5380.082393] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5380.082397] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5380.082401] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 5380.082404] RDX: 0000000000003ae6 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 5380.082406] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 5380.082409] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 5380.082411] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5380.082414] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 5380.082417] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5380.082420] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 5380.082428] Call Trace: [ 5380.082433] [ 5380.082439] ? show_regs.cold.9+0x22/0x2f [ 5380.082448] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5380.082455] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5380.082460] ? watchdog+0x40/0x40 [ 5380.082465] ? hrtimer_interrupt+0x128/0x2c0 [ 5380.082469] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5380.082477] ? apic_timer_interrupt+0xf/0x20 [ 5380.082482] [ 5380.082485] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5380.082489] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5380.082493] do_raw_spin_lock+0x177/0x190 [ 5380.082497] _raw_spin_lock+0x21/0x30 [ 5380.082503] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5380.082882] obd_mod_free+0x46/0x140 [ptlrpc] [ 5380.083083] mdc_commit_open+0x91/0xb0 [mdc] [ 5380.083119] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5380.135606] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5380.135623] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5380.135629] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 5380.135632] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 5380.135634] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 5380.135636] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 5380.135638] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5380.566685] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5380.569352] after_reply+0xe2c/0x1780 [ptlrpc] [ 5380.571499] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5380.573770] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5380.576132] ? do_wait_intr+0xf0/0xf0 [ 5380.578381] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5380.580910] kthread+0x1d1/0x200 [ 5380.582839] ? set_kthread_struct+0x70/0x70 [ 5380.584943] ret_from_fork+0x1f/0x30 [ 5383.141473] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3424s! [ 5383.148986] Showing busy workqueues and worker pools: [ 5383.151695] workqueue events: flags=0x0 [ 5383.153789] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5383.157423] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 5383.162557] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5383.166473] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 5383.173191] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5383.176775] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 5383.180543] workqueue events_long: flags=0x0 [ 5383.182797] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5383.186765] pending: lu_objects_flush [obdclass] [ 5383.189924] workqueue events_unbound: flags=0x2 [ 5383.191891] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5383.198416] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5383.204931] workqueue events_power_efficient: flags=0x80 [ 5383.207343] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5383.210355] pending: neigh_periodic_work [ 5383.212026] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5383.214672] pending: do_cache_clean, check_lifetime [ 5383.216855] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5383.220015] pending: neigh_periodic_work [ 5383.221880] workqueue writeback: flags=0x4a [ 5383.223708] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5383.226420] pending: wb_workfn, wb_workfn [ 5383.228590] workqueue xprtiod: flags=0xa [ 5383.230195] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5383.233015] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 5383.236622] workqueue ipv6_addrconf: flags=0x40008 [ 5383.238740] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5383.241664] pending: addrconf_verify_work [ 5383.243794] workqueue knbd0-recv: flags=0x1a [ 5383.245769] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5383.248505] in-flight: 657:recv_work [ 5383.250123] workqueue ptlrpc_pinger: flags=0x2 [ 5383.251714] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5383.253439] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 5383.255864] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3424s workers=4 idle: 20831 12642 28859 [ 5383.259725] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5373s workers=2 idle: 659 [ 5388.060827] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 5388.066868] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5388.107543] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5388.112377] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5388.117095] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5388.126715] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5388.129907] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5388.146484] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 5388.148674] RAX: 0000000000004309 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 5388.154456] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 5388.160921] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 5388.167811] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 5388.175269] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 5388.179400] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 5388.182639] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5388.184838] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 5388.188514] Call Trace: [ 5388.189465] [ 5388.190323] ? show_regs.cold.9+0x22/0x2f [ 5388.191871] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5388.193945] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5388.195803] ? watchdog+0x40/0x40 [ 5388.197224] ? hrtimer_interrupt+0x128/0x2c0 [ 5388.198525] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5388.200444] ? apic_timer_interrupt+0xf/0x20 [ 5388.202303] [ 5388.204142] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5388.205970] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5388.217964] do_raw_spin_lock+0x177/0x190 [ 5388.220974] _raw_spin_lock+0x21/0x30 [ 5388.222678] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5388.224906] process_one_work+0x2c8/0x700 [ 5388.227621] worker_thread+0x296/0x6e0 [ 5388.229349] ? rescuer_thread+0x570/0x570 [ 5388.232469] kthread+0x1d1/0x200 [ 5388.233430] ? set_kthread_struct+0x70/0x70 [ 5388.235307] ret_from_fork+0x1f/0x30 [ 5396.063565] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 5396.079361] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5396.115804] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5396.126117] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5396.138842] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5396.149772] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5396.175245] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5396.182274] RAX: 0000000000007e97 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 5396.189644] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 5396.194543] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5396.196802] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 5396.203287] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 5396.212877] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 5396.222745] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5396.231787] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 5396.243739] Call Trace: [ 5396.246688] [ 5396.247879] ? show_regs.cold.9+0x22/0x2f [ 5396.250281] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5396.255219] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5396.259120] ? watchdog+0x40/0x40 [ 5396.263254] ? hrtimer_interrupt+0x128/0x2c0 [ 5396.266775] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5396.269722] ? apic_timer_interrupt+0xf/0x20 [ 5396.272136] [ 5396.273742] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5396.279471] do_raw_spin_lock+0x177/0x190 [ 5396.281293] _raw_spin_lock+0x21/0x30 [ 5396.284634] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5396.288359] ? do_raw_spin_unlock+0x75/0x190 [ 5396.289902] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5396.295852] ? do_wait_intr+0xf0/0xf0 [ 5396.298966] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5396.304795] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5396.311203] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5396.317853] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5396.323348] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5396.330402] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5396.339211] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5396.345228] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5396.351562] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5396.358285] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5396.361647] lmv_intent_open+0x338/0x1150 [lmv] [ 5396.365440] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5396.371242] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5396.374604] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5396.376850] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5396.378885] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5396.383594] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5396.390158] ? do_raw_spin_unlock+0x75/0x190 [ 5396.394194] ? _raw_spin_unlock+0x12/0x30 [ 5396.396152] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5396.398828] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5396.402246] ll_atomic_open+0x36a/0xd70 [lustre] [ 5396.406423] lookup_open+0x356/0xb40 [ 5396.408781] do_last+0x313/0xfc0 [ 5396.410864] ? nd_jump_root+0xe5/0x160 [ 5396.413630] ? path_init+0x437/0x520 [ 5396.416782] path_openat+0xf7/0x500 [ 5396.419751] ? do_raw_spin_unlock+0x75/0x190 [ 5396.423222] do_filp_open+0x99/0x140 [ 5396.426050] ? getname_flags+0x6e/0x330 [ 5396.427365] ? __check_object_size+0xff/0x256 [ 5396.429802] ? do_raw_spin_unlock+0x75/0x190 [ 5396.431274] ? _raw_spin_unlock+0x12/0x30 [ 5396.435202] do_sys_openat2+0x2b4/0x410 [ 5396.438145] do_sys_open+0x73/0xa0 [ 5396.446247] __x64_sys_openat+0x24/0x30 [ 5396.456395] do_syscall_64+0xc1/0x440 [ 5396.465377] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5396.473526] RIP: 0033:0x7ff54af45332 [ 5396.480279] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5396.508889] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5396.528195] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 5396.540420] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 5396.552413] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 5396.558736] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 5396.568257] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 5408.055076] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 5408.058033] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 5408.059569] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) [ 5408.063184] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5408.063284] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5408.063293] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5408.063297] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5408.063316] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5408.063321] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5408.063325] RAX: 0000000000000349 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 5408.063328] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 5408.063330] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5408.063332] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 5408.063335] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 5408.063338] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 5408.063341] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5408.063343] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 5408.063353] Call Trace: [ 5408.063361] [ 5408.063369] ? show_regs.cold.9+0x22/0x2f [ 5408.077456] mdd(O) mgs(O) [ 5408.097338] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5408.097365] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5408.097371] ? watchdog+0x40/0x40 [ 5408.097375] ? hrtimer_interrupt+0x128/0x2c0 [ 5408.097380] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5408.097389] ? apic_timer_interrupt+0xf/0x20 [ 5408.097395] [ 5408.097400] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5408.097412] do_raw_spin_lock+0x177/0x190 [ 5408.097420] _raw_spin_lock+0x21/0x30 [ 5408.097425] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5408.097823] ? do_raw_spin_unlock+0x75/0x190 [ 5408.097829] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5408.098086] ? do_wait_intr+0xf0/0xf0 [ 5408.098110] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5408.098327] mdc_close+0x53b/0xd70 [mdc] [ 5408.098378] lmv_close+0x18d/0x490 [lmv] [ 5408.098411] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 5408.098601] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5408.098735] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5408.098817] ll_md_close+0x274/0x800 [lustre] [ 5408.099052] ll_file_release+0x128/0xc50 [lustre] [ 5408.099134] __fput+0x10d/0x440 [ 5408.099146] ____fput+0x12/0x20 [ 5408.099150] task_work_run+0xc6/0x110 [ 5408.099158] exit_to_usermode_loop+0x1dd/0x1f0 [ 5408.099168] do_syscall_64+0x3d6/0x440 [ 5408.099173] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5408.099182] RIP: 0033:0x7ff54af45cd8 [ 5408.099192] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5408.099197] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5408.099203] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 5408.099206] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 5408.099208] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 5408.099210] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 5408.099213] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5408.340240] lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5408.376319] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5408.383035] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5408.388273] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5408.390321] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5408.409064] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5408.421558] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 5408.423939] RDX: 0000000000005201 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 5408.430164] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 5408.435033] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 5408.443388] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5408.449067] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 5408.459565] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5408.462894] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 5408.469478] Call Trace: [ 5408.474130] [ 5408.475624] ? show_regs.cold.9+0x22/0x2f [ 5408.478283] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5408.486764] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5408.491158] ? watchdog+0x40/0x40 [ 5408.493310] ? hrtimer_interrupt+0x128/0x2c0 [ 5408.497608] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5408.505934] ? apic_timer_interrupt+0xf/0x20 [ 5408.507357] [ 5408.511480] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5408.515134] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5408.520451] do_raw_spin_lock+0x177/0x190 [ 5408.523322] _raw_spin_lock+0x21/0x30 [ 5408.529081] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5408.532940] obd_mod_free+0x46/0x140 [ptlrpc] [ 5408.535659] mdc_commit_open+0x91/0xb0 [mdc] [ 5408.538602] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5408.541712] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5408.544338] after_reply+0xe2c/0x1780 [ptlrpc] [ 5408.546666] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5408.548336] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5408.550531] ? do_wait_intr+0xf0/0xf0 [ 5408.551727] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5408.555286] kthread+0x1d1/0x200 [ 5408.557168] ? set_kthread_struct+0x70/0x70 [ 5408.559425] ret_from_fork+0x1f/0x30 [ 5413.348937] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3454s! [ 5413.360132] Showing busy workqueues and worker pools: [ 5413.369028] workqueue events: flags=0x0 [ 5413.370290] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5413.383276] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 5413.401458] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5413.422956] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 5413.455665] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5413.457900] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 5413.468038] workqueue events_long: flags=0x0 [ 5413.469496] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5413.474364] pending: lu_objects_flush [obdclass] [ 5413.481370] workqueue events_unbound: flags=0x2 [ 5413.489935] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5413.497845] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5413.504990] workqueue events_power_efficient: flags=0x80 [ 5413.508082] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5413.509866] pending: neigh_periodic_work [ 5413.511129] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5413.514935] pending: do_cache_clean, check_lifetime [ 5413.523703] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5413.530035] pending: neigh_periodic_work [ 5413.532208] workqueue writeback: flags=0x4a [ 5413.534981] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5413.542898] pending: wb_workfn, wb_workfn [ 5413.545379] workqueue xprtiod: flags=0xa [ 5413.548715] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5413.555564] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 5413.562716] workqueue ipv6_addrconf: flags=0x40008 [ 5413.571485] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5413.577048] pending: addrconf_verify_work [ 5413.584719] workqueue knbd0-recv: flags=0x1a [ 5413.587506] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5413.593141] in-flight: 657:recv_work [ 5413.596188] workqueue ptlrpc_pinger: flags=0x2 [ 5413.597882] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5413.608209] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 5413.615022] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3454s workers=4 idle: 20831 12642 28859 [ 5413.622084] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5403s workers=2 idle: 659 [ 5416.063650] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 5416.071853] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5416.125983] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5416.134041] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5416.138987] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5416.143032] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5416.146502] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5416.156942] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5416.160942] RAX: 0000000000003e82 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 5416.164584] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 5416.167860] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 5416.171185] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 5416.174482] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 5416.186601] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 5416.191477] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5416.195204] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 5416.199650] Call Trace: [ 5416.200973] [ 5416.202623] ? show_regs.cold.9+0x22/0x2f [ 5416.204494] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5416.209644] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5416.214809] ? watchdog+0x40/0x40 [ 5416.215787] ? hrtimer_interrupt+0x128/0x2c0 [ 5416.221426] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5416.224204] ? apic_timer_interrupt+0xf/0x20 [ 5416.226391] [ 5416.227886] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5416.234625] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5416.238681] do_raw_spin_lock+0x177/0x190 [ 5416.242132] _raw_spin_lock+0x21/0x30 [ 5416.243389] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5416.247554] process_one_work+0x2c8/0x700 [ 5416.250463] worker_thread+0x296/0x6e0 [ 5416.254014] ? rescuer_thread+0x570/0x570 [ 5416.258408] kthread+0x1d1/0x200 [ 5416.260493] ? set_kthread_struct+0x70/0x70 [ 5416.267280] ret_from_fork+0x1f/0x30 [ 5424.061070] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 5424.077371] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5424.122519] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5424.128764] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5424.133806] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5424.137058] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5424.147510] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5424.153445] RAX: 0000000000007d39 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 5424.160449] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 5424.164325] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5424.168115] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 5424.172114] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 5424.176183] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 5424.180412] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5424.183212] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 5424.186847] Call Trace: [ 5424.187941] [ 5424.189026] ? show_regs.cold.9+0x22/0x2f [ 5424.191079] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5424.193764] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5424.195472] ? watchdog+0x40/0x40 [ 5424.196988] ? hrtimer_interrupt+0x128/0x2c0 [ 5424.199253] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5424.202199] ? apic_timer_interrupt+0xf/0x20 [ 5424.204384] [ 5424.205537] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5424.208249] do_raw_spin_lock+0x177/0x190 [ 5424.210224] _raw_spin_lock+0x21/0x30 [ 5424.212181] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5424.214991] ? do_raw_spin_unlock+0x75/0x190 [ 5424.217218] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5424.220022] ? do_wait_intr+0xf0/0xf0 [ 5424.221996] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5424.224889] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5424.227636] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5424.231055] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5424.234003] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5424.236955] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5424.239355] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5424.241601] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5424.244481] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5424.247712] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5424.250966] lmv_intent_open+0x338/0x1150 [lmv] [ 5424.253290] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5424.256205] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5424.259028] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5424.261395] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5424.263901] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5424.266711] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5424.269256] ? do_raw_spin_unlock+0x75/0x190 [ 5424.271458] ? _raw_spin_unlock+0x12/0x30 [ 5424.273541] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5424.276362] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5424.278998] ll_atomic_open+0x36a/0xd70 [lustre] [ 5424.281479] lookup_open+0x356/0xb40 [ 5424.283443] do_last+0x313/0xfc0 [ 5424.285163] ? nd_jump_root+0xe5/0x160 [ 5424.287069] ? path_init+0x437/0x520 [ 5424.288714] path_openat+0xf7/0x500 [ 5424.290546] ? do_raw_spin_unlock+0x75/0x190 [ 5424.292980] do_filp_open+0x99/0x140 [ 5424.294976] ? getname_flags+0x6e/0x330 [ 5424.297023] ? __check_object_size+0xff/0x256 [ 5424.299177] ? do_raw_spin_unlock+0x75/0x190 [ 5424.301949] ? _raw_spin_unlock+0x12/0x30 [ 5424.303961] do_sys_openat2+0x2b4/0x410 [ 5424.305863] do_sys_open+0x73/0xa0 [ 5424.307570] __x64_sys_openat+0x24/0x30 [ 5424.309495] do_syscall_64+0xc1/0x440 [ 5424.311377] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5424.314022] RIP: 0033:0x7ff54af45332 [ 5424.315781] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5424.325834] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5424.329403] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 5424.332413] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 5424.335219] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 5424.337827] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 5424.340486] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 5433.943881] rcu: INFO: rcu_sched self-detected stall on CPU [ 5433.950206] rcu: 0-...!: (1807778 ticks this GP) idle=98a/1/0x4000000000000002 softirq=125154/125154 fqs=959 [ 5433.959291] (t=3480103 jiffies g=253585 q=3925) [ 5433.963438] rcu: rcu_sched kthread starved for 3475063 jiffies! g253585 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=3 [ 5433.979193] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 5433.984919] rcu: RCU grace-period kthread stack dump: [ 5433.999022] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 5434.010283] Call Trace: [ 5434.011430] __schedule+0x351/0xcb0 [ 5434.015268] schedule+0xc0/0x180 [ 5434.018161] schedule_timeout+0xb4/0x190 [ 5434.021396] ? do_raw_spin_unlock+0x75/0x190 [ 5434.028027] ? __next_timer_interrupt+0x160/0x160 [ 5434.033242] ? prepare_to_swait_event+0xf3/0x140 [ 5434.038471] rcu_gp_kthread+0x8dd/0x15b0 [ 5434.045729] ? force_qs_rnp+0x2c0/0x2c0 [ 5434.047663] kthread+0x1d1/0x200 [ 5434.052389] ? set_kthread_struct+0x70/0x70 [ 5434.058360] ret_from_fork+0x1f/0x30 [ 5434.060444] rcu: Stack dump where RCU GP kthread last ran: [ 5434.067393] Sending NMI from CPU 0 to CPUs 3: [ 5434.076140] NMI backtrace for cpu 3 [ 5434.076147] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5434.076149] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5434.076151] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5434.076153] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5434.076155] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000212 [ 5434.076157] RAX: 00000000000019dc RBX: ffff96848811cae0 RCX: 0000000000000001 [ 5434.076159] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 5434.076161] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5434.076162] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 5434.076163] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 5434.076165] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 5434.076166] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5434.076168] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 5434.076169] Call Trace: [ 5434.076169] [ 5434.076170] ? show_regs.cold.9+0x22/0x2f [ 5434.076171] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 5434.076173] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 5434.076174] ? nmi_handle+0xbd/0x210 [ 5434.076175] ? default_do_nmi+0x4b/0x1b0 [ 5434.076176] ? do_nmi+0x280/0x380 [ 5434.076177] ? end_repeat_nmi+0x16/0x1f [ 5434.076178] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5434.076179] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5434.076180] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5434.076181] [ 5434.076182] do_raw_spin_lock+0x177/0x190 [ 5434.076184] _raw_spin_lock+0x21/0x30 [ 5434.076185] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5434.076186] ? do_raw_spin_unlock+0x75/0x190 [ 5434.076187] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5434.076189] ? do_wait_intr+0xf0/0xf0 [ 5434.076190] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5434.076191] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5434.076192] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5434.076193] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5434.076194] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5434.076196] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5434.076197] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5434.076198] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5434.076199] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5434.076201] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5434.076201] lmv_intent_open+0x338/0x1150 [lmv] [ 5434.076203] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5434.076204] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5434.076205] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5434.076206] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5434.076207] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5434.076208] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5434.076209] ? do_raw_spin_unlock+0x75/0x190 [ 5434.076210] ? _raw_spin_unlock+0x12/0x30 [ 5434.076211] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5434.076212] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5434.076213] ll_atomic_open+0x36a/0xd70 [lustre] [ 5434.076214] lookup_open+0x356/0xb40 [ 5434.076215] do_last+0x313/0xfc0 [ 5434.076216] ? nd_jump_root+0xe5/0x160 [ 5434.076217] ? path_init+0x437/0x520 [ 5434.076218] path_openat+0xf7/0x500 [ 5434.076219] ? do_raw_spin_unlock+0x75/0x190 [ 5434.076220] do_filp_open+0x99/0x140 [ 5434.076221] ? getname_flags+0x6e/0x330 [ 5434.076222] ? __check_object_size+0xff/0x256 [ 5434.076224] ? do_raw_spin_unlock+0x75/0x190 [ 5434.076225] ? _raw_spin_unlock+0x12/0x30 [ 5434.076226] do_sys_openat2+0x2b4/0x410 [ 5434.076227] do_sys_open+0x73/0xa0 [ 5434.076228] __x64_sys_openat+0x24/0x30 [ 5434.076229] do_syscall_64+0xc1/0x440 [ 5434.076230] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5434.076231] RIP: 0033:0x7ff54af45332 [ 5434.076233] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5434.076235] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5434.076238] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 5434.076239] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 5434.076241] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 5434.076242] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 5434.076243] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 5434.080953] NMI backtrace for cpu 0 [ 5434.458634] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5434.473347] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5434.482788] Call Trace: [ 5434.484392] [ 5434.486423] ? dump_stack+0xbb/0x10e [ 5434.488694] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 5434.493393] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 5434.495156] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 5434.497815] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 5434.502490] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 5434.507153] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 5434.510239] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 5434.518834] ? update_process_times+0x6d/0xc0 [ 5434.521401] ? tick_sched_handle.isra.18+0x27/0x90 [ 5434.524188] ? tick_sched_timer+0x53/0xb0 [ 5434.526283] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5434.532898] ? tick_sched_handle.isra.18+0x90/0x90 [ 5434.540772] ? hrtimer_interrupt+0x128/0x2c0 [ 5434.545287] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5434.547535] ? apic_timer_interrupt+0xf/0x20 [ 5434.550450] [ 5434.552905] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5434.554420] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5434.555659] ? do_raw_spin_lock+0x177/0x190 [ 5434.562525] ? _raw_spin_lock+0x21/0x30 [ 5434.564046] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5434.569135] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 5434.573634] ? mdc_commit_open+0x91/0xb0 [mdc] [ 5434.576528] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5434.581052] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5434.589033] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 5434.597009] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5434.603825] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5434.607171] ? do_wait_intr+0xf0/0xf0 [ 5434.609436] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5434.614839] ? kthread+0x1d1/0x200 [ 5434.616846] ? set_kthread_struct+0x70/0x70 [ 5434.621027] ? ret_from_fork+0x1f/0x30 [ 5436.058033] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 5436.061003] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5436.084522] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5436.100491] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5436.116309] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5436.119811] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5436.130101] RSP: 0018:ffffaf9fc1543ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5436.132854] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 5436.136155] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 5436.144032] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5436.149943] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 5436.157243] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 5436.162988] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 5436.170706] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5436.185366] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 5436.193819] Call Trace: [ 5436.194520] [ 5436.198893] ? show_regs.cold.9+0x22/0x2f [ 5436.200581] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5436.204821] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5436.207138] ? watchdog+0x40/0x40 [ 5436.212593] ? hrtimer_interrupt+0x128/0x2c0 [ 5436.214385] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5436.218171] ? apic_timer_interrupt+0xf/0x20 [ 5436.219817] [ 5436.220427] ? native_safe_halt+0x2e/0x50 [ 5436.227536] kvm_wait+0xcd/0xe0 [ 5436.230168] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5436.234691] do_raw_spin_lock+0x177/0x190 [ 5436.236253] _raw_spin_lock+0x21/0x30 [ 5436.241934] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5436.243485] ? do_raw_spin_unlock+0x75/0x190 [ 5436.244616] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5436.249659] ? do_wait_intr+0xf0/0xf0 [ 5436.257950] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5436.260899] mdc_close+0x53b/0xd70 [mdc] [ 5436.269048] lmv_close+0x18d/0x490 [lmv] [ 5436.275707] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 5436.280435] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5436.289386] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5436.291140] ll_md_close+0x274/0x800 [lustre] [ 5436.296541] ll_file_release+0x128/0xc50 [lustre] [ 5436.297820] __fput+0x10d/0x440 [ 5436.298554] ____fput+0x12/0x20 [ 5436.306436] task_work_run+0xc6/0x110 [ 5436.307778] exit_to_usermode_loop+0x1dd/0x1f0 [ 5436.322153] do_syscall_64+0x3d6/0x440 [ 5436.323559] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5436.331759] RIP: 0033:0x7ff54af45cd8 [ 5436.335885] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5436.352430] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5436.362841] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 5436.372091] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 5436.374565] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 5436.382018] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 5436.386737] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5444.059040] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 5444.064254] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5444.067503] BUG: workqueue lockup - pool [ 5444.093231] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5444.093246] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5444.093251] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5444.093687] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5444.093698] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5444.093703] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5444.093708] RAX: 0000000000007285 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 5444.093710] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 5444.093712] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 5444.093714] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 5444.093716] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 5444.093720] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 5444.093723] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5444.093725] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 5444.093734] Call Trace: [ 5444.093740] [ 5444.093745] ? show_regs.cold.9+0x22/0x2f [ 5444.093754] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5444.093761] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5444.093765] ? watchdog+0x40/0x40 [ 5444.093769] ? hrtimer_interrupt+0x128/0x2c0 [ 5444.093774] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5444.093783] ? apic_timer_interrupt+0xf/0x20 [ 5444.093788] [ 5444.093791] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5444.093796] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5444.093801] do_raw_spin_lock+0x177/0x190 [ 5444.093807] _raw_spin_lock+0x21/0x30 [ 5444.093812] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5444.098462] cpus=0-3 flags=0x4 nice=0 stuck for 3485s! [ 5444.098483] Showing busy workqueues and worker pools: [ 5444.098485] workqueue events: flags=0x0 [ 5444.098488] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5444.098496] pending: ldlm_pools_recalc_task [ptlrpc] [ 5444.104996] process_one_work+0x2c8/0x700 [ 5444.105018] worker_thread+0x296/0x6e0 [ 5444.105023] ? rescuer_thread+0x570/0x570 [ 5444.105028] kthread+0x1d1/0x200 [ 5444.105033] ? set_kthread_struct+0x70/0x70 [ 5444.105037] ret_from_fork+0x1f/0x30 [ 5444.250635] , cpu_latency_work [ptlrpc] [ 5444.253982] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5444.258058] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 5444.264564] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5444.268208] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 5444.272212] workqueue events_long: flags=0x0 [ 5444.276991] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5444.283633] pending: lu_objects_flush [obdclass] [ 5444.286606] workqueue events_unbound: flags=0x2 [ 5444.290882] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5444.294766] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5444.301336] workqueue events_power_efficient: flags=0x80 [ 5444.305503] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5444.308325] pending: neigh_periodic_work [ 5444.313116] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5444.315969] pending: do_cache_clean, check_lifetime [ 5444.326996] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5444.334046] pending: neigh_periodic_work [ 5444.345645] workqueue writeback: flags=0x4a [ 5444.348955] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5444.357932] pending: wb_workfn, wb_workfn [ 5444.364199] workqueue xprtiod: flags=0xa [ 5444.371980] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5444.379818] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 5444.386475] workqueue ipv6_addrconf: flags=0x40008 [ 5444.389462] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5444.392953] pending: addrconf_verify_work [ 5444.397780] workqueue knbd0-recv: flags=0x1a [ 5444.399920] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5444.404779] in-flight: 657:recv_work [ 5444.407367] workqueue ptlrpc_pinger: flags=0x2 [ 5444.410163] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5444.413115] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 5444.416017] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3485s workers=4 idle: 20831 12642 28859 [ 5444.421012] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5434s workers=2 idle: 659 [ 5460.067379] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [ 5460.074627] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5460.126171] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5460.141131] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5460.146966] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5460.151063] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5460.177376] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5460.188824] RAX: 0000000000007f85 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 5460.204115] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 5460.220757] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5460.234981] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 5460.250254] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 5460.263103] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 5460.284115] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5460.298836] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 5460.311235] Call Trace: [ 5460.312200] [ 5460.319162] ? show_regs.cold.9+0x22/0x2f [ 5460.321255] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5460.330876] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5460.334559] ? watchdog+0x40/0x40 [ 5460.338993] ? hrtimer_interrupt+0x128/0x2c0 [ 5460.344571] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5460.352037] ? apic_timer_interrupt+0xf/0x20 [ 5460.357720] [ 5460.360057] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5460.365054] do_raw_spin_lock+0x177/0x190 [ 5460.370329] _raw_spin_lock+0x21/0x30 [ 5460.371640] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5460.375172] ? do_raw_spin_unlock+0x75/0x190 [ 5460.377551] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5460.382002] ? do_wait_intr+0xf0/0xf0 [ 5460.384093] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5460.387107] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5460.392509] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5460.395903] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5460.398821] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5460.401974] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5460.404432] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5460.410253] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5460.416201] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5460.420222] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5460.422820] lmv_intent_open+0x338/0x1150 [lmv] [ 5460.425058] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5460.429206] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5460.432945] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5460.436494] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5460.441815] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5460.445808] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5460.449662] ? do_raw_spin_unlock+0x75/0x190 [ 5460.456300] ? _raw_spin_unlock+0x12/0x30 [ 5460.459831] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5460.463678] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5460.465414] ll_atomic_open+0x36a/0xd70 [lustre] [ 5460.469745] lookup_open+0x356/0xb40 [ 5460.471625] do_last+0x313/0xfc0 [ 5460.473427] ? nd_jump_root+0xe5/0x160 [ 5460.476808] ? path_init+0x437/0x520 [ 5460.478743] path_openat+0xf7/0x500 [ 5460.480841] ? do_raw_spin_unlock+0x75/0x190 [ 5460.488228] do_filp_open+0x99/0x140 [ 5460.490834] ? getname_flags+0x6e/0x330 [ 5460.498528] ? __check_object_size+0xff/0x256 [ 5460.500004] ? do_raw_spin_unlock+0x75/0x190 [ 5460.501267] ? _raw_spin_unlock+0x12/0x30 [ 5460.504807] do_sys_openat2+0x2b4/0x410 [ 5460.506899] do_sys_open+0x73/0xa0 [ 5460.508780] __x64_sys_openat+0x24/0x30 [ 5460.510722] do_syscall_64+0xc1/0x440 [ 5460.512557] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5460.515059] RIP: 0033:0x7ff54af45332 [ 5460.517015] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5460.531188] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5460.534718] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 5460.537852] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 5460.540530] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 5460.543534] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 5460.547394] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 5464.058082] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 5464.064622] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5464.110452] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5464.121574] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5464.128222] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5464.133422] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5464.146804] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5464.151207] RAX: 0000000000001ec0 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 5464.165627] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 5464.174272] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5464.181366] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 5464.191596] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 5464.196187] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 5464.204201] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5464.210877] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 5464.221591] Call Trace: [ 5464.226592] [ 5464.229133] ? show_regs.cold.9+0x22/0x2f [ 5464.232624] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5464.235121] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5464.237448] ? watchdog+0x40/0x40 [ 5464.239109] ? hrtimer_interrupt+0x128/0x2c0 [ 5464.243020] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5464.247216] ? apic_timer_interrupt+0xf/0x20 [ 5464.251250] [ 5464.252219] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5464.256488] do_raw_spin_lock+0x177/0x190 [ 5464.259747] _raw_spin_lock+0x21/0x30 [ 5464.262246] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5464.270444] ? do_raw_spin_unlock+0x75/0x190 [ 5464.271961] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5464.281823] ? do_wait_intr+0xf0/0xf0 [ 5464.284885] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5464.289116] mdc_close+0x53b/0xd70 [mdc] [ 5464.297213] lmv_close+0x18d/0x490 [lmv] [ 5464.302691] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 5464.305997] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5464.308144] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5464.310462] ll_md_close+0x274/0x800 [lustre] [ 5464.318320] ll_file_release+0x128/0xc50 [lustre] [ 5464.322133] __fput+0x10d/0x440 [ 5464.329579] ____fput+0x12/0x20 [ 5464.336993] task_work_run+0xc6/0x110 [ 5464.339869] exit_to_usermode_loop+0x1dd/0x1f0 [ 5464.343521] do_syscall_64+0x3d6/0x440 [ 5464.345996] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5464.349458] RIP: 0033:0x7ff54af45cd8 [ 5464.352014] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5464.372975] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5464.378982] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 5464.384944] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 5464.388483] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 5464.391225] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 5464.393798] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5472.059311] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [ 5472.064052] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 5472.072107] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5472.073385] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5472.073393] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5472.073395] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5472.073410] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5472.073415] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5472.073419] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 5472.073422] RDX: 0000000000000c33 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 5472.073424] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 5472.073426] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 5472.073429] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5472.073432] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 5472.073435] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5472.073437] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 5472.073446] Call Trace: [ 5472.073452] [ 5472.073458] ? show_regs.cold.9+0x22/0x2f [ 5472.073467] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5472.073474] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5472.073479] ? watchdog+0x40/0x40 [ 5472.073483] ? hrtimer_interrupt+0x128/0x2c0 [ 5472.073487] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5472.073495] ? apic_timer_interrupt+0xf/0x20 [ 5472.073501] [ 5472.073504] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5472.073510] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5472.073514] do_raw_spin_lock+0x177/0x190 [ 5472.073519] _raw_spin_lock+0x21/0x30 [ 5472.073525] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5472.073782] obd_mod_free+0x46/0x140 [ptlrpc] [ 5472.073987] mdc_commit_open+0x91/0xb0 [mdc] [ 5472.074483] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5472.074691] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5472.074890] after_reply+0xe2c/0x1780 [ptlrpc] [ 5472.084993] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5472.085457] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5472.085464] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5472.085470] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5472.086144] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5472.086157] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5472.086161] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5472.086165] RAX: 0000000000007d8e RBX: ffff96848811cae0 RCX: 0000000000000001 [ 5472.086168] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 5472.087722] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 5472.087727] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 5472.087730] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 5472.087733] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 5472.087736] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5472.087738] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 5472.087746] Call Trace: [ 5472.087751] [ 5472.087757] ? show_regs.cold.9+0x22/0x2f [ 5472.087768] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5472.087776] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5472.087780] ? watchdog+0x40/0x40 [ 5472.087785] ? hrtimer_interrupt+0x128/0x2c0 [ 5472.087788] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5472.087796] ? apic_timer_interrupt+0xf/0x20 [ 5472.087801] [ 5472.087804] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5472.087810] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5472.087816] do_raw_spin_lock+0x177/0x190 [ 5472.087821] _raw_spin_lock+0x21/0x30 [ 5472.087826] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5472.133225] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5472.145950] process_one_work+0x2c8/0x700 [ 5472.145971] worker_thread+0x296/0x6e0 [ 5472.145976] ? rescuer_thread+0x570/0x570 [ 5472.145980] kthread+0x1d1/0x200 [ 5472.145984] ? set_kthread_struct+0x70/0x70 [ 5472.145989] ret_from_fork+0x1f/0x30 [ 5472.576719] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5472.579043] ? do_wait_intr+0xf0/0xf0 [ 5472.583759] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5472.591126] kthread+0x1d1/0x200 [ 5472.591927] ? set_kthread_struct+0x70/0x70 [ 5472.594714] ret_from_fork+0x1f/0x30 [ 5474.783071] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3515s! [ 5474.785872] Showing busy workqueues and worker pools: [ 5474.788201] workqueue events: flags=0x0 [ 5474.790025] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5474.793443] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 5474.799462] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5474.803737] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 5474.810406] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5474.812801] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 5474.816207] workqueue events_long: flags=0x0 [ 5474.817890] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5474.824052] pending: lu_objects_flush [obdclass] [ 5474.846212] workqueue events_unbound: flags=0x2 [ 5474.853329] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5474.859652] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5474.865207] workqueue events_power_efficient: flags=0x80 [ 5474.868218] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5474.871416] pending: neigh_periodic_work [ 5474.872532] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5474.874481] pending: do_cache_clean, check_lifetime [ 5474.876232] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5474.880611] pending: neigh_periodic_work [ 5474.882673] workqueue writeback: flags=0x4a [ 5474.884620] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5474.887315] pending: wb_workfn, wb_workfn [ 5474.889363] workqueue xprtiod: flags=0xa [ 5474.890903] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5474.893622] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 5474.896943] workqueue ipv6_addrconf: flags=0x40008 [ 5474.898864] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5474.901671] pending: addrconf_verify_work [ 5474.903711] workqueue knbd0-recv: flags=0x1a [ 5474.905705] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5474.909109] in-flight: 657:recv_work [ 5474.911752] workqueue ptlrpc_pinger: flags=0x2 [ 5474.913893] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5474.916556] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 5474.919443] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3516s workers=4 idle: 20831 12642 28859 [ 5474.922922] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5465s workers=2 idle: 659 [ 5488.060007] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [ 5488.069933] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5488.129807] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5488.147576] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5488.153541] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5488.159680] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5488.173558] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 5488.179771] RAX: 00000000000044df RBX: ffff96848811cae0 RCX: 0000000000000001 [ 5488.186899] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 5488.196653] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5488.202697] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 5488.209951] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 5488.217842] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 5488.227013] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5488.231674] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 5488.235923] Call Trace: [ 5488.237929] [ 5488.239078] ? show_regs.cold.9+0x22/0x2f [ 5488.245835] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5488.254504] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5488.256837] ? watchdog+0x40/0x40 [ 5488.260000] ? hrtimer_interrupt+0x128/0x2c0 [ 5488.268673] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5488.271122] ? apic_timer_interrupt+0xf/0x20 [ 5488.273892] [ 5488.274963] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5488.278766] do_raw_spin_lock+0x177/0x190 [ 5488.280861] _raw_spin_lock+0x21/0x30 [ 5488.285497] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5488.293572] ? do_raw_spin_unlock+0x75/0x190 [ 5488.295895] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5488.299014] ? do_wait_intr+0xf0/0xf0 [ 5488.300964] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5488.308979] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5488.313348] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5488.321987] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5488.325980] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5488.333630] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5488.336346] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5488.339271] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5488.342837] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5488.346842] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5488.350451] lmv_intent_open+0x338/0x1150 [lmv] [ 5488.355652] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5488.359958] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5488.362819] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5488.370359] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5488.376730] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5488.380006] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5488.390697] ? do_raw_spin_unlock+0x75/0x190 [ 5488.394004] ? _raw_spin_unlock+0x12/0x30 [ 5488.396078] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5488.401007] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5488.403750] ll_atomic_open+0x36a/0xd70 [lustre] [ 5488.412947] lookup_open+0x356/0xb40 [ 5488.414270] do_last+0x313/0xfc0 [ 5488.420158] ? nd_jump_root+0xe5/0x160 [ 5488.423959] ? path_init+0x437/0x520 [ 5488.427877] path_openat+0xf7/0x500 [ 5488.430992] ? do_raw_spin_unlock+0x75/0x190 [ 5488.438815] do_filp_open+0x99/0x140 [ 5488.442401] ? getname_flags+0x6e/0x330 [ 5488.448311] ? __check_object_size+0xff/0x256 [ 5488.455601] ? do_raw_spin_unlock+0x75/0x190 [ 5488.457972] ? _raw_spin_unlock+0x12/0x30 [ 5488.462600] do_sys_openat2+0x2b4/0x410 [ 5488.465936] do_sys_open+0x73/0xa0 [ 5488.467986] __x64_sys_openat+0x24/0x30 [ 5488.470842] do_syscall_64+0xc1/0x440 [ 5488.474750] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5488.481001] RIP: 0033:0x7ff54af45332 [ 5488.484725] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5488.504559] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5488.507416] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 5488.509983] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 5488.512755] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 5488.515453] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 5488.518318] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 5492.083120] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 5492.087788] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5492.160394] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5492.179309] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5492.191495] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5492.201618] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5492.242630] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5492.254329] RAX: 0000000000007f85 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 5492.270626] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 5492.285759] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5492.301169] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 5492.306934] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 5492.318182] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 5492.332901] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5492.338048] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 5492.342645] Call Trace: [ 5492.344237] [ 5492.350764] ? show_regs.cold.9+0x22/0x2f [ 5492.354100] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5492.359859] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5492.361035] ? watchdog+0x40/0x40 [ 5492.366236] ? hrtimer_interrupt+0x128/0x2c0 [ 5492.367918] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5492.370139] ? apic_timer_interrupt+0xf/0x20 [ 5492.371913] [ 5492.372887] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5492.379473] do_raw_spin_lock+0x177/0x190 [ 5492.385190] _raw_spin_lock+0x21/0x30 [ 5492.394547] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5492.402691] ? do_raw_spin_unlock+0x75/0x190 [ 5492.408047] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5492.414333] ? do_wait_intr+0xf0/0xf0 [ 5492.418957] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5492.420771] mdc_close+0x53b/0xd70 [mdc] [ 5492.421964] lmv_close+0x18d/0x490 [lmv] [ 5492.433212] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 5492.440412] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5492.444099] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5492.449559] ll_md_close+0x274/0x800 [lustre] [ 5492.455299] ll_file_release+0x128/0xc50 [lustre] [ 5492.464518] __fput+0x10d/0x440 [ 5492.472866] ____fput+0x12/0x20 [ 5492.475235] task_work_run+0xc6/0x110 [ 5492.483050] exit_to_usermode_loop+0x1dd/0x1f0 [ 5492.491764] do_syscall_64+0x3d6/0x440 [ 5492.499041] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5492.503928] RIP: 0033:0x7ff54af45cd8 [ 5492.506886] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5492.532270] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5492.537111] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 5492.547275] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 5492.558455] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 5492.570187] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 5492.578851] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5500.055024] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [ 5500.060919] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5500.069283] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 5500.087813] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5500.087823] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5500.087826] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5500.087837] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5500.087840] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5500.087845] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 5500.087846] RDX: 0000000000004977 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 5500.087848] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 5500.087850] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 5500.087851] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5500.087854] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 5500.087855] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5500.087857] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 5500.087863] Call Trace: [ 5500.087867] [ 5500.087872] ? show_regs.cold.9+0x22/0x2f [ 5500.087878] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5500.087883] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5500.087885] ? watchdog+0x40/0x40 [ 5500.087888] ? hrtimer_interrupt+0x128/0x2c0 [ 5500.087890] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5500.087897] ? apic_timer_interrupt+0xf/0x20 [ 5500.087900] [ 5500.087902] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5500.087905] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5500.087907] do_raw_spin_lock+0x177/0x190 [ 5500.087911] _raw_spin_lock+0x21/0x30 [ 5500.087916] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5500.102796] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5500.102877] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5500.102884] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5500.102891] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5500.110397] obd_mod_free+0x46/0x140 [ptlrpc] [ 5500.110686] mdc_commit_open+0x91/0xb0 [mdc] [ 5500.110721] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5500.110900] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5500.111103] after_reply+0xe2c/0x1780 [ptlrpc] [ 5500.111297] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5500.111482] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5500.111683] ? do_wait_intr+0xf0/0xf0 [ 5500.111693] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5500.112379] kthread+0x1d1/0x200 [ 5500.112386] ? set_kthread_struct+0x70/0x70 [ 5500.112390] ret_from_fork+0x1f/0x30 [ 5500.257970] [ 5500.258467] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5500.270528] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5500.296574] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5500.307395] RAX: 0000000000007786 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 5500.310745] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 5500.314663] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 5500.318693] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 5500.324864] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 5500.333378] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 5500.338664] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5500.342950] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 5500.351406] Call Trace: [ 5500.354434] [ 5500.358421] ? show_regs.cold.9+0x22/0x2f [ 5500.365614] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5500.373699] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5500.376381] ? watchdog+0x40/0x40 [ 5500.380655] ? hrtimer_interrupt+0x128/0x2c0 [ 5500.383782] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5500.386836] ? apic_timer_interrupt+0xf/0x20 [ 5500.389762] [ 5500.391369] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5500.394683] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5500.397285] do_raw_spin_lock+0x177/0x190 [ 5500.399625] _raw_spin_lock+0x21/0x30 [ 5500.401375] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5500.404792] process_one_work+0x2c8/0x700 [ 5500.407653] worker_thread+0x296/0x6e0 [ 5500.410711] ? rescuer_thread+0x570/0x570 [ 5500.415419] kthread+0x1d1/0x200 [ 5500.418913] ? set_kthread_struct+0x70/0x70 [ 5500.420374] ret_from_fork+0x1f/0x30 [ 5504.994828] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3546s! [ 5504.999898] Showing busy workqueues and worker pools: [ 5505.002303] workqueue events: flags=0x0 [ 5505.005619] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5505.015704] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 5505.024206] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5505.031927] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 5505.046697] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5505.057500] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 5505.062610] workqueue events_long: flags=0x0 [ 5505.067266] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5505.077975] pending: lu_objects_flush [obdclass] [ 5505.082898] workqueue events_unbound: flags=0x2 [ 5505.088229] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5505.093218] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5505.102312] workqueue events_power_efficient: flags=0x80 [ 5505.110707] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5505.118692] pending: neigh_periodic_work [ 5505.126614] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5505.134564] pending: do_cache_clean, check_lifetime [ 5505.139412] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5505.150761] pending: neigh_periodic_work [ 5505.152690] workqueue writeback: flags=0x4a [ 5505.157524] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5505.165505] pending: wb_workfn, wb_workfn [ 5505.171547] workqueue xprtiod: flags=0xa [ 5505.178118] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5505.187520] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 5505.204958] workqueue ipv6_addrconf: flags=0x40008 [ 5505.215953] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5505.225544] pending: addrconf_verify_work [ 5505.233494] workqueue knbd0-recv: flags=0x1a [ 5505.237483] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5505.246668] in-flight: 657:recv_work [ 5505.249602] workqueue ptlrpc_pinger: flags=0x2 [ 5505.255579] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5505.263183] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 5505.273573] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3546s workers=4 idle: 20831 12642 28859 [ 5505.279660] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5495s workers=2 idle: 659 [ 5516.063441] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [ 5516.081365] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5516.200733] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5516.218321] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5516.225547] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5516.239435] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5516.255123] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5516.259345] RAX: 0000000000007d8d RBX: ffff96848811cae0 RCX: 0000000000000001 [ 5516.263119] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 5516.267028] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5516.271507] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 5516.276379] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 5516.285298] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 5516.296116] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5516.306736] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 5516.313428] Call Trace: [ 5516.315494] [ 5516.323612] ? show_regs.cold.9+0x22/0x2f [ 5516.327759] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5516.335645] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5516.342961] ? watchdog+0x40/0x40 [ 5516.345652] ? hrtimer_interrupt+0x128/0x2c0 [ 5516.352026] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5516.361131] ? apic_timer_interrupt+0xf/0x20 [ 5516.366457] [ 5516.367502] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5516.377634] do_raw_spin_lock+0x177/0x190 [ 5516.381465] _raw_spin_lock+0x21/0x30 [ 5516.382616] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5516.389316] ? do_raw_spin_unlock+0x75/0x190 [ 5516.396109] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5516.404653] ? do_wait_intr+0xf0/0xf0 [ 5516.412708] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5516.416587] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5516.421542] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5516.428016] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5516.433211] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5516.436615] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5516.443970] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5516.447360] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5516.450479] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5516.453513] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5516.461345] lmv_intent_open+0x338/0x1150 [lmv] [ 5516.466695] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5516.469498] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5516.474455] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5516.477984] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5516.481069] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5516.483621] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5516.488443] ? do_raw_spin_unlock+0x75/0x190 [ 5516.491668] ? _raw_spin_unlock+0x12/0x30 [ 5516.494392] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5516.497488] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5516.501734] ll_atomic_open+0x36a/0xd70 [lustre] [ 5516.506428] lookup_open+0x356/0xb40 [ 5516.508066] do_last+0x313/0xfc0 [ 5516.509382] ? nd_jump_root+0xe5/0x160 [ 5516.511763] ? path_init+0x437/0x520 [ 5516.515066] path_openat+0xf7/0x500 [ 5516.518576] ? do_raw_spin_unlock+0x75/0x190 [ 5516.520567] do_filp_open+0x99/0x140 [ 5516.523520] ? getname_flags+0x6e/0x330 [ 5516.526949] ? __check_object_size+0xff/0x256 [ 5516.530487] ? do_raw_spin_unlock+0x75/0x190 [ 5516.535155] ? _raw_spin_unlock+0x12/0x30 [ 5516.539435] do_sys_openat2+0x2b4/0x410 [ 5516.541513] do_sys_open+0x73/0xa0 [ 5516.544477] __x64_sys_openat+0x24/0x30 [ 5516.546401] do_syscall_64+0xc1/0x440 [ 5516.547642] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5516.549212] RIP: 0033:0x7ff54af45332 [ 5516.551512] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5516.571141] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5516.577716] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 5516.584394] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 5516.586671] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 5516.591294] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 5516.596518] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 5520.063098] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 5520.069686] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5520.164342] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5520.183332] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5520.188850] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5520.192321] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5520.203833] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5520.208132] RAX: 0000000000007f19 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 5520.215339] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 5520.220035] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5520.224460] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 5520.232265] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 5520.236522] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 5520.247238] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5520.252342] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 5520.256387] Call Trace: [ 5520.257617] [ 5520.258642] ? show_regs.cold.9+0x22/0x2f [ 5520.260339] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5520.266698] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5520.273715] ? watchdog+0x40/0x40 [ 5520.276458] ? hrtimer_interrupt+0x128/0x2c0 [ 5520.279236] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5520.284321] ? apic_timer_interrupt+0xf/0x20 [ 5520.287299] [ 5520.289796] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5520.295623] do_raw_spin_lock+0x177/0x190 [ 5520.300579] _raw_spin_lock+0x21/0x30 [ 5520.302953] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5520.306317] ? do_raw_spin_unlock+0x75/0x190 [ 5520.308442] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5520.315733] ? do_wait_intr+0xf0/0xf0 [ 5520.318341] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5520.325309] mdc_close+0x53b/0xd70 [mdc] [ 5520.332088] lmv_close+0x18d/0x490 [lmv] [ 5520.335926] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 5520.340359] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5520.347037] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5520.353540] ll_md_close+0x274/0x800 [lustre] [ 5520.358083] ll_file_release+0x128/0xc50 [lustre] [ 5520.363579] __fput+0x10d/0x440 [ 5520.372068] ____fput+0x12/0x20 [ 5520.373257] task_work_run+0xc6/0x110 [ 5520.378379] exit_to_usermode_loop+0x1dd/0x1f0 [ 5520.382348] do_syscall_64+0x3d6/0x440 [ 5520.386327] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5520.388309] RIP: 0033:0x7ff54af45cd8 [ 5520.391585] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5520.418776] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5520.422355] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 5520.430260] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 5520.442692] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 5520.455503] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 5520.462351] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5528.061747] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 5528.069710] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5528.101762] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5528.109293] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5528.113263] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5528.117099] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5528.119587] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5528.131262] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5528.135235] RAX: 0000000000005989 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 5528.138670] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 5528.142319] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 5528.146654] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 5528.150447] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 5528.153951] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 5528.158162] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5528.160958] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 5528.165297] Call Trace: [ 5528.166420] [ 5528.167441] ? show_regs.cold.9+0x22/0x2f [ 5528.169501] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5528.172381] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5528.174465] ? watchdog+0x40/0x40 [ 5528.176962] ? hrtimer_interrupt+0x128/0x2c0 [ 5528.179005] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5528.181628] ? apic_timer_interrupt+0xf/0x20 [ 5528.183990] [ 5528.184973] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5528.187556] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5528.190265] do_raw_spin_lock+0x177/0x190 [ 5528.192669] _raw_spin_lock+0x21/0x30 [ 5528.195266] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5528.198076] process_one_work+0x2c8/0x700 [ 5528.200233] worker_thread+0x296/0x6e0 [ 5528.201989] ? rescuer_thread+0x570/0x570 [ 5528.203646] kthread+0x1d1/0x200 [ 5528.205202] ? set_kthread_struct+0x70/0x70 [ 5528.207007] ret_from_fork+0x1f/0x30 [ 5532.055158] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [ 5532.060266] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5532.116168] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5532.124871] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5532.133347] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5532.136963] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5532.148581] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5532.152275] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 5532.155929] RDX: 000000000000674f RSI: 0000000000000003 RDI: ffff96848811cae0 [ 5532.158374] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 5532.161898] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 5532.165661] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5532.168798] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 5532.177238] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5532.180920] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 5532.183524] Call Trace: [ 5532.184465] [ 5532.185710] ? show_regs.cold.9+0x22/0x2f [ 5532.187815] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5532.191511] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5532.194210] ? watchdog+0x40/0x40 [ 5532.198366] ? hrtimer_interrupt+0x128/0x2c0 [ 5532.200240] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5532.202509] ? apic_timer_interrupt+0xf/0x20 [ 5532.205190] [ 5532.206543] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5532.209531] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5532.212509] do_raw_spin_lock+0x177/0x190 [ 5532.215547] _raw_spin_lock+0x21/0x30 [ 5532.217615] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5532.221322] obd_mod_free+0x46/0x140 [ptlrpc] [ 5532.225501] mdc_commit_open+0x91/0xb0 [mdc] [ 5532.228342] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5532.230926] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5532.232499] after_reply+0xe2c/0x1780 [ptlrpc] [ 5532.234398] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5532.236359] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5532.238903] ? do_wait_intr+0xf0/0xf0 [ 5532.240382] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5532.242503] kthread+0x1d1/0x200 [ 5532.243408] ? set_kthread_struct+0x70/0x70 [ 5532.245019] ret_from_fork+0x1f/0x30 [ 5535.720241] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3576s! [ 5535.737098] Showing busy workqueues and worker pools: [ 5535.740445] workqueue events: flags=0x0 [ 5535.747288] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5535.753547] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 5535.762927] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5535.770844] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 5535.780207] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5535.792082] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 5535.800356] workqueue events_long: flags=0x0 [ 5535.804534] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5535.808642] pending: lu_objects_flush [obdclass] [ 5535.814030] workqueue events_unbound: flags=0x2 [ 5535.816499] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5535.820722] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5535.830810] workqueue events_power_efficient: flags=0x80 [ 5535.834659] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5535.838619] pending: neigh_periodic_work [ 5535.841208] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5535.846321] pending: do_cache_clean, check_lifetime [ 5535.849289] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5535.853240] pending: neigh_periodic_work [ 5535.855428] workqueue writeback: flags=0x4a [ 5535.858974] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5535.861870] pending: wb_workfn, wb_workfn [ 5535.864027] workqueue xprtiod: flags=0xa [ 5535.866047] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5535.873037] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 5535.881713] workqueue ipv6_addrconf: flags=0x40008 [ 5535.885658] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5535.890407] pending: addrconf_verify_work [ 5535.894989] workqueue knbd0-recv: flags=0x1a [ 5535.898012] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5535.907466] in-flight: 657:recv_work [ 5535.913354] workqueue ptlrpc_pinger: flags=0x2 [ 5535.918475] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5535.924887] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 5535.931666] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3577s workers=4 idle: 20831 12642 28859 [ 5535.939757] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5526s workers=2 idle: 659 [ 5544.060025] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [ 5544.063778] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5544.087693] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5544.093350] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5544.097949] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5544.099897] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5544.109687] RSP: 0000:ffffaf9fc15234f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5544.113840] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 5544.117263] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 5544.120778] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5544.126859] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 5544.130640] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 5544.133809] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 5544.138711] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5544.141865] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 5544.145029] Call Trace: [ 5544.146361] [ 5544.147366] ? show_regs.cold.9+0x22/0x2f [ 5544.149385] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5544.151959] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5544.154051] ? watchdog+0x40/0x40 [ 5544.156320] ? hrtimer_interrupt+0x128/0x2c0 [ 5544.158656] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5544.161149] ? apic_timer_interrupt+0xf/0x20 [ 5544.163931] [ 5544.164976] ? native_safe_halt+0x2e/0x50 [ 5544.167434] kvm_wait+0xcd/0xe0 [ 5544.169057] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5544.172683] do_raw_spin_lock+0x177/0x190 [ 5544.174371] _raw_spin_lock+0x21/0x30 [ 5544.176286] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5544.178927] ? do_raw_spin_unlock+0x75/0x190 [ 5544.181131] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5544.183965] ? do_wait_intr+0xf0/0xf0 [ 5544.186132] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5544.189321] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5544.192607] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5544.195833] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5544.198844] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5544.201833] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5544.203796] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5544.206623] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5544.209469] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5544.212827] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5544.216130] lmv_intent_open+0x338/0x1150 [lmv] [ 5544.219018] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5544.222116] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5544.225087] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5544.227395] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5544.230544] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5544.233407] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5544.236351] ? do_raw_spin_unlock+0x75/0x190 [ 5544.238986] ? _raw_spin_unlock+0x12/0x30 [ 5544.240892] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5544.244421] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5544.246957] ll_atomic_open+0x36a/0xd70 [lustre] [ 5544.249431] lookup_open+0x356/0xb40 [ 5544.251814] do_last+0x313/0xfc0 [ 5544.252961] ? nd_jump_root+0xe5/0x160 [ 5544.254739] ? path_init+0x437/0x520 [ 5544.256356] path_openat+0xf7/0x500 [ 5544.258305] ? do_raw_spin_unlock+0x75/0x190 [ 5544.260169] do_filp_open+0x99/0x140 [ 5544.261858] ? getname_flags+0x6e/0x330 [ 5544.264338] ? __check_object_size+0xff/0x256 [ 5544.267418] ? do_raw_spin_unlock+0x75/0x190 [ 5544.270049] ? _raw_spin_unlock+0x12/0x30 [ 5544.272763] do_sys_openat2+0x2b4/0x410 [ 5544.275191] do_sys_open+0x73/0xa0 [ 5544.277115] __x64_sys_openat+0x24/0x30 [ 5544.278505] do_syscall_64+0xc1/0x440 [ 5544.280300] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5544.285406] RIP: 0033:0x7ff54af45332 [ 5544.287373] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5544.298418] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5544.314913] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 5544.325553] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 5544.334142] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 5544.342330] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 5544.353450] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 5548.058935] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [ 5548.069292] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5548.110848] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5548.116476] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5548.124324] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5548.127793] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5548.143262] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5548.149920] RAX: 0000000000003309 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 5548.153957] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 5548.157005] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5548.159818] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 5548.162174] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 5548.166931] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 5548.171868] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5548.174544] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 5548.178247] Call Trace: [ 5548.184832] [ 5548.185565] ? show_regs.cold.9+0x22/0x2f [ 5548.187534] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5548.191172] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5548.197786] ? watchdog+0x40/0x40 [ 5548.203195] ? hrtimer_interrupt+0x128/0x2c0 [ 5548.204775] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5548.208698] ? apic_timer_interrupt+0xf/0x20 [ 5548.210085] [ 5548.210828] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5548.223296] do_raw_spin_lock+0x177/0x190 [ 5548.227910] _raw_spin_lock+0x21/0x30 [ 5548.230309] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5548.239791] ? do_raw_spin_unlock+0x75/0x190 [ 5548.244898] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5548.255313] ? do_wait_intr+0xf0/0xf0 [ 5548.257955] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5548.266267] mdc_close+0x53b/0xd70 [mdc] [ 5548.268449] lmv_close+0x18d/0x490 [lmv] [ 5548.271946] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 5548.276422] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5548.285619] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5548.294108] ll_md_close+0x274/0x800 [lustre] [ 5548.302740] ll_file_release+0x128/0xc50 [lustre] [ 5548.306879] __fput+0x10d/0x440 [ 5548.309872] ____fput+0x12/0x20 [ 5548.311481] task_work_run+0xc6/0x110 [ 5548.313327] exit_to_usermode_loop+0x1dd/0x1f0 [ 5548.324262] do_syscall_64+0x3d6/0x440 [ 5548.330422] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5548.332494] RIP: 0033:0x7ff54af45cd8 [ 5548.341354] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5548.361097] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5548.369581] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 5548.373892] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 5548.378486] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 5548.381845] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 5548.385399] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5556.061403] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 5556.068609] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5556.129896] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5556.142329] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5556.157858] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5556.167056] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5556.179112] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5556.203749] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5556.211351] RAX: 0000000000007487 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 5556.221599] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 5556.230728] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 5556.240316] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 5556.257043] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 5556.266927] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 5556.272027] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5556.279233] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 5556.289987] Call Trace: [ 5556.290975] [ 5556.295498] ? show_regs.cold.9+0x22/0x2f [ 5556.299692] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5556.305285] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5556.312330] ? watchdog+0x40/0x40 [ 5556.314318] ? hrtimer_interrupt+0x128/0x2c0 [ 5556.316341] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5556.325041] ? apic_timer_interrupt+0xf/0x20 [ 5556.334012] [ 5556.334821] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5556.338268] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5556.343325] do_raw_spin_lock+0x177/0x190 [ 5556.346142] _raw_spin_lock+0x21/0x30 [ 5556.351670] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5556.356736] process_one_work+0x2c8/0x700 [ 5556.359902] worker_thread+0x296/0x6e0 [ 5556.361474] ? rescuer_thread+0x570/0x570 [ 5556.363346] kthread+0x1d1/0x200 [ 5556.364502] ? set_kthread_struct+0x70/0x70 [ 5556.369235] ret_from_fork+0x1f/0x30 [ 5560.055011] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [ 5560.062447] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5560.115601] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5560.125137] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5560.132619] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5560.138852] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5560.154555] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5560.163744] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 5560.169204] RDX: 0000000000003892 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 5560.173552] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 5560.180385] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 5560.188154] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5560.197694] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 5560.203961] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5560.207437] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 5560.211140] Call Trace: [ 5560.213754] [ 5560.214806] ? show_regs.cold.9+0x22/0x2f [ 5560.217584] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5560.220036] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5560.222721] ? watchdog+0x40/0x40 [ 5560.224769] ? hrtimer_interrupt+0x128/0x2c0 [ 5560.227265] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5560.230474] ? apic_timer_interrupt+0xf/0x20 [ 5560.233140] [ 5560.234481] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5560.235930] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5560.237826] do_raw_spin_lock+0x177/0x190 [ 5560.240065] _raw_spin_lock+0x21/0x30 [ 5560.242265] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5560.246140] obd_mod_free+0x46/0x140 [ptlrpc] [ 5560.248092] mdc_commit_open+0x91/0xb0 [mdc] [ 5560.249797] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5560.253632] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5560.255153] after_reply+0xe2c/0x1780 [ptlrpc] [ 5560.262130] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5560.263415] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5560.267271] ? do_wait_intr+0xf0/0xf0 [ 5560.269685] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5560.273307] kthread+0x1d1/0x200 [ 5560.275143] ? set_kthread_struct+0x70/0x70 [ 5560.277762] ret_from_fork+0x1f/0x30 [ 5566.431044] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3607s! [ 5566.433991] Showing busy workqueues and worker pools: [ 5566.435808] workqueue events: flags=0x0 [ 5566.437176] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5566.439354] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 5566.444232] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5566.447184] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 5566.463638] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5566.471403] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 5566.478989] workqueue events_long: flags=0x0 [ 5566.485719] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5566.493195] pending: lu_objects_flush [obdclass] [ 5566.495795] workqueue events_unbound: flags=0x2 [ 5566.499405] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5566.517157] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5566.537197] workqueue events_power_efficient: flags=0x80 [ 5566.542024] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5566.555682] pending: neigh_periodic_work [ 5566.558737] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5566.561926] pending: do_cache_clean, check_lifetime [ 5566.565770] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5566.576867] pending: neigh_periodic_work [ 5566.580721] workqueue writeback: flags=0x4a [ 5566.590892] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5566.595812] pending: wb_workfn, wb_workfn [ 5566.600240] workqueue xprtiod: flags=0xa [ 5566.603173] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5566.605222] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 5566.608830] workqueue ipv6_addrconf: flags=0x40008 [ 5566.611128] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5566.614062] pending: addrconf_verify_work [ 5566.615756] workqueue knbd0-recv: flags=0x1a [ 5566.617557] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5566.621351] in-flight: 657:recv_work [ 5566.625648] workqueue ptlrpc_pinger: flags=0x2 [ 5566.628984] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5566.635529] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 5566.640938] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3607s workers=4 idle: 20831 12642 28859 [ 5566.644460] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5556s workers=2 idle: 659 [ 5572.061714] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 5572.077294] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5572.127871] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5572.139942] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5572.147749] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5572.156013] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5572.174519] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5572.179998] RAX: 000000000000698d RBX: ffff96848811cae0 RCX: 0000000000000001 [ 5572.184257] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 5572.192882] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5572.202005] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 5572.215626] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 5572.228898] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 5572.241018] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5572.247958] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 5572.252645] Call Trace: [ 5572.254519] [ 5572.257790] ? show_regs.cold.9+0x22/0x2f [ 5572.259273] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5572.265188] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5572.269786] ? watchdog+0x40/0x40 [ 5572.273860] ? hrtimer_interrupt+0x128/0x2c0 [ 5572.277832] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5572.280489] ? apic_timer_interrupt+0xf/0x20 [ 5572.282896] [ 5572.283608] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5572.288158] do_raw_spin_lock+0x177/0x190 [ 5572.290166] _raw_spin_lock+0x21/0x30 [ 5572.292037] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5572.295260] ? do_raw_spin_unlock+0x75/0x190 [ 5572.297198] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5572.300695] ? do_wait_intr+0xf0/0xf0 [ 5572.301883] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5572.304493] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5572.306838] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5572.309642] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5572.311860] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5572.314636] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5572.316667] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5572.318617] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5572.320990] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5572.325049] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5572.327714] lmv_intent_open+0x338/0x1150 [lmv] [ 5572.330403] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5572.332482] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5572.337571] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5572.340702] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5572.342828] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5572.345506] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5572.347768] ? do_raw_spin_unlock+0x75/0x190 [ 5572.351124] ? _raw_spin_unlock+0x12/0x30 [ 5572.352614] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5572.354225] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5572.356866] ll_atomic_open+0x36a/0xd70 [lustre] [ 5572.360745] lookup_open+0x356/0xb40 [ 5572.362091] do_last+0x313/0xfc0 [ 5572.363777] ? nd_jump_root+0xe5/0x160 [ 5572.365738] ? path_init+0x437/0x520 [ 5572.367617] path_openat+0xf7/0x500 [ 5572.368955] ? do_raw_spin_unlock+0x75/0x190 [ 5572.370855] do_filp_open+0x99/0x140 [ 5572.372058] ? getname_flags+0x6e/0x330 [ 5572.375097] ? __check_object_size+0xff/0x256 [ 5572.379537] ? do_raw_spin_unlock+0x75/0x190 [ 5572.380849] ? _raw_spin_unlock+0x12/0x30 [ 5572.382940] do_sys_openat2+0x2b4/0x410 [ 5572.385869] do_sys_open+0x73/0xa0 [ 5572.388109] __x64_sys_openat+0x24/0x30 [ 5572.391173] do_syscall_64+0xc1/0x440 [ 5572.392986] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5572.395602] RIP: 0033:0x7ff54af45332 [ 5572.397810] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5572.410153] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5572.414667] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 5572.423308] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 5572.429110] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 5572.434004] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 5572.444469] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 5576.060708] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [ 5576.080987] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5576.133383] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5576.143802] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5576.156920] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5576.163810] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5576.179732] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5576.187804] RAX: 0000000000000582 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 5576.192422] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 5576.200192] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5576.207196] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 5576.217503] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 5576.222112] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 5576.227938] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5576.232329] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 5576.248419] Call Trace: [ 5576.249249] [ 5576.249971] ? show_regs.cold.9+0x22/0x2f [ 5576.268274] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5576.273557] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5576.282448] ? watchdog+0x40/0x40 [ 5576.285969] ? hrtimer_interrupt+0x128/0x2c0 [ 5576.299614] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5576.301049] ? apic_timer_interrupt+0xf/0x20 [ 5576.302191] [ 5576.310455] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5576.317569] do_raw_spin_lock+0x177/0x190 [ 5576.323663] _raw_spin_lock+0x21/0x30 [ 5576.328880] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5576.336891] ? do_raw_spin_unlock+0x75/0x190 [ 5576.343616] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5576.348539] ? do_wait_intr+0xf0/0xf0 [ 5576.350650] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5576.352893] mdc_close+0x53b/0xd70 [mdc] [ 5576.360498] lmv_close+0x18d/0x490 [lmv] [ 5576.367503] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 5576.370951] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5576.376087] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5576.377923] ll_md_close+0x274/0x800 [lustre] [ 5576.390380] ll_file_release+0x128/0xc50 [lustre] [ 5576.393141] __fput+0x10d/0x440 [ 5576.395805] ____fput+0x12/0x20 [ 5576.399745] task_work_run+0xc6/0x110 [ 5576.403840] exit_to_usermode_loop+0x1dd/0x1f0 [ 5576.405590] do_syscall_64+0x3d6/0x440 [ 5576.407634] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5576.410101] RIP: 0033:0x7ff54af45cd8 [ 5576.413868] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5576.429836] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5576.437761] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 5576.445698] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 5576.451040] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 5576.455386] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 5576.458854] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5584.059196] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 5584.061751] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5584.075444] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5584.078641] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5584.081091] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5584.083373] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5584.084738] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5584.090184] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5584.092428] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 5584.094517] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 5584.096929] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 5584.099140] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 5584.101063] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 5584.103275] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 5584.105628] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5584.107331] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 5584.109478] Call Trace: [ 5584.111547] [ 5584.112235] ? show_regs.cold.9+0x22/0x2f [ 5584.113448] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5584.114773] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5584.116716] ? watchdog+0x40/0x40 [ 5584.118466] ? hrtimer_interrupt+0x128/0x2c0 [ 5584.121019] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5584.122541] ? apic_timer_interrupt+0xf/0x20 [ 5584.123966] [ 5584.125039] ? native_safe_halt+0x2e/0x50 [ 5584.126482] kvm_wait+0xcd/0xe0 [ 5584.127671] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5584.129211] do_raw_spin_lock+0x177/0x190 [ 5584.130652] _raw_spin_lock+0x21/0x30 [ 5584.131834] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5584.134076] process_one_work+0x2c8/0x700 [ 5584.135414] worker_thread+0x296/0x6e0 [ 5584.136734] ? rescuer_thread+0x570/0x570 [ 5584.137897] kthread+0x1d1/0x200 [ 5584.138980] ? set_kthread_struct+0x70/0x70 [ 5584.140342] ret_from_fork+0x1f/0x30 [ 5592.057087] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [ 5592.065889] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5592.094223] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5592.098798] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5592.106074] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5592.111484] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5592.135360] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5592.139867] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 5592.145495] RDX: 00000000000001d2 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 5592.150626] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 5592.157283] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 5592.162797] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5592.167304] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 5592.171994] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5592.174602] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 5592.186468] Call Trace: [ 5592.189459] [ 5592.192327] ? show_regs.cold.9+0x22/0x2f [ 5592.194980] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5592.197467] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5592.199456] ? watchdog+0x40/0x40 [ 5592.200599] ? hrtimer_interrupt+0x128/0x2c0 [ 5592.202701] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5592.204992] ? apic_timer_interrupt+0xf/0x20 [ 5592.207600] [ 5592.208799] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5592.211500] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5592.214690] do_raw_spin_lock+0x177/0x190 [ 5592.218777] _raw_spin_lock+0x21/0x30 [ 5592.220792] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5592.223830] obd_mod_free+0x46/0x140 [ptlrpc] [ 5592.226214] mdc_commit_open+0x91/0xb0 [mdc] [ 5592.229981] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5592.233289] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5592.236772] after_reply+0xe2c/0x1780 [ptlrpc] [ 5592.238662] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5592.241191] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5592.244246] ? do_wait_intr+0xf0/0xf0 [ 5592.245711] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5592.248395] kthread+0x1d1/0x200 [ 5592.249712] ? set_kthread_struct+0x70/0x70 [ 5592.251666] ret_from_fork+0x1f/0x30 [ 5597.151118] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3638s! [ 5597.153910] Showing busy workqueues and worker pools: [ 5597.155987] workqueue events: flags=0x0 [ 5597.158180] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5597.165704] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 5597.170490] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5597.177784] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 5597.189780] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5597.195309] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 5597.199349] workqueue events_long: flags=0x0 [ 5597.201623] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5597.209051] pending: lu_objects_flush [obdclass] [ 5597.212646] workqueue events_unbound: flags=0x2 [ 5597.215307] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5597.218913] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5597.227293] workqueue events_power_efficient: flags=0x80 [ 5597.230435] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5597.234401] pending: neigh_periodic_work [ 5597.238104] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5597.244919] pending: do_cache_clean, check_lifetime [ 5597.253656] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5597.257533] pending: neigh_periodic_work [ 5597.259118] workqueue writeback: flags=0x4a [ 5597.261580] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5597.265321] pending: wb_workfn, wb_workfn [ 5597.268353] workqueue xprtiod: flags=0xa [ 5597.271644] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5597.275276] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 5597.279890] workqueue ipv6_addrconf: flags=0x40008 [ 5597.282615] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5597.288188] pending: addrconf_verify_work [ 5597.292300] workqueue knbd0-recv: flags=0x1a [ 5597.300861] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5597.309667] in-flight: 657:recv_work [ 5597.316393] workqueue ptlrpc_pinger: flags=0x2 [ 5597.319880] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5597.323108] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 5597.328432] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3638s workers=4 idle: 20831 12642 28859 [ 5597.334324] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5587s workers=2 idle: 659 [ 5600.060891] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 5600.075395] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5600.136037] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5600.145897] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5600.157431] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5600.163083] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5600.181060] RSP: 0000:ffffaf9fc15234f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5600.193088] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 5600.198902] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 5600.204768] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5600.214196] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 5600.224807] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 5600.230064] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 5600.239778] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5600.246291] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 5600.254598] Call Trace: [ 5600.258237] [ 5600.260821] ? show_regs.cold.9+0x22/0x2f [ 5600.272118] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5600.279886] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5600.289189] ? watchdog+0x40/0x40 [ 5600.294253] ? hrtimer_interrupt+0x128/0x2c0 [ 5600.302490] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5600.310479] ? apic_timer_interrupt+0xf/0x20 [ 5600.315864] [ 5600.318880] ? native_safe_halt+0x2e/0x50 [ 5600.323123] kvm_wait+0xcd/0xe0 [ 5600.327406] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5600.329375] do_raw_spin_lock+0x177/0x190 [ 5600.335894] _raw_spin_lock+0x21/0x30 [ 5600.341412] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5600.345165] ? do_raw_spin_unlock+0x75/0x190 [ 5600.350240] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5600.357100] ? do_wait_intr+0xf0/0xf0 [ 5600.360501] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5600.370003] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5600.375962] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5600.378604] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5600.385168] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5600.392428] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5600.407051] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5600.416452] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5600.427909] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5600.436297] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5600.445289] lmv_intent_open+0x338/0x1150 [lmv] [ 5600.447840] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5600.457290] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5600.465495] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5600.470443] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5600.473612] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5600.478445] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5600.483848] ? do_raw_spin_unlock+0x75/0x190 [ 5600.487459] ? _raw_spin_unlock+0x12/0x30 [ 5600.492099] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5600.497964] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5600.499866] ll_atomic_open+0x36a/0xd70 [lustre] [ 5600.510846] lookup_open+0x356/0xb40 [ 5600.518637] do_last+0x313/0xfc0 [ 5600.519766] ? nd_jump_root+0xe5/0x160 [ 5600.525393] ? path_init+0x437/0x520 [ 5600.529076] path_openat+0xf7/0x500 [ 5600.538407] ? do_raw_spin_unlock+0x75/0x190 [ 5600.544312] do_filp_open+0x99/0x140 [ 5600.548975] ? getname_flags+0x6e/0x330 [ 5600.550376] ? __check_object_size+0xff/0x256 [ 5600.560155] ? do_raw_spin_unlock+0x75/0x190 [ 5600.562439] ? _raw_spin_unlock+0x12/0x30 [ 5600.564374] do_sys_openat2+0x2b4/0x410 [ 5600.567379] do_sys_open+0x73/0xa0 [ 5600.571508] __x64_sys_openat+0x24/0x30 [ 5600.576410] do_syscall_64+0xc1/0x440 [ 5600.580222] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5600.588228] RIP: 0033:0x7ff54af45332 [ 5600.593175] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5600.608371] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5600.614468] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 5600.626424] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 5600.633966] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 5600.642307] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 5600.647739] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 5604.060064] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [ 5604.064267] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5604.111731] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5604.120708] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5604.132516] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5604.145819] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5604.182053] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5604.194716] RAX: 0000000000007e0a RBX: ffff96848811cae0 RCX: 0000000000000001 [ 5604.199979] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 5604.206893] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5604.208666] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 5604.216257] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 5604.224520] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 5604.234078] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5604.237141] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 5604.241130] Call Trace: [ 5604.243379] [ 5604.245866] ? show_regs.cold.9+0x22/0x2f [ 5604.247565] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5604.256299] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5604.261288] ? watchdog+0x40/0x40 [ 5604.262801] ? hrtimer_interrupt+0x128/0x2c0 [ 5604.265271] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5604.267731] ? apic_timer_interrupt+0xf/0x20 [ 5604.270335] [ 5604.271383] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5604.273861] do_raw_spin_lock+0x177/0x190 [ 5604.275730] _raw_spin_lock+0x21/0x30 [ 5604.277464] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5604.280480] ? do_raw_spin_unlock+0x75/0x190 [ 5604.282640] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5604.285260] ? do_wait_intr+0xf0/0xf0 [ 5604.287743] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5604.290305] mdc_close+0x53b/0xd70 [mdc] [ 5604.292483] lmv_close+0x18d/0x490 [lmv] [ 5604.295462] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 5604.297903] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5604.299923] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5604.301418] ll_md_close+0x274/0x800 [lustre] [ 5604.305127] ll_file_release+0x128/0xc50 [lustre] [ 5604.306968] __fput+0x10d/0x440 [ 5604.307970] ____fput+0x12/0x20 [ 5604.308708] task_work_run+0xc6/0x110 [ 5604.309539] exit_to_usermode_loop+0x1dd/0x1f0 [ 5604.311207] do_syscall_64+0x3d6/0x440 [ 5604.312287] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5604.314044] RIP: 0033:0x7ff54af45cd8 [ 5604.314779] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5604.326180] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5604.332540] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 5604.342890] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 5604.356926] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 5604.365003] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 5604.375019] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5612.064050] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 5612.074081] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5612.108569] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5612.115781] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5612.121021] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5612.128259] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5612.130425] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5612.139608] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5612.143238] RAX: 0000000000007b5d RBX: ffff96848811cae0 RCX: 0000000000000001 [ 5612.147659] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 5612.151211] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 5612.156882] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 5612.159733] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 5612.163098] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 5612.165614] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5612.170711] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 5612.173637] Call Trace: [ 5612.174678] [ 5612.175867] ? show_regs.cold.9+0x22/0x2f [ 5612.181066] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5612.183165] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5612.185156] ? watchdog+0x40/0x40 [ 5612.187323] ? hrtimer_interrupt+0x128/0x2c0 [ 5612.188879] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5612.191392] ? apic_timer_interrupt+0xf/0x20 [ 5612.193328] [ 5612.194262] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5612.197179] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5612.199878] do_raw_spin_lock+0x177/0x190 [ 5612.201298] _raw_spin_lock+0x21/0x30 [ 5612.202699] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5612.209337] process_one_work+0x2c8/0x700 [ 5612.211579] worker_thread+0x296/0x6e0 [ 5612.217009] ? rescuer_thread+0x570/0x570 [ 5612.219436] kthread+0x1d1/0x200 [ 5612.221129] ? set_kthread_struct+0x70/0x70 [ 5612.224620] ret_from_fork+0x1f/0x30 [ 5613.948149] rcu: INFO: rcu_sched self-detected stall on CPU [ 5613.951140] rcu: 0-...!: (1901664 ticks this GP) idle=98a/1/0x4000000000000002 softirq=125154/125154 fqs=959 [ 5613.958985] (t=3660108 jiffies g=253585 q=3971) [ 5613.961214] rcu: rcu_sched kthread starved for 3655068 jiffies! g253585 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [ 5613.969268] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 5613.973683] rcu: RCU grace-period kthread stack dump: [ 5613.979172] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 5613.984289] Call Trace: [ 5613.985018] __schedule+0x351/0xcb0 [ 5613.986950] schedule+0xc0/0x180 [ 5613.988273] schedule_timeout+0xb4/0x190 [ 5613.989694] ? do_raw_spin_unlock+0x75/0x190 [ 5613.992531] ? __next_timer_interrupt+0x160/0x160 [ 5613.994665] ? prepare_to_swait_event+0xf3/0x140 [ 5613.996605] rcu_gp_kthread+0x8dd/0x15b0 [ 5613.999068] ? force_qs_rnp+0x2c0/0x2c0 [ 5614.000386] kthread+0x1d1/0x200 [ 5614.001537] ? set_kthread_struct+0x70/0x70 [ 5614.005632] ret_from_fork+0x1f/0x30 [ 5614.008014] rcu: Stack dump where RCU GP kthread last ran: [ 5614.010896] Sending NMI from CPU 0 to CPUs 2: [ 5614.016406] NMI backtrace for cpu 2 [ 5614.016413] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5614.016415] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5614.016417] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5614.016419] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5614.016421] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5614.016423] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 [ 5614.016425] RAX: 0000000000006fb7 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 5614.016426] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 5614.016428] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 5614.016429] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 5614.016430] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 5614.016431] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 5614.016433] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5614.016434] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 5614.016436] Call Trace: [ 5614.016437] [ 5614.016438] ? show_regs.cold.9+0x22/0x2f [ 5614.016439] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 5614.016440] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 5614.016441] ? nmi_handle+0xbd/0x210 [ 5614.016442] ? default_do_nmi+0x4b/0x1b0 [ 5614.016443] ? do_nmi+0x280/0x380 [ 5614.016444] ? end_repeat_nmi+0x16/0x1f [ 5614.016445] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5614.016447] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5614.016448] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5614.016449] [ 5614.016450] do_raw_spin_lock+0x177/0x190 [ 5614.016452] _raw_spin_lock+0x21/0x30 [ 5614.016453] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5614.016454] process_one_work+0x2c8/0x700 [ 5614.016455] worker_thread+0x296/0x6e0 [ 5614.016456] ? rescuer_thread+0x570/0x570 [ 5614.016458] kthread+0x1d1/0x200 [ 5614.016458] ? set_kthread_struct+0x70/0x70 [ 5614.016460] ret_from_fork+0x1f/0x30 [ 5614.017153] NMI backtrace for cpu 0 [ 5614.148930] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5614.156881] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5614.160944] Call Trace: [ 5614.162245] [ 5614.162944] ? dump_stack+0xbb/0x10e [ 5614.165104] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 5614.168191] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 5614.171769] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 5614.174678] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 5614.177257] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 5614.184668] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 5614.186958] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 5614.191447] ? update_process_times+0x6d/0xc0 [ 5614.192841] ? tick_sched_handle.isra.18+0x27/0x90 [ 5614.198497] ? tick_sched_timer+0x53/0xb0 [ 5614.200706] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5614.205998] ? tick_sched_handle.isra.18+0x90/0x90 [ 5614.210280] ? hrtimer_interrupt+0x128/0x2c0 [ 5614.212447] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5614.216943] ? apic_timer_interrupt+0xf/0x20 [ 5614.219608] [ 5614.225129] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5614.228607] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5614.235109] ? do_raw_spin_lock+0x177/0x190 [ 5614.238005] ? _raw_spin_lock+0x21/0x30 [ 5614.243687] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5614.249417] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 5614.251795] ? mdc_commit_open+0x91/0xb0 [mdc] [ 5614.258197] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5614.267002] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5614.273307] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 5614.278644] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5614.283174] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5614.290656] ? do_wait_intr+0xf0/0xf0 [ 5614.291978] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5614.300067] ? kthread+0x1d1/0x200 [ 5614.302563] ? set_kthread_struct+0x70/0x70 [ 5614.306689] ? ret_from_fork+0x1f/0x30 [ 5627.359245] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3668s! [ 5627.382370] Showing busy workqueues and worker pools: [ 5627.389410] workqueue events: flags=0x0 [ 5627.395189] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5627.401936] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 5627.416544] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5627.418790] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 5627.445367] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5627.463334] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 5627.474465] workqueue events_long: flags=0x0 [ 5627.477275] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5627.482301] pending: lu_objects_flush [obdclass] [ 5627.485190] workqueue events_unbound: flags=0x2 [ 5627.488014] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5627.495518] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5627.508261] workqueue events_power_efficient: flags=0x80 [ 5627.516654] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5627.519016] pending: neigh_periodic_work [ 5627.528316] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5627.540191] pending: do_cache_clean, check_lifetime [ 5627.549064] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5627.554009] pending: neigh_periodic_work [ 5627.557970] workqueue writeback: flags=0x4a [ 5627.559416] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5627.572023] pending: wb_workfn, wb_workfn [ 5627.588090] workqueue xprtiod: flags=0xa [ 5627.590478] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5627.600600] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 5627.609613] workqueue ipv6_addrconf: flags=0x40008 [ 5627.611292] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5627.616958] pending: addrconf_verify_work [ 5627.618617] workqueue knbd0-recv: flags=0x1a [ 5627.625492] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5627.630440] in-flight: 657:recv_work [ 5627.633596] workqueue ptlrpc_pinger: flags=0x2 [ 5627.637137] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5627.642195] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 5627.651201] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3668s workers=4 idle: 20831 12642 28859 [ 5627.663475] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5617s workers=2 idle: 659 [ 5628.060027] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 5628.063282] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5628.110370] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5628.129178] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5628.143472] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5628.147579] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5628.172336] RSP: 0000:ffffaf9fc15234f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5628.177609] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 5628.181414] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 5628.184617] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5628.189331] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 5628.193159] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 5628.197478] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 5628.204180] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5628.213267] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 5628.225261] Call Trace: [ 5628.228511] [ 5628.229765] ? show_regs.cold.9+0x22/0x2f [ 5628.237781] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5628.242597] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5628.249260] ? watchdog+0x40/0x40 [ 5628.252700] ? hrtimer_interrupt+0x128/0x2c0 [ 5628.253973] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5628.264258] ? apic_timer_interrupt+0xf/0x20 [ 5628.267688] [ 5628.269197] ? native_safe_halt+0x2e/0x50 [ 5628.274633] kvm_wait+0xcd/0xe0 [ 5628.279311] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5628.285516] do_raw_spin_lock+0x177/0x190 [ 5628.292054] _raw_spin_lock+0x21/0x30 [ 5628.299003] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5628.300813] ? do_raw_spin_unlock+0x75/0x190 [ 5628.311211] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5628.315566] ? do_wait_intr+0xf0/0xf0 [ 5628.328971] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5628.336713] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5628.340594] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5628.355112] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5628.363274] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5628.365904] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5628.370794] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5628.374147] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5628.380220] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5628.393207] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5628.398180] lmv_intent_open+0x338/0x1150 [lmv] [ 5628.405675] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5628.416824] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5628.420035] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5628.424879] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5628.427350] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5628.437500] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5628.443212] ? do_raw_spin_unlock+0x75/0x190 [ 5628.448347] ? _raw_spin_unlock+0x12/0x30 [ 5628.456409] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5628.460526] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5628.465784] ll_atomic_open+0x36a/0xd70 [lustre] [ 5628.468508] lookup_open+0x356/0xb40 [ 5628.472176] do_last+0x313/0xfc0 [ 5628.473631] ? nd_jump_root+0xe5/0x160 [ 5628.480792] ? path_init+0x437/0x520 [ 5628.481953] path_openat+0xf7/0x500 [ 5628.490297] ? do_raw_spin_unlock+0x75/0x190 [ 5628.494676] do_filp_open+0x99/0x140 [ 5628.497205] ? getname_flags+0x6e/0x330 [ 5628.501315] ? __check_object_size+0xff/0x256 [ 5628.507359] ? do_raw_spin_unlock+0x75/0x190 [ 5628.512455] ? _raw_spin_unlock+0x12/0x30 [ 5628.514347] do_sys_openat2+0x2b4/0x410 [ 5628.516363] do_sys_open+0x73/0xa0 [ 5628.518434] __x64_sys_openat+0x24/0x30 [ 5628.521147] do_syscall_64+0xc1/0x440 [ 5628.524623] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5628.528520] RIP: 0033:0x7ff54af45332 [ 5628.529639] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5628.568222] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5628.570596] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 5628.596485] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 5628.605102] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 5628.614652] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 5628.624478] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 5632.058566] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [ 5632.065939] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5632.106531] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5632.116406] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5632.126347] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5632.132665] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5632.149530] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5632.152253] RAX: 0000000000006d8b RBX: ffff96848811cae0 RCX: 0000000000000001 [ 5632.155667] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 5632.159302] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5632.168052] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 5632.176648] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 5632.182028] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 5632.190578] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5632.195234] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 5632.200401] Call Trace: [ 5632.201925] [ 5632.203191] ? show_regs.cold.9+0x22/0x2f [ 5632.205125] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5632.207908] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5632.209990] ? watchdog+0x40/0x40 [ 5632.224564] ? hrtimer_interrupt+0x128/0x2c0 [ 5632.231496] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5632.241433] ? apic_timer_interrupt+0xf/0x20 [ 5632.242930] [ 5632.243536] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5632.245778] do_raw_spin_lock+0x177/0x190 [ 5632.247132] _raw_spin_lock+0x21/0x30 [ 5632.248357] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5632.250199] ? do_raw_spin_unlock+0x75/0x190 [ 5632.251576] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5632.253298] ? do_wait_intr+0xf0/0xf0 [ 5632.254399] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5632.256095] mdc_close+0x53b/0xd70 [mdc] [ 5632.257426] lmv_close+0x18d/0x490 [lmv] [ 5632.258610] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 5632.260514] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5632.262200] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5632.264322] ll_md_close+0x274/0x800 [lustre] [ 5632.266081] ll_file_release+0x128/0xc50 [lustre] [ 5632.267603] __fput+0x10d/0x440 [ 5632.268548] ____fput+0x12/0x20 [ 5632.269766] task_work_run+0xc6/0x110 [ 5632.271041] exit_to_usermode_loop+0x1dd/0x1f0 [ 5632.272709] do_syscall_64+0x3d6/0x440 [ 5632.274148] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5632.276724] RIP: 0033:0x7ff54af45cd8 [ 5632.277581] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5632.284103] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5632.286283] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 5632.288248] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 5632.290319] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 5632.293726] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 5632.298065] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5640.059134] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 5640.066152] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5640.125164] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5640.139477] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5640.149319] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5640.155698] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5640.158832] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5640.190848] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5640.203344] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 5640.214919] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 5640.225206] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 5640.234299] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 5640.240060] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 5640.243283] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 5640.257456] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5640.265291] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 5640.274822] Call Trace: [ 5640.276030] [ 5640.277004] ? show_regs.cold.9+0x22/0x2f [ 5640.289338] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5640.299031] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5640.301148] ? watchdog+0x40/0x40 [ 5640.315389] ? hrtimer_interrupt+0x128/0x2c0 [ 5640.319131] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5640.326177] ? apic_timer_interrupt+0xf/0x20 [ 5640.335453] [ 5640.336950] ? native_safe_halt+0x2e/0x50 [ 5640.339435] kvm_wait+0xcd/0xe0 [ 5640.343321] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5640.351492] do_raw_spin_lock+0x177/0x190 [ 5640.354845] _raw_spin_lock+0x21/0x30 [ 5640.356300] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5640.366279] process_one_work+0x2c8/0x700 [ 5640.370191] worker_thread+0x296/0x6e0 [ 5640.375437] ? rescuer_thread+0x570/0x570 [ 5640.378376] kthread+0x1d1/0x200 [ 5640.382456] ? set_kthread_struct+0x70/0x70 [ 5640.390093] ret_from_fork+0x1f/0x30 [ 5652.058991] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 5652.065504] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5652.089945] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5652.094979] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5652.098756] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5652.102423] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5652.110397] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5652.114151] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 5652.118984] RDX: 0000000000007c0b RSI: 0000000000000003 RDI: ffff96848811cae0 [ 5652.121355] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 5652.127333] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 5652.131073] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5652.141243] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 5652.151377] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5652.159791] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 5652.167919] Call Trace: [ 5652.169881] [ 5652.170980] ? show_regs.cold.9+0x22/0x2f [ 5652.172627] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5652.175062] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5652.186145] ? watchdog+0x40/0x40 [ 5652.195824] ? hrtimer_interrupt+0x128/0x2c0 [ 5652.199778] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5652.204653] ? apic_timer_interrupt+0xf/0x20 [ 5652.210911] [ 5652.211589] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5652.219885] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5652.227135] do_raw_spin_lock+0x177/0x190 [ 5652.230411] _raw_spin_lock+0x21/0x30 [ 5652.233990] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5652.238707] obd_mod_free+0x46/0x140 [ptlrpc] [ 5652.242955] mdc_commit_open+0x91/0xb0 [mdc] [ 5652.247521] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5652.252025] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5652.257037] after_reply+0xe2c/0x1780 [ptlrpc] [ 5652.263177] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5652.269215] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5652.273414] ? do_wait_intr+0xf0/0xf0 [ 5652.276751] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5652.279850] kthread+0x1d1/0x200 [ 5652.282771] ? set_kthread_struct+0x70/0x70 [ 5652.284996] ret_from_fork+0x1f/0x30 [ 5656.066369] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 5656.085130] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5656.141324] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5656.150631] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5656.160186] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5656.162126] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5656.197579] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5656.211228] RAX: 000000000000474d RBX: ffff96848811cae0 RCX: 0000000000000001 [ 5656.215176] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 5656.224924] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5656.237043] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 5656.249154] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 5656.251559] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 5656.258288] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5656.267487] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 5656.278174] Call Trace: [ 5656.279033] [ 5656.286093] ? show_regs.cold.9+0x22/0x2f [ 5656.290286] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5656.294548] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5656.307618] ? watchdog+0x40/0x40 [ 5656.319226] ? hrtimer_interrupt+0x128/0x2c0 [ 5656.320737] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5656.328646] ? apic_timer_interrupt+0xf/0x20 [ 5656.330076] [ 5656.330832] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5656.344713] do_raw_spin_lock+0x177/0x190 [ 5656.349379] _raw_spin_lock+0x21/0x30 [ 5656.356184] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5656.364571] ? do_raw_spin_unlock+0x75/0x190 [ 5656.366332] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5656.373794] ? do_wait_intr+0xf0/0xf0 [ 5656.382780] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5656.388274] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5656.398086] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5656.402347] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5656.406832] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5656.411739] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5656.421546] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5656.428255] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5656.438792] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5656.441283] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5656.452769] lmv_intent_open+0x338/0x1150 [lmv] [ 5656.469295] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5656.471271] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5656.489480] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5656.497907] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5656.503889] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5656.510357] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5656.516303] ? do_raw_spin_unlock+0x75/0x190 [ 5656.522545] ? _raw_spin_unlock+0x12/0x30 [ 5656.530910] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5656.535317] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5656.540385] ll_atomic_open+0x36a/0xd70 [lustre] [ 5656.542033] lookup_open+0x356/0xb40 [ 5656.542990] do_last+0x313/0xfc0 [ 5656.554700] ? nd_jump_root+0xe5/0x160 [ 5656.560936] ? path_init+0x437/0x520 [ 5656.567175] path_openat+0xf7/0x500 [ 5656.570594] ? do_raw_spin_unlock+0x75/0x190 [ 5656.579003] do_filp_open+0x99/0x140 [ 5656.582640] ? getname_flags+0x6e/0x330 [ 5656.589332] ? __check_object_size+0xff/0x256 [ 5656.595441] ? do_raw_spin_unlock+0x75/0x190 [ 5656.601650] ? _raw_spin_unlock+0x12/0x30 [ 5656.603184] do_sys_openat2+0x2b4/0x410 [ 5656.605150] do_sys_open+0x73/0xa0 [ 5656.605957] __x64_sys_openat+0x24/0x30 [ 5656.606808] do_syscall_64+0xc1/0x440 [ 5656.611138] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5656.613234] RIP: 0033:0x7ff54af45332 [ 5656.614618] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5656.621611] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5656.624009] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 5656.626843] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 5656.631096] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 5656.638854] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 5656.644940] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 5658.083485] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3699s! [ 5658.092830] Showing busy workqueues and worker pools: [ 5658.097222] workqueue events: flags=0x0 [ 5658.105110] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5658.110019] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 5658.119647] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5658.123377] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 5658.139198] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5658.144372] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 5658.154212] workqueue events_long: flags=0x0 [ 5658.160675] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5658.162819] pending: lu_objects_flush [obdclass] [ 5658.170226] workqueue events_unbound: flags=0x2 [ 5658.175201] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5658.185032] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5658.192832] workqueue events_power_efficient: flags=0x80 [ 5658.198735] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5658.206014] pending: neigh_periodic_work [ 5658.214913] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5658.222169] pending: do_cache_clean, check_lifetime [ 5658.226217] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5658.234132] pending: neigh_periodic_work [ 5658.240521] workqueue writeback: flags=0x4a [ 5658.246699] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5658.250512] pending: wb_workfn, wb_workfn [ 5658.253701] workqueue xprtiod: flags=0xa [ 5658.256244] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5658.259895] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 5658.264538] workqueue ipv6_addrconf: flags=0x40008 [ 5658.269680] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5658.276558] pending: addrconf_verify_work [ 5658.282204] workqueue knbd0-recv: flags=0x1a [ 5658.286356] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5658.291202] in-flight: 657:recv_work [ 5658.292700] workqueue ptlrpc_pinger: flags=0x2 [ 5658.307349] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5658.317559] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 5658.324083] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3699s workers=4 idle: 20831 12642 28859 [ 5658.330536] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5648s workers=2 idle: 659 [ 5660.058023] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [ 5660.067321] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5660.110783] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5660.118796] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5660.125864] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5660.130733] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5660.148050] RSP: 0018:ffffaf9fc1543ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5660.152729] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 5660.156812] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 5660.165565] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5660.170463] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 5660.180138] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 5660.184186] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 5660.190881] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5660.200803] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 5660.210402] Call Trace: [ 5660.211400] [ 5660.212156] ? show_regs.cold.9+0x22/0x2f [ 5660.216546] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5660.217692] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5660.218812] ? watchdog+0x40/0x40 [ 5660.229238] ? hrtimer_interrupt+0x128/0x2c0 [ 5660.231796] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5660.243180] ? apic_timer_interrupt+0xf/0x20 [ 5660.247173] [ 5660.250403] ? native_safe_halt+0x2e/0x50 [ 5660.255591] kvm_wait+0xcd/0xe0 [ 5660.258478] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5660.261926] do_raw_spin_lock+0x177/0x190 [ 5660.265579] _raw_spin_lock+0x21/0x30 [ 5660.266867] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5660.276751] ? do_raw_spin_unlock+0x75/0x190 [ 5660.283575] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5660.288602] ? do_wait_intr+0xf0/0xf0 [ 5660.291329] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5660.298704] mdc_close+0x53b/0xd70 [mdc] [ 5660.300100] lmv_close+0x18d/0x490 [lmv] [ 5660.301460] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 5660.305135] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5660.309067] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5660.312272] ll_md_close+0x274/0x800 [lustre] [ 5660.316628] ll_file_release+0x128/0xc50 [lustre] [ 5660.322414] __fput+0x10d/0x440 [ 5660.323488] ____fput+0x12/0x20 [ 5660.324597] task_work_run+0xc6/0x110 [ 5660.327188] exit_to_usermode_loop+0x1dd/0x1f0 [ 5660.337225] do_syscall_64+0x3d6/0x440 [ 5660.339727] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5660.342547] RIP: 0033:0x7ff54af45cd8 [ 5660.345775] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5660.360721] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5660.362467] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 5660.367337] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 5660.374306] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 5660.378823] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 5660.383691] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5668.060176] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 5668.066096] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5668.116498] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5668.132342] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5668.138640] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5668.144414] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5668.149438] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5668.163601] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5668.169602] RAX: 0000000000005115 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 5668.176018] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 5668.181855] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 5668.187129] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 5668.193046] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 5668.198707] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 5668.209184] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5668.213143] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 5668.219577] Call Trace: [ 5668.221399] [ 5668.223288] ? show_regs.cold.9+0x22/0x2f [ 5668.226060] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5668.230348] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5668.234109] ? watchdog+0x40/0x40 [ 5668.236334] ? hrtimer_interrupt+0x128/0x2c0 [ 5668.239470] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5668.242935] ? apic_timer_interrupt+0xf/0x20 [ 5668.246136] [ 5668.247566] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5668.252142] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5668.257406] do_raw_spin_lock+0x177/0x190 [ 5668.260426] _raw_spin_lock+0x21/0x30 [ 5668.263287] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5668.268380] process_one_work+0x2c8/0x700 [ 5668.272003] worker_thread+0x296/0x6e0 [ 5668.274713] ? rescuer_thread+0x570/0x570 [ 5668.276482] kthread+0x1d1/0x200 [ 5668.278165] ? set_kthread_struct+0x70/0x70 [ 5668.280354] ret_from_fork+0x1f/0x30 [ 5684.058401] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 5684.060139] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 5684.061040] Modules linked in: [ 5684.066059] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5684.066143] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5684.066151] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5684.066154] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5684.066170] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5684.066175] RSP: 0000:ffffaf9fc15234f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5684.066179] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 5684.066182] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 5684.066184] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5684.066186] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 5684.066189] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 5684.067376] lustre(O) [ 5684.109102] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 5684.109115] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5684.109118] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 5684.109127] Call Trace: [ 5684.109135] [ 5684.109141] ? show_regs.cold.9+0x22/0x2f [ 5684.109158] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5684.109165] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5684.109170] ? watchdog+0x40/0x40 [ 5684.109174] ? hrtimer_interrupt+0x128/0x2c0 [ 5684.109179] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5684.109187] ? apic_timer_interrupt+0xf/0x20 [ 5684.109192] [ 5684.109196] ? native_safe_halt+0x2e/0x50 [ 5684.109201] kvm_wait+0xcd/0xe0 [ 5684.109210] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5684.109220] do_raw_spin_lock+0x177/0x190 [ 5684.109225] _raw_spin_lock+0x21/0x30 [ 5684.109229] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5684.109526] ? do_raw_spin_unlock+0x75/0x190 [ 5684.109531] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5684.109735] ? do_wait_intr+0xf0/0xf0 [ 5684.109743] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5684.109951] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5684.110177] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5684.126898] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5684.126979] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5684.126988] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5684.126992] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5684.127007] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5684.127011] RSP: 0000:ffffaf9fc1b97bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5684.127015] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 5684.127018] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 5684.127019] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 5684.127021] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 5684.127024] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5684.127027] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 5684.127030] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5684.127032] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 5684.127041] Call Trace: [ 5684.127047] [ 5684.127053] ? show_regs.cold.9+0x22/0x2f [ 5684.127065] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5684.127072] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5684.127078] ? watchdog+0x40/0x40 [ 5684.127083] ? hrtimer_interrupt+0x128/0x2c0 [ 5684.127087] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5684.127093] ? apic_timer_interrupt+0xf/0x20 [ 5684.127098] [ 5684.127101] ? native_safe_halt+0x2e/0x50 [ 5684.127104] kvm_wait+0xcd/0xe0 [ 5684.127112] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5684.127119] do_raw_spin_lock+0x177/0x190 [ 5684.127123] _raw_spin_lock+0x21/0x30 [ 5684.127128] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5684.127411] obd_mod_free+0x46/0x140 [ptlrpc] [ 5684.127603] mdc_commit_open+0x91/0xb0 [mdc] [ 5684.127644] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5684.127850] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5684.128064] after_reply+0xe2c/0x1780 [ptlrpc] [ 5684.128273] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5684.128482] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5684.128684] ? do_wait_intr+0xf0/0xf0 [ 5684.128694] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5684.128897] kthread+0x1d1/0x200 [ 5684.128903] ? set_kthread_struct+0x70/0x70 [ 5684.128907] ret_from_fork+0x1f/0x30 [ 5684.500908] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5684.503667] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5684.505752] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5684.507612] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5684.510035] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5684.514116] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5684.516382] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5684.518099] lmv_intent_open+0x338/0x1150 [lmv] [ 5684.520687] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5684.522702] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5684.526269] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5684.528635] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5684.531093] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5684.535778] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5684.538544] ? do_raw_spin_unlock+0x75/0x190 [ 5684.540879] ? _raw_spin_unlock+0x12/0x30 [ 5684.542396] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5684.544450] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5684.548951] ll_atomic_open+0x36a/0xd70 [lustre] [ 5684.551036] lookup_open+0x356/0xb40 [ 5684.552936] do_last+0x313/0xfc0 [ 5684.554388] ? nd_jump_root+0xe5/0x160 [ 5684.555644] ? path_init+0x437/0x520 [ 5684.556702] path_openat+0xf7/0x500 [ 5684.557559] ? do_raw_spin_unlock+0x75/0x190 [ 5684.559462] do_filp_open+0x99/0x140 [ 5684.560955] ? getname_flags+0x6e/0x330 [ 5684.562645] ? __check_object_size+0xff/0x256 [ 5684.565133] ? do_raw_spin_unlock+0x75/0x190 [ 5684.566733] ? _raw_spin_unlock+0x12/0x30 [ 5684.568024] do_sys_openat2+0x2b4/0x410 [ 5684.574949] do_sys_open+0x73/0xa0 [ 5684.575957] __x64_sys_openat+0x24/0x30 [ 5684.579997] do_syscall_64+0xc1/0x440 [ 5684.582914] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5684.587678] RIP: 0033:0x7ff54af45332 [ 5684.588621] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5684.596760] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5684.600823] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 5684.604511] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 5684.609060] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 5684.612415] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 5684.615781] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 5688.058205] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 5688.063942] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5688.101267] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5688.105955] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5688.110475] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5688.117625] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5688.133519] RSP: 0018:ffffaf9fc1543ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5688.137649] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 5688.143709] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 5688.149668] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5688.155401] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 5688.159922] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 5688.168275] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 5688.173564] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5688.176756] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 5688.181355] Call Trace: [ 5688.182688] [ 5688.189568] ? show_regs.cold.9+0x22/0x2f [ 5688.193302] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5688.197122] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5688.198355] ? watchdog+0x40/0x40 [ 5688.199207] ? hrtimer_interrupt+0x128/0x2c0 [ 5688.202942] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5688.204333] ? apic_timer_interrupt+0xf/0x20 [ 5688.210194] [ 5688.210703] ? native_safe_halt+0x2e/0x50 [ 5688.211837] kvm_wait+0xcd/0xe0 [ 5688.212638] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5688.214504] do_raw_spin_lock+0x177/0x190 [ 5688.215543] _raw_spin_lock+0x21/0x30 [ 5688.216744] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5688.218588] ? do_raw_spin_unlock+0x75/0x190 [ 5688.219799] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5688.221657] ? do_wait_intr+0xf0/0xf0 [ 5688.222563] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5688.223900] mdc_close+0x53b/0xd70 [mdc] [ 5688.225284] lmv_close+0x18d/0x490 [lmv] [ 5688.226274] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 5688.227798] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5688.229540] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5688.230720] ll_md_close+0x274/0x800 [lustre] [ 5688.231871] ll_file_release+0x128/0xc50 [lustre] [ 5688.233246] __fput+0x10d/0x440 [ 5688.233966] ____fput+0x12/0x20 [ 5688.234848] task_work_run+0xc6/0x110 [ 5688.235773] exit_to_usermode_loop+0x1dd/0x1f0 [ 5688.236987] do_syscall_64+0x3d6/0x440 [ 5688.237951] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5688.239652] RIP: 0033:0x7ff54af45cd8 [ 5688.240539] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5688.245517] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5688.247490] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 5688.249439] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 5688.251601] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 5688.253515] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 5688.255588] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5688.799048] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3729s! [ 5688.803701] Showing busy workqueues and worker pools: [ 5688.805996] workqueue events: flags=0x0 [ 5688.808727] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5688.812181] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 5688.817523] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5688.820504] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 5688.825649] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5688.828986] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 5688.832430] workqueue events_long: flags=0x0 [ 5688.834738] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5688.838518] pending: lu_objects_flush [obdclass] [ 5688.841050] workqueue events_unbound: flags=0x2 [ 5688.843689] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5688.846879] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5688.851802] workqueue events_power_efficient: flags=0x80 [ 5688.853968] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5688.856977] pending: neigh_periodic_work [ 5688.859233] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5688.863054] pending: do_cache_clean, check_lifetime [ 5688.865498] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5688.869157] pending: neigh_periodic_work [ 5688.871048] workqueue writeback: flags=0x4a [ 5688.872537] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5688.874800] pending: wb_workfn, wb_workfn [ 5688.877176] workqueue xprtiod: flags=0xa [ 5688.879064] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5688.881982] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 5688.886924] workqueue ipv6_addrconf: flags=0x40008 [ 5688.892518] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5688.894444] pending: addrconf_verify_work [ 5688.898715] workqueue knbd0-recv: flags=0x1a [ 5688.902668] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5688.910328] in-flight: 657:recv_work [ 5688.913390] workqueue ptlrpc_pinger: flags=0x2 [ 5688.916421] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5688.922412] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 5688.925495] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3730s workers=4 idle: 20831 12642 28859 [ 5688.928561] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5679s workers=2 idle: 659 [ 5696.063376] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 5696.070299] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5696.119252] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5696.127912] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5696.138371] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5696.142698] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5696.146327] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5696.154703] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5696.167747] RAX: 0000000000007608 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 5696.174936] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 5696.181575] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 5696.184558] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 5696.187089] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 5696.192568] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 5696.198323] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5696.202233] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 5696.204732] Call Trace: [ 5696.206496] [ 5696.208614] ? show_regs.cold.9+0x22/0x2f [ 5696.209983] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5696.222772] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5696.224346] ? watchdog+0x40/0x40 [ 5696.233580] ? hrtimer_interrupt+0x128/0x2c0 [ 5696.238978] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5696.242378] ? apic_timer_interrupt+0xf/0x20 [ 5696.244688] [ 5696.245372] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5696.249445] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5696.250944] do_raw_spin_lock+0x177/0x190 [ 5696.254057] _raw_spin_lock+0x21/0x30 [ 5696.257452] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5696.261447] process_one_work+0x2c8/0x700 [ 5696.271952] worker_thread+0x296/0x6e0 [ 5696.279194] ? rescuer_thread+0x570/0x570 [ 5696.283694] kthread+0x1d1/0x200 [ 5696.290403] ? set_kthread_struct+0x70/0x70 [ 5696.296701] ret_from_fork+0x1f/0x30 [ 5712.060012] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 5712.064122] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5712.108859] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5712.118832] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5712.124566] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5712.127968] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5712.142501] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5712.148432] RAX: 0000000000001a08 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 5712.153331] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 5712.157610] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5712.160368] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 5712.165579] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 5712.169522] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 5712.174609] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5712.179747] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 5712.186236] Call Trace: [ 5712.188210] [ 5712.190188] ? show_regs.cold.9+0x22/0x2f [ 5712.193293] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5712.196633] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5712.199188] ? watchdog+0x40/0x40 [ 5712.201544] ? hrtimer_interrupt+0x128/0x2c0 [ 5712.204305] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5712.207631] ? apic_timer_interrupt+0xf/0x20 [ 5712.211258] [ 5712.211956] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5712.214266] do_raw_spin_lock+0x177/0x190 [ 5712.217145] _raw_spin_lock+0x21/0x30 [ 5712.219314] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5712.222519] ? do_raw_spin_unlock+0x75/0x190 [ 5712.225279] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5712.229787] ? do_wait_intr+0xf0/0xf0 [ 5712.232509] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5712.239573] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5712.243005] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5712.247788] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5712.251811] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5712.254786] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5712.256597] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5712.258766] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5712.260872] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5712.265802] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5712.269775] lmv_intent_open+0x338/0x1150 [lmv] [ 5712.272792] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5712.278992] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5712.282272] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5712.285171] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5712.288238] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5712.293163] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5712.296634] ? do_raw_spin_unlock+0x75/0x190 [ 5712.299398] ? _raw_spin_unlock+0x12/0x30 [ 5712.300800] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5712.305178] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5712.308485] ll_atomic_open+0x36a/0xd70 [lustre] [ 5712.310596] lookup_open+0x356/0xb40 [ 5712.313330] do_last+0x313/0xfc0 [ 5712.315781] ? nd_jump_root+0xe5/0x160 [ 5712.318543] ? path_init+0x437/0x520 [ 5712.324308] path_openat+0xf7/0x500 [ 5712.325636] ? do_raw_spin_unlock+0x75/0x190 [ 5712.330348] do_filp_open+0x99/0x140 [ 5712.333217] ? getname_flags+0x6e/0x330 [ 5712.337339] ? __check_object_size+0xff/0x256 [ 5712.339280] ? do_raw_spin_unlock+0x75/0x190 [ 5712.342272] ? _raw_spin_unlock+0x12/0x30 [ 5712.354261] do_sys_openat2+0x2b4/0x410 [ 5712.358580] do_sys_open+0x73/0xa0 [ 5712.361449] __x64_sys_openat+0x24/0x30 [ 5712.365043] do_syscall_64+0xc1/0x440 [ 5712.367590] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5712.369327] RIP: 0033:0x7ff54af45332 [ 5712.371036] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5712.384795] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5712.389177] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 5712.393691] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 5712.397799] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 5712.403285] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 5712.409263] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 5716.058083] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 5716.060608] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 5716.063277] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5716.063357] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5716.063364] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5716.063367] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5716.063382] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5716.063603] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5716.063607] RAX: 0000000000004a6a RBX: ffff96848811cae0 RCX: 0000000000000001 [ 5716.063609] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 5716.063611] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5716.063614] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 5716.063616] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 5716.063619] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 5716.063622] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5716.063624] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 5716.063632] Call Trace: [ 5716.063637] [ 5716.063643] ? show_regs.cold.9+0x22/0x2f [ 5716.063652] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5716.063659] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5716.063665] ? watchdog+0x40/0x40 [ 5716.063669] ? hrtimer_interrupt+0x128/0x2c0 [ 5716.063673] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5716.063683] ? apic_timer_interrupt+0xf/0x20 [ 5716.063688] [ 5716.063692] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5716.063699] do_raw_spin_lock+0x177/0x190 [ 5716.063704] _raw_spin_lock+0x21/0x30 [ 5716.063709] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5716.064013] ? do_raw_spin_unlock+0x75/0x190 [ 5716.064020] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5716.064235] ? do_wait_intr+0xf0/0xf0 [ 5716.064243] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5716.064747] mdc_close+0x53b/0xd70 [mdc] [ 5716.064784] lmv_close+0x18d/0x490 [lmv] [ 5716.068576] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 5716.068740] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5716.069610] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5716.069692] ll_md_close+0x274/0x800 [lustre] [ 5716.069775] ll_file_release+0x128/0xc50 [lustre] [ 5716.070316] __fput+0x10d/0x440 [ 5716.070326] ____fput+0x12/0x20 [ 5716.070330] task_work_run+0xc6/0x110 [ 5716.070338] exit_to_usermode_loop+0x1dd/0x1f0 [ 5716.070347] do_syscall_64+0x3d6/0x440 [ 5716.070351] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5716.070359] RIP: 0033:0x7ff54af45cd8 [ 5716.070366] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5716.070370] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5716.070376] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 5716.070378] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 5716.070380] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 5716.070941] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 5716.070944] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5716.323124] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5716.347153] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5716.351531] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5716.359211] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5716.366418] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5716.384999] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5716.392202] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 5716.395279] RDX: 0000000000004a35 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 5716.398249] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 5716.401344] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 5716.405090] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5716.409558] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 5716.413337] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5716.418904] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 5716.426206] Call Trace: [ 5716.427464] [ 5716.429419] ? show_regs.cold.9+0x22/0x2f [ 5716.431070] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5716.436165] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5716.438217] ? watchdog+0x40/0x40 [ 5716.439774] ? hrtimer_interrupt+0x128/0x2c0 [ 5716.441770] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5716.444202] ? apic_timer_interrupt+0xf/0x20 [ 5716.446908] [ 5716.447918] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5716.450900] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5716.453619] do_raw_spin_lock+0x177/0x190 [ 5716.455102] _raw_spin_lock+0x21/0x30 [ 5716.456496] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5716.458511] obd_mod_free+0x46/0x140 [ptlrpc] [ 5716.460995] mdc_commit_open+0x91/0xb0 [mdc] [ 5716.462988] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5716.465362] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5716.467639] after_reply+0xe2c/0x1780 [ptlrpc] [ 5716.470073] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5716.473510] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5716.480896] ? do_wait_intr+0xf0/0xf0 [ 5716.486232] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5716.488411] kthread+0x1d1/0x200 [ 5716.492914] ? set_kthread_struct+0x70/0x70 [ 5716.494518] ret_from_fork+0x1f/0x30 [ 5719.007021] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3760s! [ 5719.015237] Showing busy workqueues and worker pools: [ 5719.020224] workqueue events: flags=0x0 [ 5719.026015] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5719.033142] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 5719.042080] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5719.046134] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 5719.058994] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5719.067500] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 5719.080146] workqueue events_long: flags=0x0 [ 5719.086343] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5719.092601] pending: lu_objects_flush [obdclass] [ 5719.094761] workqueue events_unbound: flags=0x2 [ 5719.096555] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5719.098986] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5719.106300] workqueue events_power_efficient: flags=0x80 [ 5719.108439] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5719.111945] pending: neigh_periodic_work [ 5719.118100] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5719.123454] pending: do_cache_clean, check_lifetime [ 5719.130460] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5719.138923] pending: neigh_periodic_work [ 5719.142670] workqueue writeback: flags=0x4a [ 5719.146409] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5719.150060] pending: wb_workfn, wb_workfn [ 5719.153037] workqueue xprtiod: flags=0xa [ 5719.163587] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5719.173689] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 5719.187503] workqueue ipv6_addrconf: flags=0x40008 [ 5719.194592] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5719.201496] pending: addrconf_verify_work [ 5719.204396] workqueue knbd0-recv: flags=0x1a [ 5719.207522] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5719.210834] in-flight: 657:recv_work [ 5719.213666] workqueue ptlrpc_pinger: flags=0x2 [ 5719.223762] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5719.228765] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 5719.237769] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3760s workers=4 idle: 20831 12642 28859 [ 5719.243964] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5709s workers=2 idle: 659 [ 5724.059179] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 5724.067491] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5724.099445] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5724.103973] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5724.108287] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5724.110184] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5724.112246] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5724.121390] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5724.124387] RAX: 0000000000005284 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 5724.126489] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 5724.129470] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 5724.132173] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 5724.134864] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 5724.137436] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 5724.147368] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5724.150637] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 5724.155264] Call Trace: [ 5724.156229] [ 5724.156864] ? show_regs.cold.9+0x22/0x2f [ 5724.158687] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5724.160791] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5724.163675] ? watchdog+0x40/0x40 [ 5724.165541] ? hrtimer_interrupt+0x128/0x2c0 [ 5724.169660] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5724.174673] ? apic_timer_interrupt+0xf/0x20 [ 5724.177694] [ 5724.178536] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5724.180602] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5724.183095] do_raw_spin_lock+0x177/0x190 [ 5724.184724] _raw_spin_lock+0x21/0x30 [ 5724.187137] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5724.189468] process_one_work+0x2c8/0x700 [ 5724.192362] worker_thread+0x296/0x6e0 [ 5724.193633] ? rescuer_thread+0x570/0x570 [ 5724.195088] kthread+0x1d1/0x200 [ 5724.196102] ? set_kthread_struct+0x70/0x70 [ 5724.197668] ret_from_fork+0x1f/0x30 [ 5740.060370] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 5740.069408] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5740.127685] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5740.141849] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5740.151209] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5740.152689] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5740.176267] RSP: 0000:ffffaf9fc15234f0 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 5740.186420] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 5740.191734] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 5740.197357] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5740.204947] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 5740.207842] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 5740.210177] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 5740.213341] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5740.215473] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 5740.217801] Call Trace: [ 5740.218669] [ 5740.219549] ? show_regs.cold.9+0x22/0x2f [ 5740.221097] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5740.222718] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5740.224214] ? watchdog+0x40/0x40 [ 5740.225467] ? hrtimer_interrupt+0x128/0x2c0 [ 5740.227135] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5740.228999] ? apic_timer_interrupt+0xf/0x20 [ 5740.230772] [ 5740.231539] ? native_safe_halt+0x2e/0x50 [ 5740.233107] kvm_wait+0xcd/0xe0 [ 5740.234280] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5740.236698] do_raw_spin_lock+0x177/0x190 [ 5740.238111] _raw_spin_lock+0x21/0x30 [ 5740.239384] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5740.241931] ? do_raw_spin_unlock+0x75/0x190 [ 5740.243824] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5740.246296] ? do_wait_intr+0xf0/0xf0 [ 5740.247743] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5740.249888] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5740.251942] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5740.254265] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5740.256370] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5740.258678] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5740.260538] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5740.262388] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5740.264969] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5740.267357] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5740.269889] lmv_intent_open+0x338/0x1150 [lmv] [ 5740.271655] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5740.274196] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5740.276328] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5740.277928] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5740.279636] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5740.281702] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5740.283403] ? do_raw_spin_unlock+0x75/0x190 [ 5740.285068] ? _raw_spin_unlock+0x12/0x30 [ 5740.286540] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5740.288503] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5740.290306] ll_atomic_open+0x36a/0xd70 [lustre] [ 5740.292799] lookup_open+0x356/0xb40 [ 5740.294138] do_last+0x313/0xfc0 [ 5740.295549] ? nd_jump_root+0xe5/0x160 [ 5740.297082] ? path_init+0x437/0x520 [ 5740.298497] path_openat+0xf7/0x500 [ 5740.299828] ? do_raw_spin_unlock+0x75/0x190 [ 5740.301544] do_filp_open+0x99/0x140 [ 5740.304502] ? getname_flags+0x6e/0x330 [ 5740.308237] ? __check_object_size+0xff/0x256 [ 5740.311220] ? do_raw_spin_unlock+0x75/0x190 [ 5740.313207] ? _raw_spin_unlock+0x12/0x30 [ 5740.314944] do_sys_openat2+0x2b4/0x410 [ 5740.319763] do_sys_open+0x73/0xa0 [ 5740.324869] __x64_sys_openat+0x24/0x30 [ 5740.327868] do_syscall_64+0xc1/0x440 [ 5740.330848] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5740.333511] RIP: 0033:0x7ff54af45332 [ 5740.336976] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5740.356310] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5740.365505] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 5740.374983] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 5740.379916] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 5740.391345] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 5740.398383] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 5744.055024] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 5744.058028] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 5744.059080] Modules linked in: lustre(O) [ 5744.062257] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5744.062343] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5744.062351] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5744.062354] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5744.062367] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5744.062372] RSP: 0018:ffffaf9fc1543ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5744.062377] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 5744.062379] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 5744.062381] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5744.062383] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 5744.062386] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 5744.062389] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 5744.062392] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5744.062395] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 5744.062402] Call Trace: [ 5744.062408] [ 5744.062414] ? show_regs.cold.9+0x22/0x2f [ 5744.062423] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5744.062430] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5744.062434] ? watchdog+0x40/0x40 [ 5744.062439] ? hrtimer_interrupt+0x128/0x2c0 [ 5744.062443] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5744.062450] ? apic_timer_interrupt+0xf/0x20 [ 5744.062455] [ 5744.062458] ? native_safe_halt+0x2e/0x50 [ 5744.062462] kvm_wait+0xcd/0xe0 [ 5744.062469] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5744.062477] do_raw_spin_lock+0x177/0x190 [ 5744.062483] _raw_spin_lock+0x21/0x30 [ 5744.062487] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5744.065378] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror [ 5744.106409] ? do_raw_spin_unlock+0x75/0x190 [ 5744.106430] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5744.107444] ? do_wait_intr+0xf0/0xf0 [ 5744.107453] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5744.110956] mdc_close+0x53b/0xd70 [mdc] [ 5744.111000] lmv_close+0x18d/0x490 [lmv] [ 5744.111028] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 5744.111185] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5744.111293] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5744.111372] ll_md_close+0x274/0x800 [lustre] [ 5744.111459] ll_file_release+0x128/0xc50 [lustre] [ 5744.111665] __fput+0x10d/0x440 [ 5744.111676] ____fput+0x12/0x20 [ 5744.111692] task_work_run+0xc6/0x110 [ 5744.111700] exit_to_usermode_loop+0x1dd/0x1f0 [ 5744.111707] do_syscall_64+0x3d6/0x440 [ 5744.111721] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5744.111731] RIP: 0033:0x7ff54af45cd8 [ 5744.111739] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5744.111743] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5744.111748] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 5744.111750] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 5744.111752] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 5744.111755] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 5744.111757] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5744.384514] dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5744.389634] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5744.397958] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5744.401986] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5744.404724] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5744.414346] RSP: 0000:ffffaf9fc1b97bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5744.418135] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 5744.421030] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 5744.424264] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 5744.426281] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 5744.429774] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5744.433391] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 5744.438851] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5744.441808] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 5744.446068] Call Trace: [ 5744.447320] [ 5744.448156] ? show_regs.cold.9+0x22/0x2f [ 5744.449808] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5744.451928] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5744.453635] ? watchdog+0x40/0x40 [ 5744.454796] ? hrtimer_interrupt+0x128/0x2c0 [ 5744.455965] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5744.457288] ? apic_timer_interrupt+0xf/0x20 [ 5744.458940] [ 5744.459751] ? native_safe_halt+0x2e/0x50 [ 5744.463623] kvm_wait+0xcd/0xe0 [ 5744.469201] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5744.471229] do_raw_spin_lock+0x177/0x190 [ 5744.476784] _raw_spin_lock+0x21/0x30 [ 5744.478180] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5744.480891] obd_mod_free+0x46/0x140 [ptlrpc] [ 5744.484844] mdc_commit_open+0x91/0xb0 [mdc] [ 5744.487140] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5744.489314] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5744.492807] after_reply+0xe2c/0x1780 [ptlrpc] [ 5744.497238] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5744.501658] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5744.504349] ? do_wait_intr+0xf0/0xf0 [ 5744.506230] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5744.508722] kthread+0x1d1/0x200 [ 5744.510865] ? set_kthread_struct+0x70/0x70 [ 5744.512621] ret_from_fork+0x1f/0x30 [ 5749.729770] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3790s! [ 5749.735466] Showing busy workqueues and worker pools: [ 5749.739862] workqueue events: flags=0x0 [ 5749.743355] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5749.749224] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 5749.758021] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5749.764046] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 5749.774030] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5749.779914] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 5749.786736] workqueue events_long: flags=0x0 [ 5749.788386] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5749.795146] pending: lu_objects_flush [obdclass] [ 5749.799474] workqueue events_unbound: flags=0x2 [ 5749.804139] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5749.810051] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5749.819879] workqueue events_power_efficient: flags=0x80 [ 5749.825253] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5749.831919] pending: neigh_periodic_work [ 5749.834037] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5749.839155] pending: do_cache_clean, check_lifetime [ 5749.843486] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5749.849832] pending: neigh_periodic_work [ 5749.852937] workqueue writeback: flags=0x4a [ 5749.857987] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5749.862461] pending: wb_workfn, wb_workfn [ 5749.866238] workqueue xprtiod: flags=0xa [ 5749.870946] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5749.875150] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 5749.883148] workqueue ipv6_addrconf: flags=0x40008 [ 5749.889306] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5749.891475] pending: addrconf_verify_work [ 5749.893450] workqueue knbd0-recv: flags=0x1a [ 5749.895774] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5749.899893] in-flight: 657:recv_work [ 5749.902226] workqueue ptlrpc_pinger: flags=0x2 [ 5749.904233] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5749.907473] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 5749.910335] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3791s workers=4 idle: 20831 12642 28859 [ 5749.914033] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5740s workers=2 idle: 659 [ 5752.059027] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 5752.065307] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5752.124403] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5752.139391] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5752.150453] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5752.153167] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5752.158180] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5752.172890] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5752.176889] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 5752.180081] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 5752.190930] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 5752.199153] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 5752.205348] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 5752.209106] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 5752.218397] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5752.225122] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 5752.232964] Call Trace: [ 5752.235946] [ 5752.238318] ? show_regs.cold.9+0x22/0x2f [ 5752.239742] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5752.242255] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5752.247341] ? watchdog+0x40/0x40 [ 5752.249349] ? hrtimer_interrupt+0x128/0x2c0 [ 5752.253259] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5752.257951] ? apic_timer_interrupt+0xf/0x20 [ 5752.260958] [ 5752.263190] ? native_safe_halt+0x2e/0x50 [ 5752.270294] kvm_wait+0xcd/0xe0 [ 5752.271534] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5752.276137] do_raw_spin_lock+0x177/0x190 [ 5752.281883] _raw_spin_lock+0x21/0x30 [ 5752.284724] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5752.286189] process_one_work+0x2c8/0x700 [ 5752.287069] worker_thread+0x296/0x6e0 [ 5752.294262] ? rescuer_thread+0x570/0x570 [ 5752.300149] kthread+0x1d1/0x200 [ 5752.304247] ? set_kthread_struct+0x70/0x70 [ 5752.307967] ret_from_fork+0x1f/0x30 [ 5768.060153] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 5768.074099] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5768.132856] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5768.145960] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5768.153799] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5768.155963] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5768.166742] RSP: 0000:ffffaf9fc15234f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5768.173151] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 5768.177701] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 5768.182733] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5768.187848] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 5768.195190] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 5768.202814] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 5768.218731] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5768.229661] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 5768.245447] Call Trace: [ 5768.246788] [ 5768.248762] ? show_regs.cold.9+0x22/0x2f [ 5768.255437] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5768.258598] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5768.266699] ? watchdog+0x40/0x40 [ 5768.274201] ? hrtimer_interrupt+0x128/0x2c0 [ 5768.277504] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5768.288675] ? apic_timer_interrupt+0xf/0x20 [ 5768.292627] [ 5768.294094] ? native_safe_halt+0x2e/0x50 [ 5768.296711] kvm_wait+0xcd/0xe0 [ 5768.299566] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5768.304201] do_raw_spin_lock+0x177/0x190 [ 5768.307701] _raw_spin_lock+0x21/0x30 [ 5768.309213] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5768.317721] ? do_raw_spin_unlock+0x75/0x190 [ 5768.322147] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5768.323962] ? do_wait_intr+0xf0/0xf0 [ 5768.325189] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5768.333094] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5768.339365] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5768.345720] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5768.351375] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5768.357763] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5768.366414] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5768.368730] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5768.374427] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5768.381957] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5768.385728] lmv_intent_open+0x338/0x1150 [lmv] [ 5768.388993] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5768.391430] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5768.394070] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5768.395900] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5768.398281] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5768.400227] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5768.401990] ? do_raw_spin_unlock+0x75/0x190 [ 5768.403625] ? _raw_spin_unlock+0x12/0x30 [ 5768.405063] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5768.406957] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5768.408903] ll_atomic_open+0x36a/0xd70 [lustre] [ 5768.410736] lookup_open+0x356/0xb40 [ 5768.412075] do_last+0x313/0xfc0 [ 5768.413242] ? nd_jump_root+0xe5/0x160 [ 5768.414588] ? path_init+0x437/0x520 [ 5768.415883] path_openat+0xf7/0x500 [ 5768.417279] ? do_raw_spin_unlock+0x75/0x190 [ 5768.418883] do_filp_open+0x99/0x140 [ 5768.420213] ? getname_flags+0x6e/0x330 [ 5768.421807] ? __check_object_size+0xff/0x256 [ 5768.423692] ? do_raw_spin_unlock+0x75/0x190 [ 5768.425416] ? _raw_spin_unlock+0x12/0x30 [ 5768.427172] do_sys_openat2+0x2b4/0x410 [ 5768.429713] do_sys_open+0x73/0xa0 [ 5768.430826] __x64_sys_openat+0x24/0x30 [ 5768.432671] do_syscall_64+0xc1/0x440 [ 5768.434361] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5768.436568] RIP: 0033:0x7ff54af45332 [ 5768.438081] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5768.464566] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5768.473384] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 5768.481022] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 5768.484641] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 5768.487451] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 5768.489792] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 5772.062135] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 5772.070711] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5772.125769] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5772.131836] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5772.137416] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5772.144636] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5772.165000] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5772.168775] RAX: 0000000000006f08 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 5772.172603] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 5772.181807] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5772.186333] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 5772.190785] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 5772.198951] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 5772.203075] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5772.204930] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 5772.212322] Call Trace: [ 5772.215230] [ 5772.216860] ? show_regs.cold.9+0x22/0x2f [ 5772.224312] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5772.227167] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5772.229708] ? watchdog+0x40/0x40 [ 5772.234617] ? hrtimer_interrupt+0x128/0x2c0 [ 5772.239225] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5772.244655] ? apic_timer_interrupt+0xf/0x20 [ 5772.247074] [ 5772.248899] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5772.251890] do_raw_spin_lock+0x177/0x190 [ 5772.256350] _raw_spin_lock+0x21/0x30 [ 5772.258754] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5772.262049] ? do_raw_spin_unlock+0x75/0x190 [ 5772.266544] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5772.271597] ? do_wait_intr+0xf0/0xf0 [ 5772.274244] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5772.281795] mdc_close+0x53b/0xd70 [mdc] [ 5772.287793] lmv_close+0x18d/0x490 [lmv] [ 5772.293137] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 5772.304216] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5772.317780] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5772.322861] ll_md_close+0x274/0x800 [lustre] [ 5772.332928] ll_file_release+0x128/0xc50 [lustre] [ 5772.335949] __fput+0x10d/0x440 [ 5772.337853] ____fput+0x12/0x20 [ 5772.341531] task_work_run+0xc6/0x110 [ 5772.343820] exit_to_usermode_loop+0x1dd/0x1f0 [ 5772.346047] do_syscall_64+0x3d6/0x440 [ 5772.347546] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5772.353118] RIP: 0033:0x7ff54af45cd8 [ 5772.355072] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5772.362963] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5772.366516] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 5772.376468] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 5772.379953] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 5772.384174] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 5772.391127] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5776.055466] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 5776.064501] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5776.122185] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5776.133373] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5776.143370] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5776.149677] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5776.165804] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5776.171230] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 5776.173028] RDX: 000000000000322c RSI: 0000000000000003 RDI: ffff96848811cae0 [ 5776.189442] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 5776.202055] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 5776.208476] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5776.224991] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 5776.242532] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5776.254602] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 5776.267339] Call Trace: [ 5776.268336] [ 5776.270281] ? show_regs.cold.9+0x22/0x2f [ 5776.273850] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5776.277467] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5776.281942] ? watchdog+0x40/0x40 [ 5776.285676] ? hrtimer_interrupt+0x128/0x2c0 [ 5776.289062] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5776.292585] ? apic_timer_interrupt+0xf/0x20 [ 5776.295616] [ 5776.296779] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5776.303110] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5776.309147] do_raw_spin_lock+0x177/0x190 [ 5776.313047] _raw_spin_lock+0x21/0x30 [ 5776.318101] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5776.323260] obd_mod_free+0x46/0x140 [ptlrpc] [ 5776.327736] mdc_commit_open+0x91/0xb0 [mdc] [ 5776.331678] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5776.335335] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5776.341680] after_reply+0xe2c/0x1780 [ptlrpc] [ 5776.345627] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5776.353291] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5776.359143] ? do_wait_intr+0xf0/0xf0 [ 5776.364735] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5776.372990] kthread+0x1d1/0x200 [ 5776.375460] ? set_kthread_struct+0x70/0x70 [ 5776.379142] ret_from_fork+0x1f/0x30 [ 5779.939657] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3821s! [ 5779.946502] Showing busy workqueues and worker pools: [ 5779.952158] workqueue events: flags=0x0 [ 5779.953544] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5779.968491] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 5779.976752] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5779.985949] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 5779.993717] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5780.000780] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 5780.014711] workqueue events_long: flags=0x0 [ 5780.016714] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5780.023441] pending: lu_objects_flush [obdclass] [ 5780.030520] workqueue events_unbound: flags=0x2 [ 5780.033576] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5780.041002] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5780.047325] workqueue events_power_efficient: flags=0x80 [ 5780.055879] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5780.059085] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 5780.059899] pending: neigh_periodic_work [ 5780.063960] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5780.064349] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5780.064358] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5780.064376] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5780.064908] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5780.064921] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5780.064926] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5780.064930] RAX: 000000000000088f RBX: ffff96848811cae0 RCX: 0000000000000001 [ 5780.064933] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 5780.064936] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 5780.064938] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 5780.064940] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 5780.064944] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 5780.064947] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5780.064949] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 5780.064959] Call Trace: [ 5780.064964] [ 5780.064970] ? show_regs.cold.9+0x22/0x2f [ 5780.064981] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5780.064987] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5780.064992] ? watchdog+0x40/0x40 [ 5780.064997] ? hrtimer_interrupt+0x128/0x2c0 [ 5780.065001] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5780.065181] ? apic_timer_interrupt+0xf/0x20 [ 5780.065186] [ 5780.065190] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5780.065195] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5780.065201] do_raw_spin_lock+0x177/0x190 [ 5780.065207] _raw_spin_lock+0x21/0x30 [ 5780.065212] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5780.065438] process_one_work+0x2c8/0x700 [ 5780.065446] worker_thread+0x296/0x6e0 [ 5780.065450] ? rescuer_thread+0x570/0x570 [ 5780.065455] kthread+0x1d1/0x200 [ 5780.065460] ? set_kthread_struct+0x70/0x70 [ 5780.065464] ret_from_fork+0x1f/0x30 [ 5780.256783] [ 5780.259761] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5780.263537] pending: do_cache_clean, check_lifetime [ 5780.274386] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5780.280082] pending: neigh_periodic_work [ 5780.283177] workqueue writeback: flags=0x4a [ 5780.285696] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5780.289328] pending: wb_workfn, wb_workfn [ 5780.291422] workqueue xprtiod: flags=0xa [ 5780.297995] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5780.300872] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 5780.304628] workqueue ipv6_addrconf: flags=0x40008 [ 5780.306876] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5780.309598] pending: addrconf_verify_work [ 5780.311294] workqueue knbd0-recv: flags=0x1a [ 5780.312935] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5780.315318] in-flight: 657:recv_work [ 5780.317269] workqueue ptlrpc_pinger: flags=0x2 [ 5780.318938] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5780.321094] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 5780.323354] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3821s workers=4 idle: 20831 12642 28859 [ 5780.326645] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5770s workers=2 idle: 659 [ 5793.954753] rcu: INFO: rcu_sched self-detected stall on CPU [ 5793.958961] rcu: 0-...!: (2000787 ticks this GP) idle=98a/1/0x4000000000000002 softirq=125154/125154 fqs=959 [ 5793.968603] (t=3840114 jiffies g=253585 q=4025) [ 5793.972242] rcu: rcu_sched kthread starved for 3835074 jiffies! g253585 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [ 5793.983545] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 5793.996818] rcu: RCU grace-period kthread stack dump: [ 5793.998342] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 5794.012973] Call Trace: [ 5794.016355] __schedule+0x351/0xcb0 [ 5794.018790] schedule+0xc0/0x180 [ 5794.022934] schedule_timeout+0xb4/0x190 [ 5794.025182] ? do_raw_spin_unlock+0x75/0x190 [ 5794.026688] ? __next_timer_interrupt+0x160/0x160 [ 5794.033546] ? prepare_to_swait_event+0xf3/0x140 [ 5794.039161] rcu_gp_kthread+0x8dd/0x15b0 [ 5794.040647] ? force_qs_rnp+0x2c0/0x2c0 [ 5794.044638] kthread+0x1d1/0x200 [ 5794.050824] ? set_kthread_struct+0x70/0x70 [ 5794.058942] ret_from_fork+0x1f/0x30 [ 5794.063412] rcu: Stack dump where RCU GP kthread last ran: [ 5794.067686] NMI backtrace for cpu 0 [ 5794.071571] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5794.088644] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5794.097014] Call Trace: [ 5794.098443] [ 5794.100810] ? dump_stack+0xbb/0x10e [ 5794.102625] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 5794.107862] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 5794.114034] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 5794.115770] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 5794.130373] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 5794.132647] ? rcu_check_gp_kthread_starvation+0x16f/0x188 [ 5794.135607] ? rcu_sched_clock_irq.cold.51+0xce/0x307 [ 5794.144016] ? update_process_times+0x6d/0xc0 [ 5794.148604] ? tick_sched_handle.isra.18+0x27/0x90 [ 5794.155147] ? tick_sched_timer+0x53/0xb0 [ 5794.157638] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5794.163904] ? tick_sched_handle.isra.18+0x90/0x90 [ 5794.169658] ? hrtimer_interrupt+0x128/0x2c0 [ 5794.172379] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5794.176548] ? apic_timer_interrupt+0xf/0x20 [ 5794.183271] [ 5794.184015] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5794.185828] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5794.196404] ? do_raw_spin_lock+0x177/0x190 [ 5794.199246] ? _raw_spin_lock+0x21/0x30 [ 5794.201358] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5794.204568] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 5794.207086] ? mdc_commit_open+0x91/0xb0 [mdc] [ 5794.209992] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5794.218478] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5794.225016] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 5794.229046] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5794.232349] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5794.241018] ? do_wait_intr+0xf0/0xf0 [ 5794.243995] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5794.248334] ? kthread+0x1d1/0x200 [ 5794.250609] ? set_kthread_struct+0x70/0x70 [ 5794.253345] ? ret_from_fork+0x1f/0x30 [ 5794.255854] NMI backtrace for cpu 0 [ 5794.260165] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5794.265517] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5794.284797] Call Trace: [ 5794.288441] [ 5794.295415] ? dump_stack+0xbb/0x10e [ 5794.299044] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 5794.306319] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 5794.311234] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 5794.313895] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 5794.319444] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 5794.325417] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 5794.328808] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 5794.335554] ? update_process_times+0x6d/0xc0 [ 5794.341151] ? tick_sched_handle.isra.18+0x27/0x90 [ 5794.345630] ? tick_sched_timer+0x53/0xb0 [ 5794.348848] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5794.356537] ? tick_sched_handle.isra.18+0x90/0x90 [ 5794.360637] ? hrtimer_interrupt+0x128/0x2c0 [ 5794.367505] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5794.376392] ? apic_timer_interrupt+0xf/0x20 [ 5794.381876] [ 5794.382626] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5794.394380] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5794.401934] ? do_raw_spin_lock+0x177/0x190 [ 5794.407608] ? _raw_spin_lock+0x21/0x30 [ 5794.408894] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5794.428771] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 5794.430566] ? mdc_commit_open+0x91/0xb0 [mdc] [ 5794.435638] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5794.440822] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5794.448158] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 5794.451525] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5794.456342] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5794.458335] ? do_wait_intr+0xf0/0xf0 [ 5794.461964] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5794.465382] ? kthread+0x1d1/0x200 [ 5794.466601] ? set_kthread_struct+0x70/0x70 [ 5794.468997] ? ret_from_fork+0x1f/0x30 [ 5796.061884] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [ 5796.066377] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5796.103733] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5796.111090] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5796.116417] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5796.119775] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5796.131935] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5796.137026] RAX: 0000000000001a8a RBX: ffff96848811cae0 RCX: 0000000000000001 [ 5796.141727] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 5796.148388] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5796.159377] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 5796.163982] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 5796.167194] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 5796.172698] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5796.175017] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 5796.176878] Call Trace: [ 5796.179357] [ 5796.179922] ? show_regs.cold.9+0x22/0x2f [ 5796.181022] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5796.183641] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5796.185730] ? watchdog+0x40/0x40 [ 5796.188202] ? hrtimer_interrupt+0x128/0x2c0 [ 5796.189939] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5796.191722] ? apic_timer_interrupt+0xf/0x20 [ 5796.195589] [ 5796.197565] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5796.199101] do_raw_spin_lock+0x177/0x190 [ 5796.204038] _raw_spin_lock+0x21/0x30 [ 5796.205948] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5796.209132] ? do_raw_spin_unlock+0x75/0x190 [ 5796.211288] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5796.213689] ? do_wait_intr+0xf0/0xf0 [ 5796.214817] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5796.219720] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5796.222444] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5796.225789] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5796.228813] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5796.231991] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5796.234976] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5796.237442] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5796.241499] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5796.246124] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5796.249325] lmv_intent_open+0x338/0x1150 [lmv] [ 5796.251998] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5796.254952] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5796.258309] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5796.260696] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5796.263347] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5796.266368] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5796.269249] ? do_raw_spin_unlock+0x75/0x190 [ 5796.271731] ? _raw_spin_unlock+0x12/0x30 [ 5796.273850] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5796.276696] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5796.279462] ll_atomic_open+0x36a/0xd70 [lustre] [ 5796.282227] lookup_open+0x356/0xb40 [ 5796.284187] do_last+0x313/0xfc0 [ 5796.285916] ? nd_jump_root+0xe5/0x160 [ 5796.287968] ? path_init+0x437/0x520 [ 5796.289862] path_openat+0xf7/0x500 [ 5796.291815] ? do_raw_spin_unlock+0x75/0x190 [ 5796.294113] do_filp_open+0x99/0x140 [ 5796.296011] ? getname_flags+0x6e/0x330 [ 5796.298224] ? __check_object_size+0xff/0x256 [ 5796.300778] ? do_raw_spin_unlock+0x75/0x190 [ 5796.303161] ? _raw_spin_unlock+0x12/0x30 [ 5796.305252] do_sys_openat2+0x2b4/0x410 [ 5796.307278] do_sys_open+0x73/0xa0 [ 5796.309460] __x64_sys_openat+0x24/0x30 [ 5796.311777] do_syscall_64+0xc1/0x440 [ 5796.313973] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5796.316587] RIP: 0033:0x7ff54af45332 [ 5796.318531] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5796.328866] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5796.334278] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 5796.338438] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 5796.342375] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 5796.346378] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 5796.350022] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 5800.060930] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 5800.073043] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5800.120019] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5800.123842] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5800.128668] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5800.131278] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5800.141764] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5800.147865] RAX: 0000000000006584 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 5800.152314] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 5800.156740] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5800.160668] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 5800.165157] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 5800.170530] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 5800.173135] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5800.175868] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 5800.179625] Call Trace: [ 5800.180913] [ 5800.181975] ? show_regs.cold.9+0x22/0x2f [ 5800.183467] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5800.185113] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5800.186867] ? watchdog+0x40/0x40 [ 5800.188113] ? hrtimer_interrupt+0x128/0x2c0 [ 5800.190301] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5800.191763] ? apic_timer_interrupt+0xf/0x20 [ 5800.193525] [ 5800.194299] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5800.196521] do_raw_spin_lock+0x177/0x190 [ 5800.198046] _raw_spin_lock+0x21/0x30 [ 5800.199311] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5800.201226] ? do_raw_spin_unlock+0x75/0x190 [ 5800.202463] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5800.204271] ? do_wait_intr+0xf0/0xf0 [ 5800.205709] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5800.209113] mdc_close+0x53b/0xd70 [mdc] [ 5800.210808] lmv_close+0x18d/0x490 [lmv] [ 5800.212457] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 5800.214738] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5800.216939] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5800.218942] ll_md_close+0x274/0x800 [lustre] [ 5800.220827] ll_file_release+0x128/0xc50 [lustre] [ 5800.222880] __fput+0x10d/0x440 [ 5800.223885] ____fput+0x12/0x20 [ 5800.225122] task_work_run+0xc6/0x110 [ 5800.226487] exit_to_usermode_loop+0x1dd/0x1f0 [ 5800.227854] do_syscall_64+0x3d6/0x440 [ 5800.229118] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5800.233094] RIP: 0033:0x7ff54af45cd8 [ 5800.234506] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5800.254214] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5800.262842] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 5800.268937] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 5800.275268] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 5800.278219] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 5800.284926] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5808.060199] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 5808.066219] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5808.102725] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5808.113628] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5808.126444] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5808.128785] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5808.138517] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5808.156646] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 5808.162338] RAX: 000000000000118e RBX: ffff96848811cae0 RCX: 0000000000000001 [ 5808.173773] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 5808.178756] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 5808.185955] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 5808.190840] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 5808.201401] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 5808.211821] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5808.219451] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 5808.233758] Call Trace: [ 5808.242646] [ 5808.243744] ? show_regs.cold.9+0x22/0x2f [ 5808.249266] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5808.258114] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5808.263609] ? watchdog+0x40/0x40 [ 5808.268340] ? hrtimer_interrupt+0x128/0x2c0 [ 5808.269874] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5808.281433] ? apic_timer_interrupt+0xf/0x20 [ 5808.282845] [ 5808.298085] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5808.305552] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5808.310829] do_raw_spin_lock+0x177/0x190 [ 5808.312159] _raw_spin_lock+0x21/0x30 [ 5808.329526] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5808.337757] process_one_work+0x2c8/0x700 [ 5808.341126] worker_thread+0x296/0x6e0 [ 5808.343894] ? rescuer_thread+0x570/0x570 [ 5808.349885] kthread+0x1d1/0x200 [ 5808.351039] ? set_kthread_struct+0x70/0x70 [ 5808.375309] ret_from_fork+0x1f/0x30 [ 5810.657469] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3851s! [ 5810.665454] Showing busy workqueues and worker pools: [ 5810.668547] workqueue events: flags=0x0 [ 5810.671540] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5810.679565] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 5810.683346] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5810.689936] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 5810.703036] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5810.712242] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 5810.718424] workqueue events_long: flags=0x0 [ 5810.723411] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5810.728390] pending: lu_objects_flush [obdclass] [ 5810.731594] workqueue events_unbound: flags=0x2 [ 5810.733030] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5810.738368] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5810.757768] workqueue events_power_efficient: flags=0x80 [ 5810.764026] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5810.772539] pending: neigh_periodic_work [ 5810.775780] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5810.780375] pending: do_cache_clean, check_lifetime [ 5810.782022] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5810.785999] pending: neigh_periodic_work [ 5810.788393] workqueue writeback: flags=0x4a [ 5810.790606] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5810.794654] pending: wb_workfn, wb_workfn [ 5810.796582] workqueue xprtiod: flags=0xa [ 5810.798447] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5810.800796] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 5810.803699] workqueue ipv6_addrconf: flags=0x40008 [ 5810.805647] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5810.808425] pending: addrconf_verify_work [ 5810.810617] workqueue knbd0-recv: flags=0x1a [ 5810.813673] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5810.817630] in-flight: 657:recv_work [ 5810.820679] workqueue ptlrpc_pinger: flags=0x2 [ 5810.822673] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5810.825413] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 5810.827728] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3851s workers=4 idle: 20831 12642 28859 [ 5810.831428] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5801s workers=2 idle: 659 [ 5824.060021] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [ 5824.062871] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5824.083224] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5824.089167] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5824.093322] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5824.095652] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5824.105270] RSP: 0000:ffffaf9fc15234f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5824.108571] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 5824.111573] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 5824.114714] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5824.118012] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 5824.121351] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 5824.124586] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 5824.128308] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5824.130949] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 5824.134305] Call Trace: [ 5824.135489] [ 5824.136399] ? show_regs.cold.9+0x22/0x2f [ 5824.138335] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5824.140488] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5824.142666] ? watchdog+0x40/0x40 [ 5824.144153] ? hrtimer_interrupt+0x128/0x2c0 [ 5824.146038] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5824.148065] ? apic_timer_interrupt+0xf/0x20 [ 5824.149889] [ 5824.150816] ? native_safe_halt+0x2e/0x50 [ 5824.152495] kvm_wait+0xcd/0xe0 [ 5824.153956] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5824.156198] do_raw_spin_lock+0x177/0x190 [ 5824.157895] _raw_spin_lock+0x21/0x30 [ 5824.159469] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5824.162138] ? do_raw_spin_unlock+0x75/0x190 [ 5824.164040] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5824.166335] ? do_wait_intr+0xf0/0xf0 [ 5824.168128] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5824.170462] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5824.172882] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5824.175524] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5824.177963] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5824.180292] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5824.182583] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5824.185233] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5824.188355] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5824.191128] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5824.193590] lmv_intent_open+0x338/0x1150 [lmv] [ 5824.196158] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5824.198668] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5824.201446] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5824.203515] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5824.205672] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5824.208076] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5824.210451] ? do_raw_spin_unlock+0x75/0x190 [ 5824.212880] ? _raw_spin_unlock+0x12/0x30 [ 5824.214968] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5824.217323] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5824.219964] ll_atomic_open+0x36a/0xd70 [lustre] [ 5824.222013] lookup_open+0x356/0xb40 [ 5824.223503] do_last+0x313/0xfc0 [ 5824.224972] ? nd_jump_root+0xe5/0x160 [ 5824.227180] ? path_init+0x437/0x520 [ 5824.229051] path_openat+0xf7/0x500 [ 5824.230881] ? do_raw_spin_unlock+0x75/0x190 [ 5824.233397] do_filp_open+0x99/0x140 [ 5824.235043] ? getname_flags+0x6e/0x330 [ 5824.236820] ? __check_object_size+0xff/0x256 [ 5824.239126] ? do_raw_spin_unlock+0x75/0x190 [ 5824.241058] ? _raw_spin_unlock+0x12/0x30 [ 5824.242751] do_sys_openat2+0x2b4/0x410 [ 5824.244734] do_sys_open+0x73/0xa0 [ 5824.246436] __x64_sys_openat+0x24/0x30 [ 5824.248152] do_syscall_64+0xc1/0x440 [ 5824.249884] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5824.252062] RIP: 0033:0x7ff54af45332 [ 5824.253631] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5824.263684] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5824.268514] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 5824.272667] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 5824.277024] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 5824.281673] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 5824.286497] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 5828.058018] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 5828.064178] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5828.132138] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5828.144636] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5828.159586] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5828.163699] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5828.199561] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5828.207817] RAX: 000000000000671c RBX: ffff96848811cae0 RCX: 0000000000000001 [ 5828.213063] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 5828.220272] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5828.225978] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 5828.233610] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 5828.241619] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 5828.246948] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5828.249740] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 5828.255131] Call Trace: [ 5828.256546] [ 5828.257848] ? show_regs.cold.9+0x22/0x2f [ 5828.259980] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5828.262920] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5828.265736] ? watchdog+0x40/0x40 [ 5828.266994] ? hrtimer_interrupt+0x128/0x2c0 [ 5828.269015] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5828.272296] ? apic_timer_interrupt+0xf/0x20 [ 5828.279532] [ 5828.282048] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5828.288525] do_raw_spin_lock+0x177/0x190 [ 5828.299931] _raw_spin_lock+0x21/0x30 [ 5828.303029] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5828.309666] ? do_raw_spin_unlock+0x75/0x190 [ 5828.315001] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5828.319262] ? do_wait_intr+0xf0/0xf0 [ 5828.321238] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5828.324462] mdc_close+0x53b/0xd70 [mdc] [ 5828.326677] lmv_close+0x18d/0x490 [lmv] [ 5828.328650] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 5828.331650] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5828.333928] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5828.335872] ll_md_close+0x274/0x800 [lustre] [ 5828.338812] ll_file_release+0x128/0xc50 [lustre] [ 5828.341521] __fput+0x10d/0x440 [ 5828.343221] ____fput+0x12/0x20 [ 5828.344924] task_work_run+0xc6/0x110 [ 5828.346558] exit_to_usermode_loop+0x1dd/0x1f0 [ 5828.348651] do_syscall_64+0x3d6/0x440 [ 5828.350503] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5828.353538] RIP: 0033:0x7ff54af45cd8 [ 5828.355561] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5828.367043] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5828.370643] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 5828.374794] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 5828.378335] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 5828.382373] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 5828.386024] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5836.055350] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [ 5836.059533] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 5836.064324] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5836.064406] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5836.064414] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5836.064417] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5836.064431] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5836.064436] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5836.064814] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 5836.064826] RDX: 0000000000002188 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 5836.064829] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 5836.064831] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 5836.064833] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5836.064837] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 5836.064840] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5836.064842] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 5836.064850] Call Trace: [ 5836.064855] [ 5836.064862] ? show_regs.cold.9+0x22/0x2f [ 5836.064872] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5836.064878] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5836.064882] ? watchdog+0x40/0x40 [ 5836.064887] ? hrtimer_interrupt+0x128/0x2c0 [ 5836.064892] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5836.064900] ? apic_timer_interrupt+0xf/0x20 [ 5836.064905] [ 5836.064908] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5836.064913] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5836.064918] do_raw_spin_lock+0x177/0x190 [ 5836.064923] _raw_spin_lock+0x21/0x30 [ 5836.064929] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5836.065413] obd_mod_free+0x46/0x140 [ptlrpc] [ 5836.079754] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5836.079843] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5836.079850] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5836.079856] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5836.080952] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5836.080965] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5836.080970] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5836.080975] RAX: 0000000000005e82 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 5836.080977] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 5836.080980] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 5836.080982] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 5836.080984] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 5836.080987] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 5836.080990] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5836.080992] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 5836.081002] Call Trace: [ 5836.081008] [ 5836.081013] ? show_regs.cold.9+0x22/0x2f [ 5836.081654] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5836.081665] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5836.081670] ? watchdog+0x40/0x40 [ 5836.081675] ? hrtimer_interrupt+0x128/0x2c0 [ 5836.081679] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5836.081688] ? apic_timer_interrupt+0xf/0x20 [ 5836.081755] [ 5836.081758] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5836.081763] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5836.081769] do_raw_spin_lock+0x177/0x190 [ 5836.081774] _raw_spin_lock+0x21/0x30 [ 5836.081780] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5836.138230] mdc_commit_open+0x91/0xb0 [mdc] [ 5836.138290] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5836.151225] process_one_work+0x2c8/0x700 [ 5836.151247] worker_thread+0x296/0x6e0 [ 5836.151252] ? rescuer_thread+0x570/0x570 [ 5836.151256] kthread+0x1d1/0x200 [ 5836.151266] ? set_kthread_struct+0x70/0x70 [ 5836.151270] ret_from_fork+0x1f/0x30 [ 5836.561129] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5836.568412] after_reply+0xe2c/0x1780 [ptlrpc] [ 5836.579570] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5836.583570] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5836.587284] ? do_wait_intr+0xf0/0xf0 [ 5836.588639] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5836.595550] kthread+0x1d1/0x200 [ 5836.599602] ? set_kthread_struct+0x70/0x70 [ 5836.602066] ret_from_fork+0x1f/0x30 [ 5840.864185] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3881s! [ 5840.869452] Showing busy workqueues and worker pools: [ 5840.872026] workqueue events: flags=0x0 [ 5840.873704] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5840.879193] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 5840.884878] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5840.887400] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 5840.894040] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5840.900013] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 5840.905155] workqueue events_long: flags=0x0 [ 5840.911640] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5840.917594] pending: lu_objects_flush [obdclass] [ 5840.920482] workqueue events_unbound: flags=0x2 [ 5840.922677] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5840.926367] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5840.937339] workqueue events_power_efficient: flags=0x80 [ 5840.946324] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5840.952292] pending: neigh_periodic_work [ 5840.955303] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5840.959287] pending: do_cache_clean, check_lifetime [ 5840.962059] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5840.966474] pending: neigh_periodic_work [ 5840.971976] workqueue writeback: flags=0x4a [ 5840.985353] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5840.996718] pending: wb_workfn, wb_workfn [ 5840.999182] workqueue xprtiod: flags=0xa [ 5841.012229] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5841.019718] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 5841.032213] workqueue ipv6_addrconf: flags=0x40008 [ 5841.038904] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5841.051847] pending: addrconf_verify_work [ 5841.056975] workqueue knbd0-recv: flags=0x1a [ 5841.058625] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5841.061431] in-flight: 657:recv_work [ 5841.067943] workqueue ptlrpc_pinger: flags=0x2 [ 5841.070057] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5841.072523] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 5841.075417] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3882s workers=4 idle: 20831 12642 28859 [ 5841.083072] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5831s workers=2 idle: 659 [ 5852.060263] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [ 5852.064822] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5852.100008] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5852.109287] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5852.113498] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5852.115545] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5852.124884] RSP: 0000:ffffaf9fc15234f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5852.128806] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 5852.131693] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 5852.136267] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5852.139629] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 5852.144199] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 5852.146526] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 5852.149315] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5852.151204] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 5852.154588] Call Trace: [ 5852.155450] [ 5852.155962] ? show_regs.cold.9+0x22/0x2f [ 5852.156967] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5852.158206] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5852.159571] ? watchdog+0x40/0x40 [ 5852.160844] ? hrtimer_interrupt+0x128/0x2c0 [ 5852.164129] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5852.166597] ? apic_timer_interrupt+0xf/0x20 [ 5852.167573] [ 5852.168518] ? native_safe_halt+0x2e/0x50 [ 5852.169947] kvm_wait+0xcd/0xe0 [ 5852.173325] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5852.177361] do_raw_spin_lock+0x177/0x190 [ 5852.179392] _raw_spin_lock+0x21/0x30 [ 5852.181241] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5852.184740] ? do_raw_spin_unlock+0x75/0x190 [ 5852.187400] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5852.190040] ? do_wait_intr+0xf0/0xf0 [ 5852.191829] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5852.194549] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5852.197867] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5852.199366] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5852.201929] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5852.204197] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5852.221777] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5852.226798] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5852.233317] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5852.242763] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5852.256404] lmv_intent_open+0x338/0x1150 [lmv] [ 5852.257995] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5852.259644] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5852.268436] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5852.273179] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5852.275972] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5852.279782] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5852.293256] ? do_raw_spin_unlock+0x75/0x190 [ 5852.299275] ? _raw_spin_unlock+0x12/0x30 [ 5852.307043] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5852.314362] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5852.320348] ll_atomic_open+0x36a/0xd70 [lustre] [ 5852.329035] lookup_open+0x356/0xb40 [ 5852.330715] do_last+0x313/0xfc0 [ 5852.337150] ? nd_jump_root+0xe5/0x160 [ 5852.348508] ? path_init+0x437/0x520 [ 5852.349767] path_openat+0xf7/0x500 [ 5852.356261] ? do_raw_spin_unlock+0x75/0x190 [ 5852.367322] do_filp_open+0x99/0x140 [ 5852.368455] ? getname_flags+0x6e/0x330 [ 5852.379209] ? __check_object_size+0xff/0x256 [ 5852.384634] ? do_raw_spin_unlock+0x75/0x190 [ 5852.387127] ? _raw_spin_unlock+0x12/0x30 [ 5852.388582] do_sys_openat2+0x2b4/0x410 [ 5852.390031] do_sys_open+0x73/0xa0 [ 5852.391313] __x64_sys_openat+0x24/0x30 [ 5852.392727] do_syscall_64+0xc1/0x440 [ 5852.394192] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5852.396217] RIP: 0033:0x7ff54af45332 [ 5852.397601] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5852.404837] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5852.407526] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 5852.410271] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 5852.413036] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 5852.415843] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 5852.418633] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 5856.059087] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 5856.070012] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5856.126786] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5856.142306] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5856.153131] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5856.158100] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5856.186185] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5856.190638] RAX: 0000000000007ca5 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 5856.194646] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 5856.198074] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5856.202661] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 5856.206892] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 5856.211113] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 5856.216153] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5856.219856] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 5856.224174] Call Trace: [ 5856.225731] [ 5856.226839] ? show_regs.cold.9+0x22/0x2f [ 5856.229286] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5856.232219] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5856.235006] ? watchdog+0x40/0x40 [ 5856.236952] ? hrtimer_interrupt+0x128/0x2c0 [ 5856.239427] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5856.242794] ? apic_timer_interrupt+0xf/0x20 [ 5856.245175] [ 5856.246773] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5856.249816] do_raw_spin_lock+0x177/0x190 [ 5856.252058] _raw_spin_lock+0x21/0x30 [ 5856.254342] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5856.258115] ? do_raw_spin_unlock+0x75/0x190 [ 5856.260773] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5856.263897] ? do_wait_intr+0xf0/0xf0 [ 5856.265726] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5856.268875] mdc_close+0x53b/0xd70 [mdc] [ 5856.270804] lmv_close+0x18d/0x490 [lmv] [ 5856.273187] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 5856.276757] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5856.280311] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5856.283661] ll_md_close+0x274/0x800 [lustre] [ 5856.286383] ll_file_release+0x128/0xc50 [lustre] [ 5856.289682] __fput+0x10d/0x440 [ 5856.291436] ____fput+0x12/0x20 [ 5856.293727] task_work_run+0xc6/0x110 [ 5856.296131] exit_to_usermode_loop+0x1dd/0x1f0 [ 5856.298800] do_syscall_64+0x3d6/0x440 [ 5856.301254] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5856.304705] RIP: 0033:0x7ff54af45cd8 [ 5856.306894] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5856.319341] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5856.323347] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 5856.327344] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 5856.332134] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 5856.337160] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 5856.341324] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5864.060770] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 5864.071060] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5864.134905] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5864.155163] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5864.172162] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5864.174400] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5864.182796] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5864.220438] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5864.229914] RAX: 0000000000005a66 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 5864.240518] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 5864.249034] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 5864.262047] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 5864.273748] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 5864.280418] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 5864.289312] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5864.296058] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 5864.311345] Call Trace: [ 5864.314103] [ 5864.319054] ? show_regs.cold.9+0x22/0x2f [ 5864.325362] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5864.331710] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5864.333268] ? watchdog+0x40/0x40 [ 5864.342955] ? hrtimer_interrupt+0x128/0x2c0 [ 5864.347887] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5864.361252] ? apic_timer_interrupt+0xf/0x20 [ 5864.370922] [ 5864.380983] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5864.384890] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5864.389767] do_raw_spin_lock+0x177/0x190 [ 5864.391025] _raw_spin_lock+0x21/0x30 [ 5864.401756] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5864.416564] process_one_work+0x2c8/0x700 [ 5864.418009] worker_thread+0x296/0x6e0 [ 5864.419277] ? rescuer_thread+0x570/0x570 [ 5864.428871] kthread+0x1d1/0x200 [ 5864.433678] ? set_kthread_struct+0x70/0x70 [ 5864.435573] ret_from_fork+0x1f/0x30 [ 5868.057463] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [ 5868.062150] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5868.091313] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5868.097034] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5868.104318] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5868.110140] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5868.123071] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5868.127432] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 5868.130689] RDX: 00000000000070e6 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 5868.134238] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 5868.138216] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 5868.142109] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5868.145858] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 5868.150556] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5868.152839] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 5868.157648] Call Trace: [ 5868.159049] [ 5868.160167] ? show_regs.cold.9+0x22/0x2f [ 5868.162767] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5868.165556] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5868.167903] ? watchdog+0x40/0x40 [ 5868.169802] ? hrtimer_interrupt+0x128/0x2c0 [ 5868.172059] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5868.174895] ? apic_timer_interrupt+0xf/0x20 [ 5868.176901] [ 5868.177740] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5868.181208] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5868.184241] do_raw_spin_lock+0x177/0x190 [ 5868.187075] _raw_spin_lock+0x21/0x30 [ 5868.189653] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5868.192665] obd_mod_free+0x46/0x140 [ptlrpc] [ 5868.196132] mdc_commit_open+0x91/0xb0 [mdc] [ 5868.198262] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5868.202020] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5868.208842] after_reply+0xe2c/0x1780 [ptlrpc] [ 5868.216169] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5868.220794] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5868.223227] ? do_wait_intr+0xf0/0xf0 [ 5868.225887] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5868.228958] kthread+0x1d1/0x200 [ 5868.231069] ? set_kthread_struct+0x70/0x70 [ 5868.234199] ret_from_fork+0x1f/0x30 [ 5871.583154] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3912s! [ 5871.587147] Showing busy workqueues and worker pools: [ 5871.590539] workqueue events: flags=0x0 [ 5871.598449] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5871.614738] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 5871.643059] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5871.654621] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 5871.674850] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5871.702796] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 5871.712642] workqueue events_long: flags=0x0 [ 5871.723030] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5871.747728] pending: lu_objects_flush [obdclass] [ 5871.765280] workqueue events_unbound: flags=0x2 [ 5871.777417] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5871.790137] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5871.805135] workqueue events_power_efficient: flags=0x80 [ 5871.811012] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5871.818702] pending: neigh_periodic_work [ 5871.822911] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5871.848847] pending: do_cache_clean, check_lifetime [ 5871.855290] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5871.869702] pending: neigh_periodic_work [ 5871.878623] workqueue writeback: flags=0x4a [ 5871.890453] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5871.900221] pending: wb_workfn, wb_workfn [ 5871.903011] workqueue xprtiod: flags=0xa [ 5871.905446] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5871.910625] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 5871.918996] workqueue ipv6_addrconf: flags=0x40008 [ 5871.924029] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5871.935613] pending: addrconf_verify_work [ 5871.941027] workqueue knbd0-recv: flags=0x1a [ 5871.946105] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5871.950748] in-flight: 657:recv_work [ 5871.952646] workqueue ptlrpc_pinger: flags=0x2 [ 5871.962629] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5871.969772] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 5871.983531] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3913s workers=4 idle: 20831 12642 28859 [ 5872.002135] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5862s workers=2 idle: 659 [ 5880.061743] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [ 5880.068703] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5880.118822] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5880.131584] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5880.145630] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5880.157832] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5880.173685] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5880.185401] RAX: 0000000000004f2b RBX: ffff96848811cae0 RCX: 0000000000000001 [ 5880.189677] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 5880.193505] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5880.199842] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 5880.202781] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 5880.208859] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 5880.216898] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5880.225102] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 5880.230608] Call Trace: [ 5880.233534] [ 5880.234430] ? show_regs.cold.9+0x22/0x2f [ 5880.237011] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5880.238960] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5880.241206] ? watchdog+0x40/0x40 [ 5880.243724] ? hrtimer_interrupt+0x128/0x2c0 [ 5880.245355] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5880.247637] ? apic_timer_interrupt+0xf/0x20 [ 5880.254033] [ 5880.254988] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5880.262678] do_raw_spin_lock+0x177/0x190 [ 5880.264130] _raw_spin_lock+0x21/0x30 [ 5880.272047] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5880.273689] ? do_raw_spin_unlock+0x75/0x190 [ 5880.279556] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5880.283346] ? do_wait_intr+0xf0/0xf0 [ 5880.286022] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5880.287995] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5880.293923] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5880.305402] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5880.310701] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5880.315014] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5880.323162] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5880.329893] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5880.332234] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5880.336459] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5880.346792] lmv_intent_open+0x338/0x1150 [lmv] [ 5880.350617] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5880.361401] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5880.368044] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5880.370069] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5880.380000] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5880.382642] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5880.395004] ? do_raw_spin_unlock+0x75/0x190 [ 5880.398300] ? _raw_spin_unlock+0x12/0x30 [ 5880.403317] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5880.405527] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5880.410351] ll_atomic_open+0x36a/0xd70 [lustre] [ 5880.414416] lookup_open+0x356/0xb40 [ 5880.419826] do_last+0x313/0xfc0 [ 5880.421938] ? nd_jump_root+0xe5/0x160 [ 5880.426666] ? path_init+0x437/0x520 [ 5880.435812] path_openat+0xf7/0x500 [ 5880.443241] ? do_raw_spin_unlock+0x75/0x190 [ 5880.450687] do_filp_open+0x99/0x140 [ 5880.453889] ? getname_flags+0x6e/0x330 [ 5880.455196] ? __check_object_size+0xff/0x256 [ 5880.461490] ? do_raw_spin_unlock+0x75/0x190 [ 5880.465457] ? _raw_spin_unlock+0x12/0x30 [ 5880.469699] do_sys_openat2+0x2b4/0x410 [ 5880.470906] do_sys_open+0x73/0xa0 [ 5880.473490] __x64_sys_openat+0x24/0x30 [ 5880.475446] do_syscall_64+0xc1/0x440 [ 5880.477359] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5880.481477] RIP: 0033:0x7ff54af45332 [ 5880.483028] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5880.504052] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5880.509358] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 5880.512935] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 5880.520981] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 5880.529425] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 5880.534425] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 5884.062477] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 5884.072049] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5884.103054] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5884.111753] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5884.115700] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5884.122383] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5884.137137] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5884.142082] RAX: 0000000000001f27 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 5884.150621] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 5884.155774] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5884.161946] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 5884.170679] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 5884.177269] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 5884.181723] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5884.186470] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 5884.190756] Call Trace: [ 5884.191719] [ 5884.192712] ? show_regs.cold.9+0x22/0x2f [ 5884.196580] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5884.201097] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5884.206802] ? watchdog+0x40/0x40 [ 5884.209217] ? hrtimer_interrupt+0x128/0x2c0 [ 5884.213491] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5884.218463] ? apic_timer_interrupt+0xf/0x20 [ 5884.221754] [ 5884.224403] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5884.229960] do_raw_spin_lock+0x177/0x190 [ 5884.232334] _raw_spin_lock+0x21/0x30 [ 5884.233980] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5884.236792] ? do_raw_spin_unlock+0x75/0x190 [ 5884.240152] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5884.251815] ? do_wait_intr+0xf0/0xf0 [ 5884.257163] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5884.262581] mdc_close+0x53b/0xd70 [mdc] [ 5884.265803] lmv_close+0x18d/0x490 [lmv] [ 5884.274970] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 5884.280906] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5884.285886] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5884.290429] ll_md_close+0x274/0x800 [lustre] [ 5884.293714] ll_file_release+0x128/0xc50 [lustre] [ 5884.299574] __fput+0x10d/0x440 [ 5884.303001] ____fput+0x12/0x20 [ 5884.310738] task_work_run+0xc6/0x110 [ 5884.313151] exit_to_usermode_loop+0x1dd/0x1f0 [ 5884.316015] do_syscall_64+0x3d6/0x440 [ 5884.321048] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5884.328030] RIP: 0033:0x7ff54af45cd8 [ 5884.333565] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5884.346003] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5884.356734] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 5884.369066] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 5884.372514] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 5884.380475] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 5884.384819] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5892.059034] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 5892.063487] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5892.100972] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5892.111939] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5892.123709] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5892.128955] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5892.131634] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5892.145973] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5892.149993] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 5892.164383] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 5892.168509] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 5892.176435] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 5892.185252] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 5892.190673] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 5892.193753] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5892.195752] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 5892.198620] Call Trace: [ 5892.199597] [ 5892.200424] ? show_regs.cold.9+0x22/0x2f [ 5892.201917] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5892.203973] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5892.205694] ? watchdog+0x40/0x40 [ 5892.206951] ? hrtimer_interrupt+0x128/0x2c0 [ 5892.208495] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5892.210726] ? apic_timer_interrupt+0xf/0x20 [ 5892.212378] [ 5892.213061] ? native_safe_halt+0x2e/0x50 [ 5892.215045] kvm_wait+0xcd/0xe0 [ 5892.216494] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5892.218593] do_raw_spin_lock+0x177/0x190 [ 5892.220050] _raw_spin_lock+0x21/0x30 [ 5892.221538] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5892.223812] process_one_work+0x2c8/0x700 [ 5892.225426] worker_thread+0x296/0x6e0 [ 5892.227093] ? rescuer_thread+0x570/0x570 [ 5892.228723] kthread+0x1d1/0x200 [ 5892.230043] ? set_kthread_struct+0x70/0x70 [ 5892.231915] ret_from_fork+0x1f/0x30 [ 5896.055022] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [ 5896.057190] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5896.089859] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5896.100475] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5896.106973] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5896.109362] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5896.126336] RSP: 0000:ffffaf9fc1b97bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5896.134742] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 5896.143421] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 5896.150458] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 5896.156947] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 5896.161382] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5896.166897] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 5896.174846] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5896.177468] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 5896.182388] Call Trace: [ 5896.186926] [ 5896.189670] ? show_regs.cold.9+0x22/0x2f [ 5896.196917] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5896.198351] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5896.200138] ? watchdog+0x40/0x40 [ 5896.201622] ? hrtimer_interrupt+0x128/0x2c0 [ 5896.203826] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5896.206848] ? apic_timer_interrupt+0xf/0x20 [ 5896.234157] [ 5896.235323] ? native_safe_halt+0x2e/0x50 [ 5896.237716] kvm_wait+0xcd/0xe0 [ 5896.238808] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5896.245395] do_raw_spin_lock+0x177/0x190 [ 5896.247350] _raw_spin_lock+0x21/0x30 [ 5896.248592] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5896.250785] obd_mod_free+0x46/0x140 [ptlrpc] [ 5896.252672] mdc_commit_open+0x91/0xb0 [mdc] [ 5896.254276] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5896.256162] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5896.258293] after_reply+0xe2c/0x1780 [ptlrpc] [ 5896.263327] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5896.266856] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5896.269390] ? do_wait_intr+0xf0/0xf0 [ 5896.271176] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5896.273571] kthread+0x1d1/0x200 [ 5896.275036] ? set_kthread_struct+0x70/0x70 [ 5896.277933] ret_from_fork+0x1f/0x30 [ 5902.303107] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3943s! [ 5902.307465] Showing busy workqueues and worker pools: [ 5902.317381] workqueue events: flags=0x0 [ 5902.320356] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5902.327206] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 5902.337167] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5902.341407] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 5902.355256] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5902.359358] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 5902.362450] workqueue events_long: flags=0x0 [ 5902.364520] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5902.371516] pending: lu_objects_flush [obdclass] [ 5902.379735] workqueue events_unbound: flags=0x2 [ 5902.387893] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5902.392509] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5902.411984] workqueue events_power_efficient: flags=0x80 [ 5902.418240] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5902.423977] pending: neigh_periodic_work [ 5902.426606] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5902.430733] pending: do_cache_clean, check_lifetime [ 5902.433210] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5902.436087] pending: neigh_periodic_work [ 5902.437781] workqueue writeback: flags=0x4a [ 5902.439962] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5902.443556] pending: wb_workfn, wb_workfn [ 5902.445698] workqueue xprtiod: flags=0xa [ 5902.448936] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5902.452816] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 5902.458900] workqueue ipv6_addrconf: flags=0x40008 [ 5902.461488] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5902.464702] pending: addrconf_verify_work [ 5902.467390] workqueue knbd0-recv: flags=0x1a [ 5902.469582] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5902.473264] in-flight: 657:recv_work [ 5902.475198] workqueue ptlrpc_pinger: flags=0x2 [ 5902.477463] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5902.480136] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 5902.482891] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3943s workers=4 idle: 20831 12642 28859 [ 5902.485130] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5892s workers=2 idle: 659 [ 5908.060039] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [ 5908.064267] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5908.092824] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5908.098086] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5908.102582] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5908.105814] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5908.126661] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5908.133987] RAX: 0000000000001485 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 5908.141094] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 5908.144563] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5908.151510] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 5908.158238] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 5908.165208] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 5908.172870] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5908.176540] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 5908.180451] Call Trace: [ 5908.181757] [ 5908.182754] ? show_regs.cold.9+0x22/0x2f [ 5908.184740] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5908.188078] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5908.189737] ? watchdog+0x40/0x40 [ 5908.191509] ? hrtimer_interrupt+0x128/0x2c0 [ 5908.193471] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5908.195482] ? apic_timer_interrupt+0xf/0x20 [ 5908.197393] [ 5908.198580] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5908.201008] do_raw_spin_lock+0x177/0x190 [ 5908.202842] _raw_spin_lock+0x21/0x30 [ 5908.204496] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5908.207223] ? do_raw_spin_unlock+0x75/0x190 [ 5908.209090] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5908.211538] ? do_wait_intr+0xf0/0xf0 [ 5908.213202] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5908.215789] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5908.218126] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5908.220826] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5908.223482] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5908.226148] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5908.228228] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5908.230236] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5908.232760] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5908.235135] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5908.237235] lmv_intent_open+0x338/0x1150 [lmv] [ 5908.239249] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5908.243576] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5908.246085] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5908.248556] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5908.250841] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5908.253797] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5908.256174] ? do_raw_spin_unlock+0x75/0x190 [ 5908.258307] ? _raw_spin_unlock+0x12/0x30 [ 5908.260432] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5908.263164] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5908.265239] ll_atomic_open+0x36a/0xd70 [lustre] [ 5908.267703] lookup_open+0x356/0xb40 [ 5908.269503] do_last+0x313/0xfc0 [ 5908.271038] ? nd_jump_root+0xe5/0x160 [ 5908.272568] ? path_init+0x437/0x520 [ 5908.274392] path_openat+0xf7/0x500 [ 5908.276095] ? do_raw_spin_unlock+0x75/0x190 [ 5908.278487] do_filp_open+0x99/0x140 [ 5908.280020] ? getname_flags+0x6e/0x330 [ 5908.281719] ? __check_object_size+0xff/0x256 [ 5908.283528] ? do_raw_spin_unlock+0x75/0x190 [ 5908.285255] ? _raw_spin_unlock+0x12/0x30 [ 5908.290679] do_sys_openat2+0x2b4/0x410 [ 5908.293545] do_sys_open+0x73/0xa0 [ 5908.294666] __x64_sys_openat+0x24/0x30 [ 5908.295962] do_syscall_64+0xc1/0x440 [ 5908.298072] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5908.300291] RIP: 0033:0x7ff54af45332 [ 5908.301951] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5908.310778] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5908.314276] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 5908.320852] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 5908.325312] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 5908.331183] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 5908.340158] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 5912.058057] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [ 5912.063099] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5912.112761] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5912.120772] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5912.125805] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5912.132635] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5912.156575] RSP: 0018:ffffaf9fc1543ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5912.165441] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 5912.171102] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 5912.177298] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5912.182862] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 5912.186990] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 5912.190492] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 5912.194633] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5912.197315] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 5912.200375] Call Trace: [ 5912.201225] [ 5912.202067] ? show_regs.cold.9+0x22/0x2f [ 5912.203798] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5912.206198] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5912.208547] ? watchdog+0x40/0x40 [ 5912.209999] ? hrtimer_interrupt+0x128/0x2c0 [ 5912.211841] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5912.214103] ? apic_timer_interrupt+0xf/0x20 [ 5912.215715] [ 5912.216682] ? native_safe_halt+0x2e/0x50 [ 5912.218641] kvm_wait+0xcd/0xe0 [ 5912.219874] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5912.221732] do_raw_spin_lock+0x177/0x190 [ 5912.223366] _raw_spin_lock+0x21/0x30 [ 5912.224812] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5912.227445] ? do_raw_spin_unlock+0x75/0x190 [ 5912.229163] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5912.231500] ? do_wait_intr+0xf0/0xf0 [ 5912.233352] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5912.235565] mdc_close+0x53b/0xd70 [mdc] [ 5912.237345] lmv_close+0x18d/0x490 [lmv] [ 5912.238720] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 5912.240742] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5912.244102] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5912.247797] ll_md_close+0x274/0x800 [lustre] [ 5912.249985] ll_file_release+0x128/0xc50 [lustre] [ 5912.252604] __fput+0x10d/0x440 [ 5912.254517] ____fput+0x12/0x20 [ 5912.258709] task_work_run+0xc6/0x110 [ 5912.263991] exit_to_usermode_loop+0x1dd/0x1f0 [ 5912.268619] do_syscall_64+0x3d6/0x440 [ 5912.274046] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5912.278599] RIP: 0033:0x7ff54af45cd8 [ 5912.285438] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5912.315627] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5912.328222] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 5912.339713] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 5912.347377] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 5912.360751] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 5912.370523] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5920.059020] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 5920.061413] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5920.081177] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5920.087247] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5920.091808] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5920.096464] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5920.099542] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5920.109078] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5920.112436] RAX: 0000000000005994 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 5920.117124] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 5920.121283] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 5920.126679] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 5920.130694] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 5920.135574] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 5920.140525] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5920.146102] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 5920.149328] Call Trace: [ 5920.150535] [ 5920.151973] ? show_regs.cold.9+0x22/0x2f [ 5920.153905] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5920.156939] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5920.159365] ? watchdog+0x40/0x40 [ 5920.161419] ? hrtimer_interrupt+0x128/0x2c0 [ 5920.163618] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5920.165972] ? apic_timer_interrupt+0xf/0x20 [ 5920.168480] [ 5920.169513] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5920.176559] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5920.180625] do_raw_spin_lock+0x177/0x190 [ 5920.184093] _raw_spin_lock+0x21/0x30 [ 5920.186533] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5920.194561] process_one_work+0x2c8/0x700 [ 5920.197954] worker_thread+0x296/0x6e0 [ 5920.199188] ? rescuer_thread+0x570/0x570 [ 5920.203261] kthread+0x1d1/0x200 [ 5920.204691] ? set_kthread_struct+0x70/0x70 [ 5920.207184] ret_from_fork+0x1f/0x30 [ 5928.057672] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 5928.061618] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5928.085123] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5928.088540] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5928.091477] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5928.093443] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5928.099505] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5928.101900] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 5928.104128] RDX: 0000000000007ddd RSI: 0000000000000003 RDI: ffff96848811cae0 [ 5928.106258] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 5928.108561] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 5928.110859] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5928.113133] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 5928.115418] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5928.117195] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 5928.119437] Call Trace: [ 5928.120168] [ 5928.120715] ? show_regs.cold.9+0x22/0x2f [ 5928.122040] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5928.123477] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5928.125045] ? watchdog+0x40/0x40 [ 5928.126061] ? hrtimer_interrupt+0x128/0x2c0 [ 5928.127639] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5928.129240] ? apic_timer_interrupt+0xf/0x20 [ 5928.130885] [ 5928.131599] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5928.133516] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5928.135332] do_raw_spin_lock+0x177/0x190 [ 5928.136767] _raw_spin_lock+0x21/0x30 [ 5928.138083] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5928.140112] obd_mod_free+0x46/0x140 [ptlrpc] [ 5928.142018] mdc_commit_open+0x91/0xb0 [mdc] [ 5928.144165] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5928.147119] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5928.149401] after_reply+0xe2c/0x1780 [ptlrpc] [ 5928.151320] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5928.153085] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5928.154575] ? do_wait_intr+0xf0/0xf0 [ 5928.155871] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5928.157893] kthread+0x1d1/0x200 [ 5928.159134] ? set_kthread_struct+0x70/0x70 [ 5928.160549] ret_from_fork+0x1f/0x30 [ 5932.511050] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3973s! [ 5932.514368] Showing busy workqueues and worker pools: [ 5932.516233] workqueue events: flags=0x0 [ 5932.517533] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5932.520817] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 5932.527511] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5932.532520] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 5932.541347] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5932.545779] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 5932.550347] workqueue events_long: flags=0x0 [ 5932.554415] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5932.559269] pending: lu_objects_flush [obdclass] [ 5932.562920] workqueue events_unbound: flags=0x2 [ 5932.567417] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5932.572825] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5932.581268] workqueue events_power_efficient: flags=0x80 [ 5932.585082] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5932.590920] pending: neigh_periodic_work [ 5932.593913] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5932.598757] pending: do_cache_clean, check_lifetime [ 5932.602333] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5932.605914] pending: neigh_periodic_work [ 5932.609156] workqueue writeback: flags=0x4a [ 5932.611996] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5932.616098] pending: wb_workfn, wb_workfn [ 5932.619356] workqueue xprtiod: flags=0xa [ 5932.622062] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5932.626851] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 5932.631842] workqueue ipv6_addrconf: flags=0x40008 [ 5932.635257] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5932.639273] pending: addrconf_verify_work [ 5932.642678] workqueue knbd0-recv: flags=0x1a [ 5932.645684] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5932.649787] in-flight: 657:recv_work [ 5932.651915] workqueue ptlrpc_pinger: flags=0x2 [ 5932.654268] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5932.658267] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 5932.661970] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3973s workers=4 idle: 20831 12642 28859 [ 5932.666788] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5922s workers=2 idle: 659 [ 5936.063137] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 5936.071585] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5936.114910] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5936.129025] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5936.140563] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5936.147586] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5936.177228] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5936.184981] RAX: 000000000000130e RBX: ffff96848811cae0 RCX: 0000000000000001 [ 5936.188694] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 5936.194422] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5936.200475] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 5936.207137] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 5936.210871] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 5936.218206] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5936.223865] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 5936.230871] Call Trace: [ 5936.232177] [ 5936.236994] ? show_regs.cold.9+0x22/0x2f [ 5936.238921] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5936.244266] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5936.246969] ? watchdog+0x40/0x40 [ 5936.253696] ? hrtimer_interrupt+0x128/0x2c0 [ 5936.255437] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5936.260215] ? apic_timer_interrupt+0xf/0x20 [ 5936.262023] [ 5936.266061] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5936.268554] do_raw_spin_lock+0x177/0x190 [ 5936.273239] _raw_spin_lock+0x21/0x30 [ 5936.275070] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5936.281059] ? do_raw_spin_unlock+0x75/0x190 [ 5936.286613] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5936.288734] ? do_wait_intr+0xf0/0xf0 [ 5936.293432] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5936.298380] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5936.302078] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5936.307291] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5936.313995] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5936.319247] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5936.328736] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5936.330274] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5936.336390] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5936.343101] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5936.347242] lmv_intent_open+0x338/0x1150 [lmv] [ 5936.355946] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5936.364813] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5936.366769] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5936.376816] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5936.378961] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5936.384249] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5936.396231] ? do_raw_spin_unlock+0x75/0x190 [ 5936.397905] ? _raw_spin_unlock+0x12/0x30 [ 5936.399901] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5936.402139] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5936.409100] ll_atomic_open+0x36a/0xd70 [lustre] [ 5936.411402] lookup_open+0x356/0xb40 [ 5936.412807] do_last+0x313/0xfc0 [ 5936.413994] ? nd_jump_root+0xe5/0x160 [ 5936.415441] ? path_init+0x437/0x520 [ 5936.425429] path_openat+0xf7/0x500 [ 5936.427099] ? do_raw_spin_unlock+0x75/0x190 [ 5936.430919] do_filp_open+0x99/0x140 [ 5936.432212] ? getname_flags+0x6e/0x330 [ 5936.439285] ? __check_object_size+0xff/0x256 [ 5936.442976] ? do_raw_spin_unlock+0x75/0x190 [ 5936.447285] ? _raw_spin_unlock+0x12/0x30 [ 5936.453748] do_sys_openat2+0x2b4/0x410 [ 5936.456857] do_sys_open+0x73/0xa0 [ 5936.459822] __x64_sys_openat+0x24/0x30 [ 5936.467736] do_syscall_64+0xc1/0x440 [ 5936.469138] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5936.477791] RIP: 0033:0x7ff54af45332 [ 5936.483178] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5936.511943] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5936.530316] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 5936.543357] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 5936.555275] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 5936.571321] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 5936.582069] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 5940.060440] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [ 5940.068373] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5940.128885] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5940.140978] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5940.149557] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5940.162583] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5940.185684] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5940.193402] RAX: 00000000000072e9 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 5940.210800] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 5940.219700] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5940.226588] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 5940.235081] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 5940.241783] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 5940.246021] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5940.248905] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 5940.256104] Call Trace: [ 5940.259528] [ 5940.260911] ? show_regs.cold.9+0x22/0x2f [ 5940.262883] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5940.271618] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5940.276930] ? watchdog+0x40/0x40 [ 5940.282020] ? hrtimer_interrupt+0x128/0x2c0 [ 5940.285165] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5940.288286] ? apic_timer_interrupt+0xf/0x20 [ 5940.292660] [ 5940.294582] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5940.304533] do_raw_spin_lock+0x177/0x190 [ 5940.308931] _raw_spin_lock+0x21/0x30 [ 5940.314013] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5940.319429] ? do_raw_spin_unlock+0x75/0x190 [ 5940.324862] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5940.332252] ? do_wait_intr+0xf0/0xf0 [ 5940.334135] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5940.339700] mdc_close+0x53b/0xd70 [mdc] [ 5940.343337] lmv_close+0x18d/0x490 [lmv] [ 5940.353173] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 5940.356461] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5940.368871] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5940.374030] ll_md_close+0x274/0x800 [lustre] [ 5940.377594] ll_file_release+0x128/0xc50 [lustre] [ 5940.380963] __fput+0x10d/0x440 [ 5940.383799] ____fput+0x12/0x20 [ 5940.387026] task_work_run+0xc6/0x110 [ 5940.391211] exit_to_usermode_loop+0x1dd/0x1f0 [ 5940.399156] do_syscall_64+0x3d6/0x440 [ 5940.403017] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5940.408494] RIP: 0033:0x7ff54af45cd8 [ 5940.413612] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5940.430636] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5940.434529] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 5940.442945] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 5940.446589] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 5940.449915] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 5940.452539] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5948.065358] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 5948.078423] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5948.158800] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5948.183265] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5948.197846] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5948.204789] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5948.221341] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5948.250338] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5948.259317] RAX: 0000000000006a85 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 5948.277725] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 5948.289114] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 5948.307519] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 5948.320141] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 5948.331202] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 5948.342469] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5948.361016] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 5948.363334] Call Trace: [ 5948.379515] [ 5948.382192] ? show_regs.cold.9+0x22/0x2f [ 5948.387012] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5948.394643] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5948.408031] ? watchdog+0x40/0x40 [ 5948.409171] ? hrtimer_interrupt+0x128/0x2c0 [ 5948.417384] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5948.428299] ? apic_timer_interrupt+0xf/0x20 [ 5948.429649] [ 5948.430361] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5948.444715] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5948.453859] do_raw_spin_lock+0x177/0x190 [ 5948.459326] _raw_spin_lock+0x21/0x30 [ 5948.460573] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5948.467475] process_one_work+0x2c8/0x700 [ 5948.471295] worker_thread+0x296/0x6e0 [ 5948.473278] ? rescuer_thread+0x570/0x570 [ 5948.475312] kthread+0x1d1/0x200 [ 5948.477067] ? set_kthread_struct+0x70/0x70 [ 5948.481294] ret_from_fork+0x1f/0x30 [ 5960.056787] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 5960.068134] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5960.097988] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5960.105513] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5960.111750] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5960.115747] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5960.127922] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5960.133021] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 5960.137772] RDX: 0000000000004e35 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 5960.142301] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 5960.147098] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 5960.151474] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5960.158215] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 5960.161231] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5960.166319] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 5960.178947] Call Trace: [ 5960.183345] [ 5960.190759] ? show_regs.cold.9+0x22/0x2f [ 5960.192268] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5960.196194] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5960.198073] ? watchdog+0x40/0x40 [ 5960.200877] ? hrtimer_interrupt+0x128/0x2c0 [ 5960.203989] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5960.212951] ? apic_timer_interrupt+0xf/0x20 [ 5960.214598] [ 5960.218355] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5960.224604] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5960.232759] do_raw_spin_lock+0x177/0x190 [ 5960.239919] _raw_spin_lock+0x21/0x30 [ 5960.243709] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5960.249026] obd_mod_free+0x46/0x140 [ptlrpc] [ 5960.261178] mdc_commit_open+0x91/0xb0 [mdc] [ 5960.270913] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5960.278704] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5960.281235] after_reply+0xe2c/0x1780 [ptlrpc] [ 5960.284448] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5960.292104] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5960.299345] ? do_wait_intr+0xf0/0xf0 [ 5960.302296] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5960.305166] kthread+0x1d1/0x200 [ 5960.314051] ? set_kthread_struct+0x70/0x70 [ 5960.316690] ret_from_fork+0x1f/0x30 [ 5962.720207] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4003s! [ 5962.726712] Showing busy workqueues and worker pools: [ 5962.728457] workqueue events: flags=0x0 [ 5962.730290] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5962.733911] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 5962.738668] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5962.742527] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 5962.749321] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5962.752447] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 5962.756031] workqueue events_long: flags=0x0 [ 5962.758098] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5962.760759] pending: lu_objects_flush [obdclass] [ 5962.762774] workqueue events_unbound: flags=0x2 [ 5962.764172] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5962.766165] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5962.770410] workqueue events_power_efficient: flags=0x80 [ 5962.772865] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5962.775166] pending: neigh_periodic_work [ 5962.776705] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5962.783909] pending: do_cache_clean, check_lifetime [ 5962.786511] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5962.788814] pending: neigh_periodic_work [ 5962.791103] workqueue writeback: flags=0x4a [ 5962.792400] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5962.794508] pending: wb_workfn, wb_workfn [ 5962.796752] workqueue xprtiod: flags=0xa [ 5962.798102] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5962.803595] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 5962.808270] workqueue ipv6_addrconf: flags=0x40008 [ 5962.813211] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5962.818175] pending: addrconf_verify_work [ 5962.820883] workqueue knbd0-recv: flags=0x1a [ 5962.823687] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5962.829505] in-flight: 657:recv_work [ 5962.832345] workqueue ptlrpc_pinger: flags=0x2 [ 5962.835630] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5962.842487] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 5962.844645] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4003s workers=4 idle: 20831 12642 28859 [ 5962.850130] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5953s workers=2 idle: 659 [ 5964.061997] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 5964.069222] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5964.105145] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5964.109524] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5964.114417] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5964.117611] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5964.127812] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5964.131488] RAX: 0000000000004604 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 5964.136616] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 5964.144128] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5964.148855] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 5964.153259] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 5964.157867] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 5964.162276] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5964.165636] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 5964.169245] Call Trace: [ 5964.170613] [ 5964.171903] ? show_regs.cold.9+0x22/0x2f [ 5964.174187] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5964.177518] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5964.184780] ? watchdog+0x40/0x40 [ 5964.186278] ? hrtimer_interrupt+0x128/0x2c0 [ 5964.188161] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5964.190645] ? apic_timer_interrupt+0xf/0x20 [ 5964.192317] [ 5964.193066] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5964.197594] do_raw_spin_lock+0x177/0x190 [ 5964.200486] _raw_spin_lock+0x21/0x30 [ 5964.201933] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5964.204538] ? do_raw_spin_unlock+0x75/0x190 [ 5964.207623] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5964.210495] ? do_wait_intr+0xf0/0xf0 [ 5964.212064] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5964.215126] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5964.222930] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5964.229328] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5964.232141] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5964.239703] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5964.245358] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5964.247772] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5964.254181] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5964.260009] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5964.263646] lmv_intent_open+0x338/0x1150 [lmv] [ 5964.269671] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5964.272594] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5964.279954] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5964.281418] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5964.295537] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5964.304201] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5964.311367] ? do_raw_spin_unlock+0x75/0x190 [ 5964.312871] ? _raw_spin_unlock+0x12/0x30 [ 5964.316927] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5964.320705] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5964.324083] ll_atomic_open+0x36a/0xd70 [lustre] [ 5964.332109] lookup_open+0x356/0xb40 [ 5964.335076] do_last+0x313/0xfc0 [ 5964.336167] ? nd_jump_root+0xe5/0x160 [ 5964.339154] ? path_init+0x437/0x520 [ 5964.340406] path_openat+0xf7/0x500 [ 5964.341563] ? do_raw_spin_unlock+0x75/0x190 [ 5964.351827] do_filp_open+0x99/0x140 [ 5964.352980] ? getname_flags+0x6e/0x330 [ 5964.358314] ? __check_object_size+0xff/0x256 [ 5964.360530] ? do_raw_spin_unlock+0x75/0x190 [ 5964.363367] ? _raw_spin_unlock+0x12/0x30 [ 5964.366767] do_sys_openat2+0x2b4/0x410 [ 5964.371560] do_sys_open+0x73/0xa0 [ 5964.373099] __x64_sys_openat+0x24/0x30 [ 5964.376316] do_syscall_64+0xc1/0x440 [ 5964.379067] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5964.389006] RIP: 0033:0x7ff54af45332 [ 5964.390227] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5964.403963] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5964.413248] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 5964.424947] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 5964.435302] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 5964.445775] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 5964.454985] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 5968.058435] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [ 5968.061932] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5968.109402] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5968.122420] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5968.129284] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5968.133437] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5968.139352] RSP: 0018:ffffaf9fc1543ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5968.142803] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 5968.145489] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 5968.148648] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5968.151393] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 5968.154946] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 5968.158681] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 5968.161644] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5968.163356] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 5968.166797] Call Trace: [ 5968.168527] [ 5968.169463] ? show_regs.cold.9+0x22/0x2f [ 5968.173960] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5968.177072] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5968.179207] ? watchdog+0x40/0x40 [ 5968.180308] ? hrtimer_interrupt+0x128/0x2c0 [ 5968.182247] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5968.183895] ? apic_timer_interrupt+0xf/0x20 [ 5968.185275] [ 5968.186024] ? native_safe_halt+0x2e/0x50 [ 5968.187611] kvm_wait+0xcd/0xe0 [ 5968.188896] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5968.190630] do_raw_spin_lock+0x177/0x190 [ 5968.192266] _raw_spin_lock+0x21/0x30 [ 5968.193482] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5968.195720] ? do_raw_spin_unlock+0x75/0x190 [ 5968.196779] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5968.198231] ? do_wait_intr+0xf0/0xf0 [ 5968.199283] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5968.201683] mdc_close+0x53b/0xd70 [mdc] [ 5968.203033] lmv_close+0x18d/0x490 [lmv] [ 5968.204216] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 5968.206305] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5968.208596] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5968.211350] ll_md_close+0x274/0x800 [lustre] [ 5968.214092] ll_file_release+0x128/0xc50 [lustre] [ 5968.216753] __fput+0x10d/0x440 [ 5968.217791] ____fput+0x12/0x20 [ 5968.219412] task_work_run+0xc6/0x110 [ 5968.221219] exit_to_usermode_loop+0x1dd/0x1f0 [ 5968.223102] do_syscall_64+0x3d6/0x440 [ 5968.225025] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5968.228815] RIP: 0033:0x7ff54af45cd8 [ 5968.230708] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5968.246273] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5968.250880] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 5968.257074] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 5968.259462] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 5968.262255] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 5968.265739] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5973.957531] rcu: INFO: rcu_sched self-detected stall on CPU [ 5973.964792] rcu: 0-...!: (2089037 ticks this GP) idle=98a/1/0x4000000000000002 softirq=125154/125154 fqs=959 [ 5973.976152] (t=4020117 jiffies g=253585 q=4129) [ 5973.983853] rcu: rcu_sched kthread starved for 4015077 jiffies! g253585 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [ 5973.999156] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 5974.012413] rcu: RCU grace-period kthread stack dump: [ 5974.017994] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 5974.031016] Call Trace: [ 5974.033142] __schedule+0x351/0xcb0 [ 5974.037916] schedule+0xc0/0x180 [ 5974.039771] schedule_timeout+0xb4/0x190 [ 5974.042280] ? do_raw_spin_unlock+0x75/0x190 [ 5974.047827] ? __next_timer_interrupt+0x160/0x160 [ 5974.055916] ? prepare_to_swait_event+0xf3/0x140 [ 5974.061663] rcu_gp_kthread+0x8dd/0x15b0 [ 5974.073212] ? force_qs_rnp+0x2c0/0x2c0 [ 5974.079797] kthread+0x1d1/0x200 [ 5974.081882] ? set_kthread_struct+0x70/0x70 [ 5974.086779] ret_from_fork+0x1f/0x30 [ 5974.089730] rcu: Stack dump where RCU GP kthread last ran: [ 5974.097991] Sending NMI from CPU 0 to CPUs 2: [ 5974.104978] NMI backtrace for cpu 2 skipped: idling at native_safe_halt+0x2e/0x50 [ 5974.107287] NMI backtrace for cpu 0 [ 5974.113776] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5974.125922] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5974.144897] Call Trace: [ 5974.150130] [ 5974.153337] ? dump_stack+0xbb/0x10e [ 5974.159792] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 5974.168476] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 5974.172902] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 5974.178086] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 5974.181215] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 5974.188232] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 5974.195270] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 5974.206857] ? update_process_times+0x6d/0xc0 [ 5974.209956] ? tick_sched_handle.isra.18+0x27/0x90 [ 5974.212806] ? tick_sched_timer+0x53/0xb0 [ 5974.215836] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5974.218629] ? tick_sched_handle.isra.18+0x90/0x90 [ 5974.222990] ? hrtimer_interrupt+0x128/0x2c0 [ 5974.227516] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5974.235776] ? apic_timer_interrupt+0xf/0x20 [ 5974.237969] [ 5974.239139] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5974.242948] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5974.245995] ? do_raw_spin_lock+0x177/0x190 [ 5974.249236] ? _raw_spin_lock+0x21/0x30 [ 5974.251241] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5974.254711] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 5974.256464] ? mdc_commit_open+0x91/0xb0 [mdc] [ 5974.271725] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5974.284173] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5974.299806] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 5974.306675] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5974.316909] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5974.325556] ? do_wait_intr+0xf0/0xf0 [ 5974.335919] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5974.345101] ? kthread+0x1d1/0x200 [ 5974.349546] ? set_kthread_struct+0x70/0x70 [ 5974.356230] ? ret_from_fork+0x1f/0x30 [ 5992.065752] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 5992.077885] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5992.132542] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5992.141946] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5992.150254] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5992.158992] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5992.189008] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5992.206117] RAX: 0000000000002be7 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 5992.214874] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 5992.227346] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5992.229678] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 5992.240718] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 5992.246726] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 5992.254112] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5992.257450] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 5992.273508] Call Trace: [ 5992.274411] [ 5992.275115] ? show_regs.cold.9+0x22/0x2f [ 5992.290923] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5992.297449] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5992.306069] ? watchdog+0x40/0x40 [ 5992.314966] ? hrtimer_interrupt+0x128/0x2c0 [ 5992.324446] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5992.326102] ? apic_timer_interrupt+0xf/0x20 [ 5992.333442] [ 5992.335101] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5992.340710] do_raw_spin_lock+0x177/0x190 [ 5992.345189] _raw_spin_lock+0x21/0x30 [ 5992.347274] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5992.350934] ? do_raw_spin_unlock+0x75/0x190 [ 5992.353516] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5992.357150] ? do_wait_intr+0xf0/0xf0 [ 5992.359319] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5992.362564] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5992.365320] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5992.368100] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5992.371099] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5992.373802] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5992.375958] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5992.379054] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5992.380918] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5992.397933] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5992.404814] lmv_intent_open+0x338/0x1150 [lmv] [ 5992.410082] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5992.417294] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5992.429682] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5992.433558] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5992.436853] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5992.441102] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5992.444921] ? do_raw_spin_unlock+0x75/0x190 [ 5992.449650] ? _raw_spin_unlock+0x12/0x30 [ 5992.451046] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5992.454060] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5992.456995] ll_atomic_open+0x36a/0xd70 [lustre] [ 5992.462872] lookup_open+0x356/0xb40 [ 5992.464930] do_last+0x313/0xfc0 [ 5992.466387] ? nd_jump_root+0xe5/0x160 [ 5992.469523] ? path_init+0x437/0x520 [ 5992.473038] path_openat+0xf7/0x500 [ 5992.477183] ? do_raw_spin_unlock+0x75/0x190 [ 5992.481093] do_filp_open+0x99/0x140 [ 5992.482831] ? getname_flags+0x6e/0x330 [ 5992.484966] ? __check_object_size+0xff/0x256 [ 5992.489966] ? do_raw_spin_unlock+0x75/0x190 [ 5992.492504] ? _raw_spin_unlock+0x12/0x30 [ 5992.494789] do_sys_openat2+0x2b4/0x410 [ 5992.496540] do_sys_open+0x73/0xa0 [ 5992.501466] __x64_sys_openat+0x24/0x30 [ 5992.503483] do_syscall_64+0xc1/0x440 [ 5992.507439] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5992.511289] RIP: 0033:0x7ff54af45332 [ 5992.514079] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5992.533112] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5992.540223] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 5992.543958] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 5992.547899] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 5992.559222] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 5992.566136] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 5992.927071] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4034s! [ 5992.935885] Showing busy workqueues and worker pools: [ 5992.943511] workqueue events: flags=0x0 [ 5992.948527] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5992.953703] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 5992.966645] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5992.972895] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 5992.983650] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5992.990958] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 5992.996086] workqueue events_long: flags=0x0 [ 5992.998079] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5993.006129] pending: lu_objects_flush [obdclass] [ 5993.011805] workqueue events_unbound: flags=0x2 [ 5993.014729] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5993.018739] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5993.024357] workqueue events_power_efficient: flags=0x80 [ 5993.028087] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5993.031739] pending: neigh_periodic_work [ 5993.033741] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5993.036290] pending: do_cache_clean, check_lifetime [ 5993.038532] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5993.043971] pending: neigh_periodic_work [ 5993.045521] workqueue writeback: flags=0x4a [ 5993.048867] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5993.054918] pending: wb_workfn, wb_workfn [ 5993.057533] workqueue xprtiod: flags=0xa [ 5993.059083] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5993.062516] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 5993.066851] workqueue ipv6_addrconf: flags=0x40008 [ 5993.073401] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5993.078613] pending: addrconf_verify_work [ 5993.080721] workqueue knbd0-recv: flags=0x1a [ 5993.084039] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5993.089267] in-flight: 657:recv_work [ 5993.090660] workqueue ptlrpc_pinger: flags=0x2 [ 5993.092128] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5993.110052] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 5993.118317] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4034s workers=4 idle: 20831 12642 28859 [ 5993.129334] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5983s workers=2 idle: 659 [ 5996.058033] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [ 5996.063173] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5996.093943] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5996.100461] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5996.104996] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5996.108252] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5996.119441] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5996.123355] RAX: 0000000000007419 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 5996.127073] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 5996.131566] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5996.135422] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 5996.140011] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 5996.147557] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 5996.152844] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5996.155957] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 5996.161231] Call Trace: [ 5996.167618] [ 5996.169312] ? show_regs.cold.9+0x22/0x2f [ 5996.172877] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5996.177354] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5996.180405] ? watchdog+0x40/0x40 [ 5996.183376] ? hrtimer_interrupt+0x128/0x2c0 [ 5996.187211] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5996.192419] ? apic_timer_interrupt+0xf/0x20 [ 5996.198065] [ 5996.199510] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5996.204928] do_raw_spin_lock+0x177/0x190 [ 5996.208559] _raw_spin_lock+0x21/0x30 [ 5996.209785] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5996.215308] ? do_raw_spin_unlock+0x75/0x190 [ 5996.218253] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5996.221367] ? do_wait_intr+0xf0/0xf0 [ 5996.223491] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5996.228496] mdc_close+0x53b/0xd70 [mdc] [ 5996.231075] lmv_close+0x18d/0x490 [lmv] [ 5996.233341] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 5996.237837] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5996.242271] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5996.247461] ll_md_close+0x274/0x800 [lustre] [ 5996.252519] ll_file_release+0x128/0xc50 [lustre] [ 5996.259084] __fput+0x10d/0x440 [ 5996.261793] ____fput+0x12/0x20 [ 5996.264437] task_work_run+0xc6/0x110 [ 5996.267534] exit_to_usermode_loop+0x1dd/0x1f0 [ 5996.272119] do_syscall_64+0x3d6/0x440 [ 5996.274162] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5996.281336] RIP: 0033:0x7ff54af45cd8 [ 5996.284064] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5996.304898] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5996.311229] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 5996.318304] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 5996.324766] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 5996.326663] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 5996.329583] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6000.059108] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 6000.064818] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6000.128442] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6000.156298] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6000.182366] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6000.195878] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6000.207268] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6000.244978] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6000.256872] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 6000.267842] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 6000.279182] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 6000.290574] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 6000.295966] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 6000.305272] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 6000.315864] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6000.320044] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 6000.331993] Call Trace: [ 6000.339862] [ 6000.340566] ? show_regs.cold.9+0x22/0x2f [ 6000.342602] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6000.349553] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6000.359024] ? watchdog+0x40/0x40 [ 6000.365095] ? hrtimer_interrupt+0x128/0x2c0 [ 6000.368560] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6000.371423] ? apic_timer_interrupt+0xf/0x20 [ 6000.374025] [ 6000.375066] ? native_safe_halt+0x2e/0x50 [ 6000.376778] kvm_wait+0xcd/0xe0 [ 6000.378251] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6000.380332] do_raw_spin_lock+0x177/0x190 [ 6000.381654] _raw_spin_lock+0x21/0x30 [ 6000.382650] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6000.384498] process_one_work+0x2c8/0x700 [ 6000.385729] worker_thread+0x296/0x6e0 [ 6000.386755] ? rescuer_thread+0x570/0x570 [ 6000.393982] kthread+0x1d1/0x200 [ 6000.398151] ? set_kthread_struct+0x70/0x70 [ 6000.400675] ret_from_fork+0x1f/0x30 [ 6020.057600] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 6020.062554] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 6020.071560] Modules linked in: lustre(O) [ 6020.076724] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6020.076802] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6020.076810] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6020.076973] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6020.076988] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6020.076992] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6020.076997] RAX: 0000000000007409 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 6020.076999] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 6020.077002] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6020.077004] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 6020.077006] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 6020.077009] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 6020.077012] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6020.077014] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 6020.077022] Call Trace: [ 6020.077027] [ 6020.077099] ? show_regs.cold.9+0x22/0x2f [ 6020.077111] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6020.077118] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6020.077122] ? watchdog+0x40/0x40 [ 6020.077126] ? hrtimer_interrupt+0x128/0x2c0 [ 6020.077130] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6020.077140] ? apic_timer_interrupt+0xf/0x20 [ 6020.077144] [ 6020.077147] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6020.077154] do_raw_spin_lock+0x177/0x190 [ 6020.077159] _raw_spin_lock+0x21/0x30 [ 6020.077164] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6020.077566] ? do_raw_spin_unlock+0x75/0x190 [ 6020.077573] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6020.077773] ? do_wait_intr+0xf0/0xf0 [ 6020.077782] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6020.078189] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6020.078554] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6020.078784] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6020.079020] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6020.079126] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6020.079154] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6020.079181] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6020.079327] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6020.079619] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6020.079650] lmv_intent_open+0x338/0x1150 [lmv] [ 6020.079681] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6020.079776] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6020.080122] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6020.080146] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6020.080534] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6020.080977] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6020.081313] ? do_raw_spin_unlock+0x75/0x190 [ 6020.081322] ? _raw_spin_unlock+0x12/0x30 [ 6020.081328] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6020.081357] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6020.081375] ll_atomic_open+0x36a/0xd70 [lustre] [ 6020.081454] lookup_open+0x356/0xb40 [ 6020.081531] do_last+0x313/0xfc0 [ 6020.081539] ? nd_jump_root+0xe5/0x160 [ 6020.081543] ? path_init+0x437/0x520 [ 6020.081547] path_openat+0xf7/0x500 [ 6020.081552] ? do_raw_spin_unlock+0x75/0x190 [ 6020.081556] do_filp_open+0x99/0x140 [ 6020.081562] ? getname_flags+0x6e/0x330 [ 6020.081566] ? __check_object_size+0xff/0x256 [ 6020.081573] ? do_raw_spin_unlock+0x75/0x190 [ 6020.081576] ? _raw_spin_unlock+0x12/0x30 [ 6020.081580] do_sys_openat2+0x2b4/0x410 [ 6020.081584] do_sys_open+0x73/0xa0 [ 6020.081588] __x64_sys_openat+0x24/0x30 [ 6020.081592] do_syscall_64+0xc1/0x440 [ 6020.081599] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6020.081604] RIP: 0033:0x7ff54af45332 [ 6020.081749] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6020.081755] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6020.081761] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 6020.081763] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 6020.081766] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 6020.081768] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 6020.081770] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 6020.537782] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6020.575566] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6020.586843] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6020.593469] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6020.597665] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6020.606800] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6020.610310] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 6020.613366] RDX: 0000000000007f20 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 6020.616205] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 6020.618115] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 6020.620388] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6020.623157] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 6020.626264] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6020.628524] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 6020.631693] Call Trace: [ 6020.632780] [ 6020.633588] ? show_regs.cold.9+0x22/0x2f [ 6020.635317] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6020.638157] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6020.645844] ? watchdog+0x40/0x40 [ 6020.647118] ? hrtimer_interrupt+0x128/0x2c0 [ 6020.656773] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6020.658468] ? apic_timer_interrupt+0xf/0x20 [ 6020.662748] [ 6020.663562] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6020.666701] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6020.671738] do_raw_spin_lock+0x177/0x190 [ 6020.673953] _raw_spin_lock+0x21/0x30 [ 6020.680530] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6020.687592] obd_mod_free+0x46/0x140 [ptlrpc] [ 6020.697688] mdc_commit_open+0x91/0xb0 [mdc] [ 6020.701687] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6020.708428] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6020.711692] after_reply+0xe2c/0x1780 [ptlrpc] [ 6020.718001] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6020.726577] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6020.729187] ? do_wait_intr+0xf0/0xf0 [ 6020.732480] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6020.738619] kthread+0x1d1/0x200 [ 6020.741449] ? set_kthread_struct+0x70/0x70 [ 6020.744683] ret_from_fork+0x1f/0x30 [ 6023.647043] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4064s! [ 6023.649983] Showing busy workqueues and worker pools: [ 6023.653685] workqueue events: flags=0x0 [ 6023.660525] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6023.673597] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 6023.686370] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6023.691396] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 6023.706469] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6023.713318] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 6023.718603] workqueue events_long: flags=0x0 [ 6023.720979] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6023.728137] pending: lu_objects_flush [obdclass] [ 6023.735333] workqueue events_unbound: flags=0x2 [ 6023.738416] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6023.744683] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6023.753304] workqueue events_power_efficient: flags=0x80 [ 6023.759071] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6023.763905] pending: neigh_periodic_work [ 6023.765767] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6023.770686] pending: do_cache_clean, check_lifetime [ 6023.777027] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6023.784487] pending: neigh_periodic_work [ 6023.789110] workqueue writeback: flags=0x4a [ 6023.792214] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6023.797555] pending: wb_workfn, wb_workfn [ 6023.800870] workqueue xprtiod: flags=0xa [ 6023.805476] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6023.808766] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 6023.815530] workqueue ipv6_addrconf: flags=0x40008 [ 6023.820939] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6023.827329] pending: addrconf_verify_work [ 6023.829119] workqueue knbd0-recv: flags=0x1a [ 6023.834301] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6023.838813] in-flight: 657:recv_work [ 6023.842726] workqueue ptlrpc_pinger: flags=0x2 [ 6023.848409] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6023.853626] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 6023.859784] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4064s workers=4 idle: 20831 12642 28859 [ 6023.867678] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6014s workers=2 idle: 659 [ 6024.063237] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [ 6024.076152] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6024.137764] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6024.148787] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6024.157541] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6024.163314] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6024.179092] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6024.186338] RAX: 0000000000006c08 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 6024.190063] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 6024.192675] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6024.199253] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 6024.215558] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 6024.224499] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 6024.238643] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6024.246083] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 6024.257670] Call Trace: [ 6024.261691] [ 6024.265521] ? show_regs.cold.9+0x22/0x2f [ 6024.272414] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6024.278658] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6024.285703] ? watchdog+0x40/0x40 [ 6024.292680] ? hrtimer_interrupt+0x128/0x2c0 [ 6024.297341] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6024.299297] ? apic_timer_interrupt+0xf/0x20 [ 6024.301653] [ 6024.302380] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6024.313383] do_raw_spin_lock+0x177/0x190 [ 6024.320016] _raw_spin_lock+0x21/0x30 [ 6024.321079] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6024.326048] ? do_raw_spin_unlock+0x75/0x190 [ 6024.327480] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6024.332716] ? do_wait_intr+0xf0/0xf0 [ 6024.334778] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6024.336772] mdc_close+0x53b/0xd70 [mdc] [ 6024.337869] lmv_close+0x18d/0x490 [lmv] [ 6024.340427] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 6024.347527] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6024.351464] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6024.356792] ll_md_close+0x274/0x800 [lustre] [ 6024.362153] ll_file_release+0x128/0xc50 [lustre] [ 6024.364842] __fput+0x10d/0x440 [ 6024.366387] ____fput+0x12/0x20 [ 6024.368988] task_work_run+0xc6/0x110 [ 6024.371788] exit_to_usermode_loop+0x1dd/0x1f0 [ 6024.377421] do_syscall_64+0x3d6/0x440 [ 6024.379111] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6024.380381] RIP: 0033:0x7ff54af45cd8 [ 6024.382514] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6024.393058] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6024.395798] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 6024.398950] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 6024.409419] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 6024.422054] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 6024.425564] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6028.062010] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 6028.065533] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6028.127108] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6028.143805] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6028.156151] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6028.165632] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6028.172440] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6028.199750] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6028.215368] RAX: 0000000000004026 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 6028.223576] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 6028.241935] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 6028.257646] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 6028.272019] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 6028.280575] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 6028.289477] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6028.292418] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 6028.294924] Call Trace: [ 6028.305387] [ 6028.309694] ? show_regs.cold.9+0x22/0x2f [ 6028.317609] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6028.323401] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6028.328653] ? watchdog+0x40/0x40 [ 6028.330475] ? hrtimer_interrupt+0x128/0x2c0 [ 6028.338936] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6028.340771] ? apic_timer_interrupt+0xf/0x20 [ 6028.346110] [ 6028.347130] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6028.353554] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6028.361581] do_raw_spin_lock+0x177/0x190 [ 6028.367192] _raw_spin_lock+0x21/0x30 [ 6028.372123] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6028.380565] process_one_work+0x2c8/0x700 [ 6028.384253] worker_thread+0x296/0x6e0 [ 6028.388630] ? rescuer_thread+0x570/0x570 [ 6028.391165] kthread+0x1d1/0x200 [ 6028.393328] ? set_kthread_struct+0x70/0x70 [ 6028.395323] ret_from_fork+0x1f/0x30 [ 6048.057626] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 6048.060302] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 6048.068593] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6048.068811] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6048.068819] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6048.068822] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6048.068837] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6048.068840] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6048.068845] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 6048.068848] RDX: 00000000000011b1 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 6048.068850] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 6048.068852] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 6048.068854] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6048.068857] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 6048.068860] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6048.068862] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 6048.068869] Call Trace: [ 6048.068875] [ 6048.068886] ? show_regs.cold.9+0x22/0x2f [ 6048.068896] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6048.068903] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6048.068907] ? watchdog+0x40/0x40 [ 6048.068912] ? hrtimer_interrupt+0x128/0x2c0 [ 6048.068915] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6048.068924] ? apic_timer_interrupt+0xf/0x20 [ 6048.068929] [ 6048.068933] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6048.068938] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6048.068943] do_raw_spin_lock+0x177/0x190 [ 6048.068948] _raw_spin_lock+0x21/0x30 [ 6048.068954] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6048.070348] obd_mod_free+0x46/0x140 [ptlrpc] [ 6048.070818] mdc_commit_open+0x91/0xb0 [mdc] [ 6048.070862] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6048.074155] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6048.075158] after_reply+0xe2c/0x1780 [ptlrpc] [ 6048.075388] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6048.076245] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6048.076464] ? do_wait_intr+0xf0/0xf0 [ 6048.076476] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6048.076874] kthread+0x1d1/0x200 [ 6048.076881] ? set_kthread_struct+0x70/0x70 [ 6048.076885] ret_from_fork+0x1f/0x30 [ 6048.330866] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6048.357132] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6048.363985] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6048.369198] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6048.372922] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6048.395162] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6048.397564] RAX: 00000000000001ad RBX: ffff96848811cae0 RCX: 0000000000000001 [ 6048.409776] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 6048.415397] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6048.420339] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 6048.432799] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 6048.438208] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 6048.445836] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6048.451827] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 6048.459153] Call Trace: [ 6048.463954] [ 6048.465633] ? show_regs.cold.9+0x22/0x2f [ 6048.468634] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6048.471177] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6048.472745] ? watchdog+0x40/0x40 [ 6048.478823] ? hrtimer_interrupt+0x128/0x2c0 [ 6048.483209] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6048.487215] ? apic_timer_interrupt+0xf/0x20 [ 6048.492187] [ 6048.493355] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6048.500198] do_raw_spin_lock+0x177/0x190 [ 6048.502815] _raw_spin_lock+0x21/0x30 [ 6048.508070] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6048.514422] ? do_raw_spin_unlock+0x75/0x190 [ 6048.520360] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6048.523419] ? do_wait_intr+0xf0/0xf0 [ 6048.525439] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6048.529144] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6048.532691] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6048.536083] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6048.538794] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6048.541967] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6048.544640] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6048.547374] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6048.550397] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6048.556191] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6048.559989] lmv_intent_open+0x338/0x1150 [lmv] [ 6048.562024] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6048.564444] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6048.568137] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6048.571018] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6048.574090] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6048.578827] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6048.581401] ? do_raw_spin_unlock+0x75/0x190 [ 6048.585889] ? _raw_spin_unlock+0x12/0x30 [ 6048.587772] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6048.589591] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6048.592092] ll_atomic_open+0x36a/0xd70 [lustre] [ 6048.594433] lookup_open+0x356/0xb40 [ 6048.596609] do_last+0x313/0xfc0 [ 6048.598903] ? nd_jump_root+0xe5/0x160 [ 6048.600544] ? path_init+0x437/0x520 [ 6048.602096] path_openat+0xf7/0x500 [ 6048.603598] ? do_raw_spin_unlock+0x75/0x190 [ 6048.605334] do_filp_open+0x99/0x140 [ 6048.607153] ? getname_flags+0x6e/0x330 [ 6048.608881] ? __check_object_size+0xff/0x256 [ 6048.610720] ? do_raw_spin_unlock+0x75/0x190 [ 6048.612442] ? _raw_spin_unlock+0x12/0x30 [ 6048.613941] do_sys_openat2+0x2b4/0x410 [ 6048.615623] do_sys_open+0x73/0xa0 [ 6048.623405] __x64_sys_openat+0x24/0x30 [ 6048.625029] do_syscall_64+0xc1/0x440 [ 6048.626592] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6048.628833] RIP: 0033:0x7ff54af45332 [ 6048.630320] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6048.640031] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6048.643324] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 6048.646155] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 6048.651211] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 6048.657559] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 6048.670353] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 6052.067368] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 6052.073039] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6052.109830] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6052.123446] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6052.129113] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6052.131178] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6052.143076] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6052.149303] RAX: 000000000000601c RBX: ffff96848811cae0 RCX: 0000000000000001 [ 6052.154103] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 6052.162463] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6052.168295] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 6052.178676] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 6052.195766] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 6052.198271] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6052.206944] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 6052.224665] Call Trace: [ 6052.227534] [ 6052.229075] ? show_regs.cold.9+0x22/0x2f [ 6052.233421] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6052.236799] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6052.239878] ? watchdog+0x40/0x40 [ 6052.241656] ? hrtimer_interrupt+0x128/0x2c0 [ 6052.244125] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6052.246610] ? apic_timer_interrupt+0xf/0x20 [ 6052.248678] [ 6052.249912] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6052.255297] do_raw_spin_lock+0x177/0x190 [ 6052.256579] _raw_spin_lock+0x21/0x30 [ 6052.260815] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6052.264947] ? do_raw_spin_unlock+0x75/0x190 [ 6052.266798] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6052.269046] ? do_wait_intr+0xf0/0xf0 [ 6052.270858] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6052.273839] mdc_close+0x53b/0xd70 [mdc] [ 6052.281633] lmv_close+0x18d/0x490 [lmv] [ 6052.283622] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 6052.290056] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6052.293694] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6052.296129] ll_md_close+0x274/0x800 [lustre] [ 6052.298824] ll_file_release+0x128/0xc50 [lustre] [ 6052.301027] __fput+0x10d/0x440 [ 6052.302382] ____fput+0x12/0x20 [ 6052.303708] task_work_run+0xc6/0x110 [ 6052.305424] exit_to_usermode_loop+0x1dd/0x1f0 [ 6052.309324] do_syscall_64+0x3d6/0x440 [ 6052.312270] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6052.315992] RIP: 0033:0x7ff54af45cd8 [ 6052.318080] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6052.344081] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6052.353838] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 6052.360080] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 6052.367952] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 6052.373253] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 6052.379885] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6054.379089] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4095s! [ 6054.403901] Showing busy workqueues and worker pools: [ 6054.410400] workqueue events: flags=0x0 [ 6054.416308] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6054.428890] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 6054.444881] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6054.460788] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 6054.472248] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6054.488548] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 6054.500833] workqueue events_long: flags=0x0 [ 6054.509783] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6054.519310] pending: lu_objects_flush [obdclass] [ 6054.525900] workqueue events_unbound: flags=0x2 [ 6054.534590] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6054.541982] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6054.559074] workqueue events_power_efficient: flags=0x80 [ 6054.569032] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6054.579420] pending: neigh_periodic_work [ 6054.589860] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6054.603025] pending: do_cache_clean, check_lifetime [ 6054.611479] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6054.623972] pending: neigh_periodic_work [ 6054.626920] workqueue writeback: flags=0x4a [ 6054.637979] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6054.651493] pending: wb_workfn, wb_workfn [ 6054.658047] workqueue xprtiod: flags=0xa [ 6054.667376] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6054.684862] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 6054.699762] workqueue ipv6_addrconf: flags=0x40008 [ 6054.703309] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6054.710452] pending: addrconf_verify_work [ 6054.726508] workqueue knbd0-recv: flags=0x1a [ 6054.732416] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6054.747946] in-flight: 657:recv_work [ 6054.752389] workqueue ptlrpc_pinger: flags=0x2 [ 6054.757821] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6054.764087] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 6054.771194] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4095s workers=4 idle: 20831 12642 28859 [ 6054.776691] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6044s workers=2 idle: 659 [ 6056.066572] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 6056.074333] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6056.112198] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6056.118286] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6056.123036] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6056.128537] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6056.138293] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6056.148187] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6056.152351] RAX: 0000000000004594 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 6056.158480] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 6056.161537] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 6056.168807] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 6056.175656] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 6056.179953] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 6056.184304] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6056.190887] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 6056.195251] Call Trace: [ 6056.196321] [ 6056.197761] ? show_regs.cold.9+0x22/0x2f [ 6056.200316] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6056.202280] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6056.204189] ? watchdog+0x40/0x40 [ 6056.217853] ? hrtimer_interrupt+0x128/0x2c0 [ 6056.219018] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6056.220222] ? apic_timer_interrupt+0xf/0x20 [ 6056.228346] [ 6056.229248] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6056.231973] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6056.234055] do_raw_spin_lock+0x177/0x190 [ 6056.236049] _raw_spin_lock+0x21/0x30 [ 6056.238050] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6056.240351] process_one_work+0x2c8/0x700 [ 6056.245154] worker_thread+0x296/0x6e0 [ 6056.246207] ? rescuer_thread+0x570/0x570 [ 6056.247209] kthread+0x1d1/0x200 [ 6056.248380] ? set_kthread_struct+0x70/0x70 [ 6056.256584] ret_from_fork+0x1f/0x30 [ 6076.062878] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 6076.079354] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6076.162082] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6076.176756] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6076.184325] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6076.190016] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6076.206115] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6076.215044] RAX: 0000000000007f86 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 6076.225644] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 6076.240140] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6076.251595] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 6076.262800] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 6076.273235] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 6076.279184] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6076.285083] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 6076.293524] Call Trace: [ 6076.297859] [ 6076.299111] ? show_regs.cold.9+0x22/0x2f [ 6076.301535] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6076.310639] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6076.313189] ? watchdog+0x40/0x40 [ 6076.318087] ? hrtimer_interrupt+0x128/0x2c0 [ 6076.325736] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6076.331857] ? apic_timer_interrupt+0xf/0x20 [ 6076.336502] [ 6076.337274] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6076.345981] do_raw_spin_lock+0x177/0x190 [ 6076.348051] _raw_spin_lock+0x21/0x30 [ 6076.351040] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6076.357968] ? do_raw_spin_unlock+0x75/0x190 [ 6076.361035] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6076.368588] ? do_wait_intr+0xf0/0xf0 [ 6076.369779] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6076.373019] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6076.379674] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6076.386558] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6076.391229] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6076.393116] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6076.400184] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6076.408209] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6076.412186] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6076.423674] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6076.430145] lmv_intent_open+0x338/0x1150 [lmv] [ 6076.446605] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6076.461080] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6076.462876] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6076.464416] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6076.471649] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6076.477600] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6076.481060] ? do_raw_spin_unlock+0x75/0x190 [ 6076.484368] ? _raw_spin_unlock+0x12/0x30 [ 6076.487672] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6076.493611] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6076.497783] ll_atomic_open+0x36a/0xd70 [lustre] [ 6076.501130] lookup_open+0x356/0xb40 [ 6076.503062] do_last+0x313/0xfc0 [ 6076.504671] ? nd_jump_root+0xe5/0x160 [ 6076.508032] ? path_init+0x437/0x520 [ 6076.510522] path_openat+0xf7/0x500 [ 6076.512469] ? do_raw_spin_unlock+0x75/0x190 [ 6076.514645] do_filp_open+0x99/0x140 [ 6076.519548] ? getname_flags+0x6e/0x330 [ 6076.522590] ? __check_object_size+0xff/0x256 [ 6076.525060] ? do_raw_spin_unlock+0x75/0x190 [ 6076.526966] ? _raw_spin_unlock+0x12/0x30 [ 6076.528516] do_sys_openat2+0x2b4/0x410 [ 6076.530258] do_sys_open+0x73/0xa0 [ 6076.532225] __x64_sys_openat+0x24/0x30 [ 6076.534294] do_syscall_64+0xc1/0x440 [ 6076.536710] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6076.540506] RIP: 0033:0x7ff54af45332 [ 6076.542658] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6076.559652] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6076.565216] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 6076.570864] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 6076.582111] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 6076.591094] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 6076.595216] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 6080.055028] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 6080.057711] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6080.062845] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 6080.072142] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6080.072158] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6080.072161] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6080.072178] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6080.072182] RSP: 0000:ffffaf9fc1b97bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6080.072189] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 6080.072192] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 6080.072195] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 6080.072197] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 6080.072200] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6080.072203] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 6080.072206] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6080.072209] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 6080.072218] Call Trace: [ 6080.072225] [ 6080.072231] ? show_regs.cold.9+0x22/0x2f [ 6080.072243] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6080.072251] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6080.072256] ? watchdog+0x40/0x40 [ 6080.072261] ? hrtimer_interrupt+0x128/0x2c0 [ 6080.072266] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6080.072274] ? apic_timer_interrupt+0xf/0x20 [ 6080.072353] [ 6080.072356] ? native_safe_halt+0x2e/0x50 [ 6080.072361] kvm_wait+0xcd/0xe0 [ 6080.072369] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6080.072378] do_raw_spin_lock+0x177/0x190 [ 6080.072383] _raw_spin_lock+0x21/0x30 [ 6080.072390] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6080.072732] obd_mod_free+0x46/0x140 [ptlrpc] [ 6080.072934] mdc_commit_open+0x91/0xb0 [mdc] [ 6080.072976] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6080.073166] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6080.073438] after_reply+0xe2c/0x1780 [ptlrpc] [ 6080.083085] Modules linked in: [ 6080.086972] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6080.087370] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6080.087602] ? do_wait_intr+0xf0/0xf0 [ 6080.087615] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6080.087846] kthread+0x1d1/0x200 [ 6080.087853] ? set_kthread_struct+0x70/0x70 [ 6080.087857] ret_from_fork+0x1f/0x30 [ 6080.166616] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6080.180944] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6080.184251] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6080.187571] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6080.190205] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6080.197564] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6080.199845] RAX: 0000000000006c87 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 6080.202080] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 6080.205032] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6080.207976] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 6080.210803] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 6080.213625] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 6080.216946] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6080.221088] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 6080.224254] Call Trace: [ 6080.228112] [ 6080.229400] ? show_regs.cold.9+0x22/0x2f [ 6080.230954] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6080.232800] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6080.234248] ? watchdog+0x40/0x40 [ 6080.235680] ? hrtimer_interrupt+0x128/0x2c0 [ 6080.237243] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6080.239707] ? apic_timer_interrupt+0xf/0x20 [ 6080.241855] [ 6080.245908] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6080.248723] do_raw_spin_lock+0x177/0x190 [ 6080.252828] _raw_spin_lock+0x21/0x30 [ 6080.254499] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6080.259741] ? do_raw_spin_unlock+0x75/0x190 [ 6080.263581] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6080.267686] ? do_wait_intr+0xf0/0xf0 [ 6080.271213] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6080.276483] mdc_close+0x53b/0xd70 [mdc] [ 6080.279020] lmv_close+0x18d/0x490 [lmv] [ 6080.285246] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 6080.288027] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6080.292084] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6080.294570] ll_md_close+0x274/0x800 [lustre] [ 6080.296688] ll_file_release+0x128/0xc50 [lustre] [ 6080.298593] __fput+0x10d/0x440 [ 6080.299807] ____fput+0x12/0x20 [ 6080.301122] task_work_run+0xc6/0x110 [ 6080.302855] exit_to_usermode_loop+0x1dd/0x1f0 [ 6080.304986] do_syscall_64+0x3d6/0x440 [ 6080.306917] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6080.309173] RIP: 0033:0x7ff54af45cd8 [ 6080.310523] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6080.319012] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6080.322710] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 6080.327512] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 6080.331549] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 6080.335143] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 6080.337996] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6084.061525] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 6084.063931] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6084.098035] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6084.105480] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6084.108362] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6084.116024] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6084.123701] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6084.156589] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6084.172773] RAX: 000000000000630c RBX: ffff96848811cae0 RCX: 0000000000000001 [ 6084.180862] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 6084.192065] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 6084.200126] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 6084.205574] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 6084.215753] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 6084.233205] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6084.237328] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 6084.242737] Call Trace: [ 6084.244359] [ 6084.245057] ? show_regs.cold.9+0x22/0x2f [ 6084.248530] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6084.257708] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6084.260097] ? watchdog+0x40/0x40 [ 6084.265585] ? hrtimer_interrupt+0x128/0x2c0 [ 6084.272795] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6084.284399] ? apic_timer_interrupt+0xf/0x20 [ 6084.287784] [ 6084.291266] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6084.295499] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6084.305757] do_raw_spin_lock+0x177/0x190 [ 6084.310462] _raw_spin_lock+0x21/0x30 [ 6084.313385] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6084.316618] process_one_work+0x2c8/0x700 [ 6084.318578] worker_thread+0x296/0x6e0 [ 6084.320550] ? rescuer_thread+0x570/0x570 [ 6084.322515] kthread+0x1d1/0x200 [ 6084.324318] ? set_kthread_struct+0x70/0x70 [ 6084.326158] ret_from_fork+0x1f/0x30 [ 6085.089602] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4126s! [ 6085.094050] Showing busy workqueues and worker pools: [ 6085.101605] workqueue events: flags=0x0 [ 6085.107119] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6085.115536] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 6085.123954] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6085.132373] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 6085.142655] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6085.147178] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 6085.150696] workqueue events_long: flags=0x0 [ 6085.153187] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6085.156697] pending: lu_objects_flush [obdclass] [ 6085.163519] workqueue events_unbound: flags=0x2 [ 6085.166358] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6085.169691] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6085.175133] workqueue events_power_efficient: flags=0x80 [ 6085.178336] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6085.181511] pending: neigh_periodic_work [ 6085.183562] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6085.186938] pending: do_cache_clean, check_lifetime [ 6085.193608] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6085.199690] pending: neigh_periodic_work [ 6085.204121] workqueue writeback: flags=0x4a [ 6085.208760] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6085.215149] pending: wb_workfn, wb_workfn [ 6085.219886] workqueue xprtiod: flags=0xa [ 6085.222665] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6085.229941] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 6085.244641] workqueue ipv6_addrconf: flags=0x40008 [ 6085.250208] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6085.256735] pending: addrconf_verify_work [ 6085.260116] workqueue knbd0-recv: flags=0x1a [ 6085.262316] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6085.269118] in-flight: 657:recv_work [ 6085.271657] workqueue ptlrpc_pinger: flags=0x2 [ 6085.274782] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6085.281445] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 6085.285864] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4126s workers=4 idle: 20831 12642 28859 [ 6085.290317] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6075s workers=2 idle: 659 [ 6104.060022] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 6104.063520] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6104.105386] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6104.126569] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6104.140444] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6104.157946] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6104.185019] RSP: 0000:ffffaf9fc15234f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6104.191022] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 6104.194580] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 6104.197491] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6104.200575] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 6104.206599] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 6104.212364] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 6104.215671] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6104.217728] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 6104.224045] Call Trace: [ 6104.228826] [ 6104.229803] ? show_regs.cold.9+0x22/0x2f [ 6104.231927] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6104.233694] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6104.235874] ? watchdog+0x40/0x40 [ 6104.237132] ? hrtimer_interrupt+0x128/0x2c0 [ 6104.239398] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6104.241926] ? apic_timer_interrupt+0xf/0x20 [ 6104.244853] [ 6104.245498] ? native_safe_halt+0x2e/0x50 [ 6104.247836] kvm_wait+0xcd/0xe0 [ 6104.249110] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6104.251269] do_raw_spin_lock+0x177/0x190 [ 6104.252784] _raw_spin_lock+0x21/0x30 [ 6104.254175] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6104.256406] ? do_raw_spin_unlock+0x75/0x190 [ 6104.258053] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6104.260268] ? do_wait_intr+0xf0/0xf0 [ 6104.261776] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6104.264112] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6104.266170] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6104.268681] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6104.270973] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6104.273174] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6104.276805] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6104.280272] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6104.289284] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6104.291519] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6104.297237] lmv_intent_open+0x338/0x1150 [lmv] [ 6104.299569] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6104.302075] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6104.308358] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6104.310886] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6104.313338] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6104.315451] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6104.319594] ? do_raw_spin_unlock+0x75/0x190 [ 6104.323878] ? _raw_spin_unlock+0x12/0x30 [ 6104.327361] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6104.330434] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6104.332991] ll_atomic_open+0x36a/0xd70 [lustre] [ 6104.335600] lookup_open+0x356/0xb40 [ 6104.337630] do_last+0x313/0xfc0 [ 6104.339692] ? nd_jump_root+0xe5/0x160 [ 6104.345163] ? path_init+0x437/0x520 [ 6104.353970] path_openat+0xf7/0x500 [ 6104.358996] ? do_raw_spin_unlock+0x75/0x190 [ 6104.361840] do_filp_open+0x99/0x140 [ 6104.363931] ? getname_flags+0x6e/0x330 [ 6104.369489] ? __check_object_size+0xff/0x256 [ 6104.370990] ? do_raw_spin_unlock+0x75/0x190 [ 6104.374732] ? _raw_spin_unlock+0x12/0x30 [ 6104.378925] do_sys_openat2+0x2b4/0x410 [ 6104.380227] do_sys_open+0x73/0xa0 [ 6104.381180] __x64_sys_openat+0x24/0x30 [ 6104.388018] do_syscall_64+0xc1/0x440 [ 6104.390501] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6104.392414] RIP: 0033:0x7ff54af45332 [ 6104.393791] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6104.400904] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6104.404099] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 6104.406232] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 6104.408907] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 6104.411659] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 6104.413831] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 6108.064525] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 6108.070719] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6108.130831] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6108.136141] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6108.140254] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6108.142662] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6108.150991] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6108.154121] RAX: 0000000000007e92 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 6108.157153] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 6108.160395] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6108.163657] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 6108.166438] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 6108.169293] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 6108.172693] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6108.175110] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 6108.178207] Call Trace: [ 6108.179294] [ 6108.180196] ? show_regs.cold.9+0x22/0x2f [ 6108.181895] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6108.183957] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6108.185967] ? watchdog+0x40/0x40 [ 6108.187230] ? hrtimer_interrupt+0x128/0x2c0 [ 6108.188956] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6108.190722] ? apic_timer_interrupt+0xf/0x20 [ 6108.191929] [ 6108.192797] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6108.195073] do_raw_spin_lock+0x177/0x190 [ 6108.196753] _raw_spin_lock+0x21/0x30 [ 6108.198328] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6108.200884] ? do_raw_spin_unlock+0x75/0x190 [ 6108.202627] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6108.204952] ? do_wait_intr+0xf0/0xf0 [ 6108.206484] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6108.208841] mdc_close+0x53b/0xd70 [mdc] [ 6108.210876] lmv_close+0x18d/0x490 [lmv] [ 6108.212464] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 6108.215355] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6108.217877] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6108.219887] ll_md_close+0x274/0x800 [lustre] [ 6108.222195] ll_file_release+0x128/0xc50 [lustre] [ 6108.224377] __fput+0x10d/0x440 [ 6108.227745] ____fput+0x12/0x20 [ 6108.230540] task_work_run+0xc6/0x110 [ 6108.232526] exit_to_usermode_loop+0x1dd/0x1f0 [ 6108.235544] do_syscall_64+0x3d6/0x440 [ 6108.236963] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6108.240190] RIP: 0033:0x7ff54af45cd8 [ 6108.242287] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6108.256562] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6108.262510] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 6108.265227] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 6108.267144] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 6108.268871] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 6108.271679] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6112.055025] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [ 6112.058430] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6112.066456] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 6112.082480] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6112.082499] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6112.082503] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6112.082519] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6112.082523] RSP: 0000:ffffaf9fc1b97bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6112.082530] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 6112.082533] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 6112.082536] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 6112.082538] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 6112.082540] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6112.082545] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 6112.082548] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6112.082550] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 6112.082560] Call Trace: [ 6112.082568] [ 6112.082575] ? show_regs.cold.9+0x22/0x2f [ 6112.082590] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6112.082597] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6112.082603] ? watchdog+0x40/0x40 [ 6112.082608] ? hrtimer_interrupt+0x128/0x2c0 [ 6112.082612] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6112.082620] ? apic_timer_interrupt+0xf/0x20 [ 6112.082625] [ 6112.082628] ? native_safe_halt+0x2e/0x50 [ 6112.082632] kvm_wait+0xcd/0xe0 [ 6112.082642] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6112.082650] do_raw_spin_lock+0x177/0x190 [ 6112.082654] _raw_spin_lock+0x21/0x30 [ 6112.082662] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6112.083281] obd_mod_free+0x46/0x140 [ptlrpc] [ 6112.083496] mdc_commit_open+0x91/0xb0 [mdc] [ 6112.083560] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6112.083749] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6112.083961] after_reply+0xe2c/0x1780 [ptlrpc] [ 6112.084852] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6112.085389] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6112.085598] ? do_wait_intr+0xf0/0xf0 [ 6112.085613] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6112.085805] kthread+0x1d1/0x200 [ 6112.085814] ? set_kthread_struct+0x70/0x70 [ 6112.085818] ret_from_fork+0x1f/0x30 [ 6112.206778] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6112.236774] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6112.245769] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6112.250677] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6112.254330] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6112.257886] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6112.270517] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6112.272763] RAX: 0000000000007985 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 6112.276364] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 6112.279828] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 6112.283271] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 6112.287313] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 6112.290712] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 6112.295637] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6112.299512] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 6112.303398] Call Trace: [ 6112.306397] [ 6112.307333] ? show_regs.cold.9+0x22/0x2f [ 6112.309267] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6112.312606] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6112.314830] ? watchdog+0x40/0x40 [ 6112.316097] ? hrtimer_interrupt+0x128/0x2c0 [ 6112.320356] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6112.324821] ? apic_timer_interrupt+0xf/0x20 [ 6112.326948] [ 6112.327825] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6112.330601] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6112.335705] do_raw_spin_lock+0x177/0x190 [ 6112.337284] _raw_spin_lock+0x21/0x30 [ 6112.338943] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6112.342285] process_one_work+0x2c8/0x700 [ 6112.344922] worker_thread+0x296/0x6e0 [ 6112.346582] ? rescuer_thread+0x570/0x570 [ 6112.348844] kthread+0x1d1/0x200 [ 6112.350214] ? set_kthread_struct+0x70/0x70 [ 6112.351893] ret_from_fork+0x1f/0x30 [ 6115.295126] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4156s! [ 6115.300465] Showing busy workqueues and worker pools: [ 6115.303384] workqueue events: flags=0x0 [ 6115.304587] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6115.308472] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 6115.314203] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6115.318140] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 6115.326152] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6115.330290] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 6115.333688] workqueue events_long: flags=0x0 [ 6115.336606] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6115.341042] pending: lu_objects_flush [obdclass] [ 6115.344697] workqueue events_unbound: flags=0x2 [ 6115.347610] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6115.350580] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6115.356439] workqueue events_power_efficient: flags=0x80 [ 6115.359495] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6115.363092] pending: neigh_periodic_work [ 6115.365595] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6115.369871] pending: do_cache_clean, check_lifetime [ 6115.373489] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6115.377674] pending: neigh_periodic_work [ 6115.380319] workqueue writeback: flags=0x4a [ 6115.382810] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6115.386972] pending: wb_workfn, wb_workfn [ 6115.388522] workqueue xprtiod: flags=0xa [ 6115.389822] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6115.393554] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 6115.396162] workqueue ipv6_addrconf: flags=0x40008 [ 6115.397512] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6115.399719] pending: addrconf_verify_work [ 6115.404723] workqueue knbd0-recv: flags=0x1a [ 6115.407173] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6115.411484] in-flight: 657:recv_work [ 6115.414046] workqueue ptlrpc_pinger: flags=0x2 [ 6115.415671] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6115.419673] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 6115.424288] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4156s workers=4 idle: 20831 12642 28859 [ 6115.429605] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6105s workers=2 idle: 659 [ 6132.064933] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 6132.069728] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6132.143904] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6132.152501] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6132.159483] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6132.170726] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6132.193380] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6132.199566] RAX: 000000000000723c RBX: ffff96848811cae0 RCX: 0000000000000001 [ 6132.203418] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 6132.207343] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6132.214043] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 6132.216546] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 6132.222562] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 6132.229120] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6132.235553] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 6132.251544] Call Trace: [ 6132.252272] [ 6132.252923] ? show_regs.cold.9+0x22/0x2f [ 6132.258473] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6132.263841] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6132.267584] ? watchdog+0x40/0x40 [ 6132.274954] ? hrtimer_interrupt+0x128/0x2c0 [ 6132.277584] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6132.288153] ? apic_timer_interrupt+0xf/0x20 [ 6132.289869] [ 6132.290664] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6132.297603] do_raw_spin_lock+0x177/0x190 [ 6132.305595] _raw_spin_lock+0x21/0x30 [ 6132.309248] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6132.312324] ? do_raw_spin_unlock+0x75/0x190 [ 6132.316474] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6132.318763] ? do_wait_intr+0xf0/0xf0 [ 6132.320010] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6132.325386] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6132.327956] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6132.331646] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6132.339566] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6132.354099] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6132.360641] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6132.363688] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6132.369792] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6132.377111] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6132.382393] lmv_intent_open+0x338/0x1150 [lmv] [ 6132.387697] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6132.391181] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6132.394708] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6132.396189] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6132.397901] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6132.401457] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6132.403638] ? do_raw_spin_unlock+0x75/0x190 [ 6132.405417] ? _raw_spin_unlock+0x12/0x30 [ 6132.407356] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6132.410064] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6132.413051] ll_atomic_open+0x36a/0xd70 [lustre] [ 6132.415347] lookup_open+0x356/0xb40 [ 6132.427373] do_last+0x313/0xfc0 [ 6132.433135] ? nd_jump_root+0xe5/0x160 [ 6132.443646] ? path_init+0x437/0x520 [ 6132.446757] path_openat+0xf7/0x500 [ 6132.456714] ? do_raw_spin_unlock+0x75/0x190 [ 6132.462593] do_filp_open+0x99/0x140 [ 6132.468543] ? getname_flags+0x6e/0x330 [ 6132.469997] ? __check_object_size+0xff/0x256 [ 6132.493274] ? do_raw_spin_unlock+0x75/0x190 [ 6132.494805] ? _raw_spin_unlock+0x12/0x30 [ 6132.508398] do_sys_openat2+0x2b4/0x410 [ 6132.511739] do_sys_open+0x73/0xa0 [ 6132.512701] __x64_sys_openat+0x24/0x30 [ 6132.514760] do_syscall_64+0xc1/0x440 [ 6132.519746] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6132.541641] RIP: 0033:0x7ff54af45332 [ 6132.562949] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6132.601816] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6132.608317] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 6132.621516] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 6132.627926] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 6132.630355] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 6132.656101] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 6136.058140] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 6136.080227] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6136.199799] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6136.226981] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6136.253971] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6136.268600] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6136.304281] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6136.317190] RAX: 0000000000004985 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 6136.325539] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 6136.334963] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6136.343415] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 6136.349551] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 6136.355324] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 6136.361257] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6136.367216] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 6136.376204] Call Trace: [ 6136.378610] [ 6136.380814] ? show_regs.cold.9+0x22/0x2f [ 6136.383215] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6136.391642] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6136.394310] ? watchdog+0x40/0x40 [ 6136.399438] ? hrtimer_interrupt+0x128/0x2c0 [ 6136.404287] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6136.408830] ? apic_timer_interrupt+0xf/0x20 [ 6136.413069] [ 6136.414556] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6136.418834] do_raw_spin_lock+0x177/0x190 [ 6136.420589] _raw_spin_lock+0x21/0x30 [ 6136.423148] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6136.425700] ? do_raw_spin_unlock+0x75/0x190 [ 6136.432844] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6136.441117] ? do_wait_intr+0xf0/0xf0 [ 6136.443129] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6136.446594] mdc_close+0x53b/0xd70 [mdc] [ 6136.455601] lmv_close+0x18d/0x490 [lmv] [ 6136.460463] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 6136.470106] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6136.474476] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6136.480897] ll_md_close+0x274/0x800 [lustre] [ 6136.484211] ll_file_release+0x128/0xc50 [lustre] [ 6136.489392] __fput+0x10d/0x440 [ 6136.493459] ____fput+0x12/0x20 [ 6136.497142] task_work_run+0xc6/0x110 [ 6136.499109] exit_to_usermode_loop+0x1dd/0x1f0 [ 6136.501208] do_syscall_64+0x3d6/0x440 [ 6136.503204] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6136.505369] RIP: 0033:0x7ff54af45cd8 [ 6136.507218] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6136.515543] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6136.519090] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 6136.522514] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 6136.534748] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 6136.545679] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 6136.548979] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6140.056963] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 6140.060121] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 6140.069448] Modules linked in: lustre(O) [ 6140.073400] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6140.073476] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6140.073483] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6140.073490] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6140.074076] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6140.074087] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6140.074091] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6140.074096] RAX: 0000000000002db0 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 6140.074099] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 6140.074101] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 6140.074103] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 6140.074111] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 6140.074115] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 6140.074118] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6140.074120] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 6140.074128] Call Trace: [ 6140.074134] [ 6140.074138] ? show_regs.cold.9+0x22/0x2f [ 6140.074148] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6140.074154] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6140.074158] ? watchdog+0x40/0x40 [ 6140.074162] ? hrtimer_interrupt+0x128/0x2c0 [ 6140.074166] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6140.074173] ? apic_timer_interrupt+0xf/0x20 [ 6140.074178] [ 6140.074181] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6140.074186] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6140.074191] do_raw_spin_lock+0x177/0x190 [ 6140.074196] _raw_spin_lock+0x21/0x30 [ 6140.074201] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6140.074462] process_one_work+0x2c8/0x700 [ 6140.074470] worker_thread+0x296/0x6e0 [ 6140.074475] ? rescuer_thread+0x570/0x570 [ 6140.074480] kthread+0x1d1/0x200 [ 6140.074484] ? set_kthread_struct+0x70/0x70 [ 6140.074488] ret_from_fork+0x1f/0x30 [ 6140.662327] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6140.754817] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6140.781685] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6140.797406] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6140.807605] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6140.842480] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6140.860368] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 6140.874800] RDX: 00000000000016df RSI: 0000000000000003 RDI: ffff96848811cae0 [ 6140.890037] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 6140.901940] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 6140.919631] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6140.931920] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 6140.961925] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6140.976400] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 6140.989661] Call Trace: [ 6140.990697] [ 6141.010639] ? show_regs.cold.9+0x22/0x2f [ 6141.032572] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6141.044481] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6141.063982] ? watchdog+0x40/0x40 [ 6141.082488] ? hrtimer_interrupt+0x128/0x2c0 [ 6141.102785] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6141.104925] ? apic_timer_interrupt+0xf/0x20 [ 6141.127602] [ 6141.128527] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6141.148352] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6141.160439] do_raw_spin_lock+0x177/0x190 [ 6141.171100] _raw_spin_lock+0x21/0x30 [ 6141.185428] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6141.187559] obd_mod_free+0x46/0x140 [ptlrpc] [ 6141.200573] mdc_commit_open+0x91/0xb0 [mdc] [ 6141.213606] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6141.215757] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6141.233026] after_reply+0xe2c/0x1780 [ptlrpc] [ 6141.248555] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6141.256970] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6141.262505] ? do_wait_intr+0xf0/0xf0 [ 6141.266492] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6141.272907] kthread+0x1d1/0x200 [ 6141.276239] ? set_kthread_struct+0x70/0x70 [ 6141.280383] ret_from_fork+0x1f/0x30 [ 6145.517739] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4186s! [ 6145.529105] Showing busy workqueues and worker pools: [ 6145.532560] workqueue events: flags=0x0 [ 6145.535347] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6145.538570] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 6145.543948] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6145.556611] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 6145.571590] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6145.583341] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 6145.599256] workqueue events_long: flags=0x0 [ 6145.607376] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6145.615539] pending: lu_objects_flush [obdclass] [ 6145.621524] workqueue events_unbound: flags=0x2 [ 6145.629534] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6145.638002] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6145.649923] workqueue events_power_efficient: flags=0x80 [ 6145.656747] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6145.662769] pending: neigh_periodic_work [ 6145.667552] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6145.673489] pending: do_cache_clean, check_lifetime [ 6145.678393] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6145.684310] pending: neigh_periodic_work [ 6145.685729] workqueue writeback: flags=0x4a [ 6145.690491] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6145.697430] pending: wb_workfn, wb_workfn [ 6145.701984] workqueue xprtiod: flags=0xa [ 6145.704723] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6145.714137] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 6145.725240] workqueue ipv6_addrconf: flags=0x40008 [ 6145.740270] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6145.749553] pending: addrconf_verify_work [ 6145.758133] workqueue knbd0-recv: flags=0x1a [ 6145.761605] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6145.767458] in-flight: 657:recv_work [ 6145.772622] workqueue ptlrpc_pinger: flags=0x2 [ 6145.774212] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6145.778243] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 6145.788332] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4186s workers=4 idle: 20831 12642 28859 [ 6145.799444] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6136s workers=2 idle: 659 [ 6153.962179] rcu: INFO: rcu_sched self-detected stall on CPU [ 6153.968970] rcu: 0-...!: (2185181 ticks this GP) idle=98a/1/0x4000000000000002 softirq=125154/125154 fqs=959 [ 6153.985796] (t=4200122 jiffies g=253585 q=4215) [ 6153.995318] rcu: rcu_sched kthread starved for 4195082 jiffies! g253585 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [ 6154.012688] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 6154.028718] rcu: RCU grace-period kthread stack dump: [ 6154.032446] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 6154.039983] Call Trace: [ 6154.041254] __schedule+0x351/0xcb0 [ 6154.042878] schedule+0xc0/0x180 [ 6154.044269] schedule_timeout+0xb4/0x190 [ 6154.046687] ? do_raw_spin_unlock+0x75/0x190 [ 6154.048785] ? __next_timer_interrupt+0x160/0x160 [ 6154.050941] ? prepare_to_swait_event+0xf3/0x140 [ 6154.052999] rcu_gp_kthread+0x8dd/0x15b0 [ 6154.055306] ? force_qs_rnp+0x2c0/0x2c0 [ 6154.056879] kthread+0x1d1/0x200 [ 6154.058280] ? set_kthread_struct+0x70/0x70 [ 6154.059893] ret_from_fork+0x1f/0x30 [ 6154.061899] rcu: Stack dump where RCU GP kthread last ran: [ 6154.068497] Sending NMI from CPU 0 to CPUs 1: [ 6154.083334] NMI backtrace for cpu 1 [ 6154.083342] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6154.083344] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6154.083346] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6154.083348] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6154.083359] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 [ 6154.083362] RAX: 0000000000007f86 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 6154.083363] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 6154.083365] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6154.083366] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 6154.083367] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 6154.083368] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 6154.083370] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6154.083371] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 6154.083381] Call Trace: [ 6154.083382] [ 6154.083383] ? show_regs.cold.9+0x22/0x2f [ 6154.083385] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 6154.083386] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 6154.083387] ? nmi_handle+0xbd/0x210 [ 6154.083389] ? default_do_nmi+0x4b/0x1b0 [ 6154.083390] ? do_nmi+0x280/0x380 [ 6154.083390] ? end_repeat_nmi+0x16/0x1f [ 6154.083391] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6154.083392] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6154.083393] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6154.083394] [ 6154.083395] do_raw_spin_lock+0x177/0x190 [ 6154.083396] _raw_spin_lock+0x21/0x30 [ 6154.083397] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6154.083398] ? do_raw_spin_unlock+0x75/0x190 [ 6154.083400] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6154.083401] ? do_wait_intr+0xf0/0xf0 [ 6154.083402] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6154.083404] mdc_close+0x53b/0xd70 [mdc] [ 6154.083405] lmv_close+0x18d/0x490 [lmv] [ 6154.083407] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 6154.083408] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6154.083409] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6154.083411] ll_md_close+0x274/0x800 [lustre] [ 6154.083412] ll_file_release+0x128/0xc50 [lustre] [ 6154.083413] __fput+0x10d/0x440 [ 6154.083414] ____fput+0x12/0x20 [ 6154.083415] task_work_run+0xc6/0x110 [ 6154.083416] exit_to_usermode_loop+0x1dd/0x1f0 [ 6154.083417] do_syscall_64+0x3d6/0x440 [ 6154.083418] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6154.083419] RIP: 0033:0x7ff54af45cd8 [ 6154.083420] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6154.083422] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6154.083425] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 6154.083426] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 6154.083427] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 6154.083428] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 6154.083429] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6154.083862] NMI backtrace for cpu 0 [ 6154.333467] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6154.338032] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6154.341368] Call Trace: [ 6154.342292] [ 6154.343061] ? dump_stack+0xbb/0x10e [ 6154.344686] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 6154.346712] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 6154.348472] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 6154.350932] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 6154.354254] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 6154.357010] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 6154.358715] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 6154.362826] ? update_process_times+0x6d/0xc0 [ 6154.367078] ? tick_sched_handle.isra.18+0x27/0x90 [ 6154.373278] ? tick_sched_timer+0x53/0xb0 [ 6154.380539] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6154.382975] ? tick_sched_handle.isra.18+0x90/0x90 [ 6154.385591] ? hrtimer_interrupt+0x128/0x2c0 [ 6154.389386] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6154.395231] ? apic_timer_interrupt+0xf/0x20 [ 6154.397073] [ 6154.398474] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6154.401313] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6154.407502] ? do_raw_spin_lock+0x177/0x190 [ 6154.409879] ? _raw_spin_lock+0x21/0x30 [ 6154.411287] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6154.417381] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 6154.420461] ? mdc_commit_open+0x91/0xb0 [mdc] [ 6154.427291] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6154.433977] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6154.444005] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 6154.449547] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6154.452679] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6154.454509] ? do_wait_intr+0xf0/0xf0 [ 6154.458699] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6154.462458] ? kthread+0x1d1/0x200 [ 6154.465362] ? set_kthread_struct+0x70/0x70 [ 6154.468152] ? ret_from_fork+0x1f/0x30 [ 6160.060459] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [ 6160.069271] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6160.126245] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6160.136997] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6160.143819] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6160.146735] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6160.158178] RSP: 0000:ffffaf9fc15234f0 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 6160.163413] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 6160.169995] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 6160.173562] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6160.185721] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 6160.201212] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 6160.206379] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 6160.219488] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6160.223690] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 6160.232443] Call Trace: [ 6160.234913] [ 6160.237396] ? show_regs.cold.9+0x22/0x2f [ 6160.239221] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6160.241499] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6160.252094] ? watchdog+0x40/0x40 [ 6160.256134] ? hrtimer_interrupt+0x128/0x2c0 [ 6160.260404] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6160.267347] ? apic_timer_interrupt+0xf/0x20 [ 6160.269371] [ 6160.275766] ? native_safe_halt+0x2e/0x50 [ 6160.280285] kvm_wait+0xcd/0xe0 [ 6160.283411] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6160.287709] do_raw_spin_lock+0x177/0x190 [ 6160.291049] _raw_spin_lock+0x21/0x30 [ 6160.293346] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6160.301473] ? do_raw_spin_unlock+0x75/0x190 [ 6160.309229] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6160.316381] ? do_wait_intr+0xf0/0xf0 [ 6160.321694] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6160.327267] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6160.331517] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6160.344192] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6160.352172] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6160.361576] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6160.373330] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6160.379227] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6160.383236] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6160.389829] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6160.394289] lmv_intent_open+0x338/0x1150 [lmv] [ 6160.396657] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6160.403328] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6160.408114] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6160.411389] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6160.418158] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6160.426094] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6160.430141] ? do_raw_spin_unlock+0x75/0x190 [ 6160.435935] ? _raw_spin_unlock+0x12/0x30 [ 6160.439663] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6160.446322] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6160.453342] ll_atomic_open+0x36a/0xd70 [lustre] [ 6160.456827] lookup_open+0x356/0xb40 [ 6160.460266] do_last+0x313/0xfc0 [ 6160.462846] ? nd_jump_root+0xe5/0x160 [ 6160.465765] ? path_init+0x437/0x520 [ 6160.468445] path_openat+0xf7/0x500 [ 6160.471109] ? do_raw_spin_unlock+0x75/0x190 [ 6160.478116] do_filp_open+0x99/0x140 [ 6160.482473] ? getname_flags+0x6e/0x330 [ 6160.485339] ? __check_object_size+0xff/0x256 [ 6160.493275] ? do_raw_spin_unlock+0x75/0x190 [ 6160.495407] ? _raw_spin_unlock+0x12/0x30 [ 6160.499459] do_sys_openat2+0x2b4/0x410 [ 6160.506781] do_sys_open+0x73/0xa0 [ 6160.509033] __x64_sys_openat+0x24/0x30 [ 6160.512331] do_syscall_64+0xc1/0x440 [ 6160.515735] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6160.518859] RIP: 0033:0x7ff54af45332 [ 6160.527843] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6160.543418] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6160.548289] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 6160.553140] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 6160.558259] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 6160.563171] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 6160.566990] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 6168.065172] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 6168.092301] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6168.154379] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6168.162386] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6168.175910] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6168.184935] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6168.192336] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6168.208200] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6168.214316] RAX: 000000000000690d RBX: ffff96848811cae0 RCX: 0000000000000001 [ 6168.219300] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 6168.225831] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 6168.231646] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 6168.237332] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 6168.243065] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 6168.249025] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6168.251117] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 6168.261776] Call Trace: [ 6168.263855] [ 6168.265380] ? show_regs.cold.9+0x22/0x2f [ 6168.268269] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6168.272961] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6168.276182] ? watchdog+0x40/0x40 [ 6168.278868] ? hrtimer_interrupt+0x128/0x2c0 [ 6168.281476] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6168.285670] ? apic_timer_interrupt+0xf/0x20 [ 6168.290496] [ 6168.291985] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6168.300381] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6168.306662] do_raw_spin_lock+0x177/0x190 [ 6168.310401] _raw_spin_lock+0x21/0x30 [ 6168.317483] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6168.322099] process_one_work+0x2c8/0x700 [ 6168.328440] worker_thread+0x296/0x6e0 [ 6168.330119] ? rescuer_thread+0x570/0x570 [ 6168.335507] kthread+0x1d1/0x200 [ 6168.338987] ? set_kthread_struct+0x70/0x70 [ 6168.341901] ret_from_fork+0x1f/0x30 [ 6176.228675] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4217s! [ 6176.240683] Showing busy workqueues and worker pools: [ 6176.245898] workqueue events: flags=0x0 [ 6176.252310] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6176.260482] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 6176.269636] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6176.278267] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 6176.295481] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6176.307056] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 6176.316644] workqueue events_long: flags=0x0 [ 6176.322038] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6176.328091] pending: lu_objects_flush [obdclass] [ 6176.337031] workqueue events_unbound: flags=0x2 [ 6176.343956] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6176.353062] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6176.370099] workqueue events_power_efficient: flags=0x80 [ 6176.378109] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6176.399133] pending: neigh_periodic_work [ 6176.402643] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6176.410688] pending: do_cache_clean, check_lifetime [ 6176.417236] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6176.425912] pending: neigh_periodic_work [ 6176.437164] workqueue writeback: flags=0x4a [ 6176.447241] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6176.463956] pending: wb_workfn, wb_workfn [ 6176.466836] workqueue xprtiod: flags=0xa [ 6176.478532] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6176.492335] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 6176.504888] workqueue ipv6_addrconf: flags=0x40008 [ 6176.518617] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6176.526002] pending: addrconf_verify_work [ 6176.536060] workqueue knbd0-recv: flags=0x1a [ 6176.543279] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6176.554979] in-flight: 657:recv_work [ 6176.558629] workqueue ptlrpc_pinger: flags=0x2 [ 6176.570805] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6176.581855] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 6176.605255] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4217s workers=4 idle: 20831 12642 28859 [ 6176.608457] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6166s workers=2 idle: 659 [ 6180.058355] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 6180.067222] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6180.151689] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6180.166243] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6180.176973] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6180.190661] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6180.215116] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6180.219331] RAX: 0000000000007b9f RBX: ffff96848811cae0 RCX: 0000000000000001 [ 6180.226141] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 6180.235615] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6180.248765] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 6180.265411] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 6180.273997] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 6180.279548] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6180.281784] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 6180.288733] Call Trace: [ 6180.290616] [ 6180.292988] ? show_regs.cold.9+0x22/0x2f [ 6180.297469] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6180.301101] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6180.304731] ? watchdog+0x40/0x40 [ 6180.308998] ? hrtimer_interrupt+0x128/0x2c0 [ 6180.313682] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6180.322458] ? apic_timer_interrupt+0xf/0x20 [ 6180.327961] [ 6180.330062] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6180.333033] do_raw_spin_lock+0x177/0x190 [ 6180.341034] _raw_spin_lock+0x21/0x30 [ 6180.344978] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6180.347037] ? do_raw_spin_unlock+0x75/0x190 [ 6180.354096] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6180.361288] ? do_wait_intr+0xf0/0xf0 [ 6180.363217] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6180.369694] mdc_close+0x53b/0xd70 [mdc] [ 6180.374492] lmv_close+0x18d/0x490 [lmv] [ 6180.383025] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 6180.396800] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6180.400847] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6180.404420] ll_md_close+0x274/0x800 [lustre] [ 6180.405951] ll_file_release+0x128/0xc50 [lustre] [ 6180.409887] __fput+0x10d/0x440 [ 6180.411778] ____fput+0x12/0x20 [ 6180.414161] task_work_run+0xc6/0x110 [ 6180.418190] exit_to_usermode_loop+0x1dd/0x1f0 [ 6180.425175] do_syscall_64+0x3d6/0x440 [ 6180.427985] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6180.432884] RIP: 0033:0x7ff54af45cd8 [ 6180.441725] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6180.458614] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6180.466131] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 6180.475603] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 6180.481617] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 6180.488849] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 6180.495916] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6188.062395] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [ 6188.080966] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6188.167883] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6188.184540] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6188.200170] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6188.204705] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6188.231654] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6188.238361] RAX: 000000000000004f RBX: ffff96848811cae0 RCX: 0000000000000001 [ 6188.243163] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 6188.248143] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6188.256062] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 6188.263010] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 6188.271202] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 6188.275891] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6188.284089] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 6188.296172] Call Trace: [ 6188.301170] [ 6188.304271] ? show_regs.cold.9+0x22/0x2f [ 6188.306866] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6188.310079] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6188.318112] ? watchdog+0x40/0x40 [ 6188.329794] ? hrtimer_interrupt+0x128/0x2c0 [ 6188.333171] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6188.340615] ? apic_timer_interrupt+0xf/0x20 [ 6188.345052] [ 6188.348057] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6188.353116] do_raw_spin_lock+0x177/0x190 [ 6188.356223] _raw_spin_lock+0x21/0x30 [ 6188.361442] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6188.367095] ? do_raw_spin_unlock+0x75/0x190 [ 6188.373009] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6188.377895] ? do_wait_intr+0xf0/0xf0 [ 6188.387474] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6188.396316] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6188.398295] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6188.414129] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6188.424489] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6188.429873] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6188.439184] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6188.452424] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6188.461643] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6188.471816] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6188.473826] lmv_intent_open+0x338/0x1150 [lmv] [ 6188.487834] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6188.499737] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6188.504527] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6188.506766] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6188.511068] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6188.516666] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6188.522749] ? do_raw_spin_unlock+0x75/0x190 [ 6188.523864] ? _raw_spin_unlock+0x12/0x30 [ 6188.530504] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6188.536765] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6188.544062] ll_atomic_open+0x36a/0xd70 [lustre] [ 6188.552498] lookup_open+0x356/0xb40 [ 6188.559591] do_last+0x313/0xfc0 [ 6188.562653] ? nd_jump_root+0xe5/0x160 [ 6188.565344] ? path_init+0x437/0x520 [ 6188.566838] path_openat+0xf7/0x500 [ 6188.574285] ? do_raw_spin_unlock+0x75/0x190 [ 6188.577630] do_filp_open+0x99/0x140 [ 6188.580319] ? getname_flags+0x6e/0x330 [ 6188.587493] ? __check_object_size+0xff/0x256 [ 6188.594779] ? do_raw_spin_unlock+0x75/0x190 [ 6188.602075] ? _raw_spin_unlock+0x12/0x30 [ 6188.606020] do_sys_openat2+0x2b4/0x410 [ 6188.613416] do_sys_open+0x73/0xa0 [ 6188.616150] __x64_sys_openat+0x24/0x30 [ 6188.633768] do_syscall_64+0xc1/0x440 [ 6188.638704] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6188.658184] RIP: 0033:0x7ff54af45332 [ 6188.661012] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6188.689807] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6188.694201] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 6188.703871] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 6188.712049] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 6188.719592] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 6188.729936] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 6196.059054] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 6196.062151] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6196.132761] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6196.153914] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6196.168616] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6196.179672] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6196.181601] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6196.199080] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6196.203917] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 6196.207621] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 6196.217461] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 6196.224189] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 6196.229046] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 6196.234844] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 6196.245455] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6196.248891] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 6196.258026] Call Trace: [ 6196.259842] [ 6196.268546] ? show_regs.cold.9+0x22/0x2f [ 6196.275456] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6196.291821] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6196.301503] ? watchdog+0x40/0x40 [ 6196.302975] ? hrtimer_interrupt+0x128/0x2c0 [ 6196.316397] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6196.323458] ? apic_timer_interrupt+0xf/0x20 [ 6196.329849] [ 6196.332679] ? native_safe_halt+0x2e/0x50 [ 6196.338909] kvm_wait+0xcd/0xe0 [ 6196.343558] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6196.355463] do_raw_spin_lock+0x177/0x190 [ 6196.363181] _raw_spin_lock+0x21/0x30 [ 6196.370237] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6196.389767] process_one_work+0x2c8/0x700 [ 6196.398785] worker_thread+0x296/0x6e0 [ 6196.403167] ? rescuer_thread+0x570/0x570 [ 6196.407216] kthread+0x1d1/0x200 [ 6196.411069] ? set_kthread_struct+0x70/0x70 [ 6196.416301] ret_from_fork+0x1f/0x30 [ 6204.058291] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 6204.065684] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6204.145156] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6204.158332] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6204.166780] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6204.173427] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6204.192828] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6204.201316] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 6204.211066] RDX: 0000000000002226 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 6204.221623] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 6204.226296] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 6204.234841] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6204.246889] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 6204.269855] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6204.280086] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 6204.286315] Call Trace: [ 6204.291921] [ 6204.298522] ? show_regs.cold.9+0x22/0x2f [ 6204.300450] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6204.307141] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6204.319135] ? watchdog+0x40/0x40 [ 6204.321981] ? hrtimer_interrupt+0x128/0x2c0 [ 6204.328582] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6204.339940] ? apic_timer_interrupt+0xf/0x20 [ 6204.347943] [ 6204.351332] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6204.356468] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6204.359999] do_raw_spin_lock+0x177/0x190 [ 6204.363214] _raw_spin_lock+0x21/0x30 [ 6204.370160] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6204.376917] obd_mod_free+0x46/0x140 [ptlrpc] [ 6204.379595] mdc_commit_open+0x91/0xb0 [mdc] [ 6204.383576] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6204.386794] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6204.389907] after_reply+0xe2c/0x1780 [ptlrpc] [ 6204.392138] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6204.399431] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6204.402614] ? do_wait_intr+0xf0/0xf0 [ 6204.408649] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6204.412523] kthread+0x1d1/0x200 [ 6204.414732] ? set_kthread_struct+0x70/0x70 [ 6204.417177] ret_from_fork+0x1f/0x30 [ 6206.945131] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4248s! [ 6206.949700] Showing busy workqueues and worker pools: [ 6206.954403] workqueue events: flags=0x0 [ 6206.958900] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6206.965619] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 6206.972248] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6206.975637] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 6206.981652] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6206.985079] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 6206.987658] workqueue events_long: flags=0x0 [ 6206.990206] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6206.993399] pending: lu_objects_flush [obdclass] [ 6206.995581] workqueue events_unbound: flags=0x2 [ 6206.997288] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6206.999756] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6207.004673] workqueue events_power_efficient: flags=0x80 [ 6207.007300] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6207.010299] pending: neigh_periodic_work [ 6207.012486] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6207.015671] pending: do_cache_clean, check_lifetime [ 6207.018170] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6207.022905] pending: neigh_periodic_work [ 6207.025162] workqueue writeback: flags=0x4a [ 6207.027126] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6207.030227] pending: wb_workfn, wb_workfn [ 6207.032366] workqueue xprtiod: flags=0xa [ 6207.034159] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6207.037632] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 6207.041830] workqueue ipv6_addrconf: flags=0x40008 [ 6207.044433] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6207.048688] pending: addrconf_verify_work [ 6207.050924] workqueue knbd0-recv: flags=0x1a [ 6207.053290] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6207.057619] in-flight: 657:recv_work [ 6207.060143] workqueue ptlrpc_pinger: flags=0x2 [ 6207.062328] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6207.065491] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 6207.068541] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4248s workers=4 idle: 20831 12642 28859 [ 6207.072880] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6197s workers=2 idle: 659 [ 6208.059162] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 6208.068462] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6208.127268] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6208.141550] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6208.156510] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6208.162353] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6208.184497] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6208.195575] RAX: 0000000000002b10 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 6208.205348] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 6208.215178] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6208.224870] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 6208.233418] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 6208.245082] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 6208.255195] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6208.257715] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 6208.267277] Call Trace: [ 6208.272567] [ 6208.276006] ? show_regs.cold.9+0x22/0x2f [ 6208.279357] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6208.285897] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6208.291612] ? watchdog+0x40/0x40 [ 6208.296502] ? hrtimer_interrupt+0x128/0x2c0 [ 6208.303818] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6208.307999] ? apic_timer_interrupt+0xf/0x20 [ 6208.309922] [ 6208.321441] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6208.324475] do_raw_spin_lock+0x177/0x190 [ 6208.329890] _raw_spin_lock+0x21/0x30 [ 6208.335872] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6208.345219] ? do_raw_spin_unlock+0x75/0x190 [ 6208.351857] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6208.358572] ? do_wait_intr+0xf0/0xf0 [ 6208.363885] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6208.373815] mdc_close+0x53b/0xd70 [mdc] [ 6208.376712] lmv_close+0x18d/0x490 [lmv] [ 6208.383508] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 6208.389724] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6208.398598] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6208.406620] ll_md_close+0x274/0x800 [lustre] [ 6208.413788] ll_file_release+0x128/0xc50 [lustre] [ 6208.420764] __fput+0x10d/0x440 [ 6208.425550] ____fput+0x12/0x20 [ 6208.427469] task_work_run+0xc6/0x110 [ 6208.429413] exit_to_usermode_loop+0x1dd/0x1f0 [ 6208.434708] do_syscall_64+0x3d6/0x440 [ 6208.438478] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6208.444001] RIP: 0033:0x7ff54af45cd8 [ 6208.447995] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6208.475561] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6208.486427] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 6208.493659] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 6208.499837] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 6208.512538] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 6208.521233] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6216.065118] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [ 6216.072571] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6216.136870] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6216.144168] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6216.152927] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6216.159855] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6216.186516] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6216.195532] RAX: 0000000000005819 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 6216.199781] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 6216.208851] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6216.216402] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 6216.227341] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 6216.231059] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 6216.238595] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6216.241276] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 6216.244864] Call Trace: [ 6216.246799] [ 6216.249582] ? show_regs.cold.9+0x22/0x2f [ 6216.252849] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6216.262095] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6216.265825] ? watchdog+0x40/0x40 [ 6216.267327] ? hrtimer_interrupt+0x128/0x2c0 [ 6216.272218] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6216.282139] ? apic_timer_interrupt+0xf/0x20 [ 6216.283249] [ 6216.283732] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6216.285044] do_raw_spin_lock+0x177/0x190 [ 6216.289242] _raw_spin_lock+0x21/0x30 [ 6216.291536] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6216.294581] ? do_raw_spin_unlock+0x75/0x190 [ 6216.296852] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6216.299885] ? do_wait_intr+0xf0/0xf0 [ 6216.302358] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6216.306878] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6216.314010] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6216.318856] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6216.321409] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6216.325468] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6216.327214] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6216.329005] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6216.331911] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6216.336162] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6216.339654] lmv_intent_open+0x338/0x1150 [lmv] [ 6216.341452] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6216.344627] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6216.349472] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6216.351028] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6216.357352] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6216.367439] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6216.373940] ? do_raw_spin_unlock+0x75/0x190 [ 6216.376647] ? _raw_spin_unlock+0x12/0x30 [ 6216.384195] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6216.388637] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6216.393335] ll_atomic_open+0x36a/0xd70 [lustre] [ 6216.395355] lookup_open+0x356/0xb40 [ 6216.396345] do_last+0x313/0xfc0 [ 6216.410902] ? nd_jump_root+0xe5/0x160 [ 6216.416900] ? path_init+0x437/0x520 [ 6216.422443] path_openat+0xf7/0x500 [ 6216.426863] ? do_raw_spin_unlock+0x75/0x190 [ 6216.431114] do_filp_open+0x99/0x140 [ 6216.434622] ? getname_flags+0x6e/0x330 [ 6216.437361] ? __check_object_size+0xff/0x256 [ 6216.439491] ? do_raw_spin_unlock+0x75/0x190 [ 6216.442524] ? _raw_spin_unlock+0x12/0x30 [ 6216.446172] do_sys_openat2+0x2b4/0x410 [ 6216.447384] do_sys_open+0x73/0xa0 [ 6216.449529] __x64_sys_openat+0x24/0x30 [ 6216.452317] do_syscall_64+0xc1/0x440 [ 6216.455211] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6216.460359] RIP: 0033:0x7ff54af45332 [ 6216.464140] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6216.483743] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6216.493387] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 6216.495365] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 6216.502189] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 6216.509217] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 6216.512417] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 6224.059326] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 6224.066790] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6224.142930] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6224.165298] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6224.179436] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6224.184487] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6224.190079] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6224.207074] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6224.214067] RAX: 0000000000007c85 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 6224.220098] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 6224.226980] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 6224.240357] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 6224.241906] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 6224.246862] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 6224.255858] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6224.263752] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 6224.272730] Call Trace: [ 6224.279202] [ 6224.279730] ? show_regs.cold.9+0x22/0x2f [ 6224.281127] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6224.289381] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6224.294650] ? watchdog+0x40/0x40 [ 6224.301483] ? hrtimer_interrupt+0x128/0x2c0 [ 6224.307505] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6224.309019] ? apic_timer_interrupt+0xf/0x20 [ 6224.316535] [ 6224.319624] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6224.329303] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6224.333479] do_raw_spin_lock+0x177/0x190 [ 6224.338013] _raw_spin_lock+0x21/0x30 [ 6224.343429] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6224.350277] process_one_work+0x2c8/0x700 [ 6224.354349] worker_thread+0x296/0x6e0 [ 6224.365964] ? rescuer_thread+0x570/0x570 [ 6224.370841] kthread+0x1d1/0x200 [ 6224.373772] ? set_kthread_struct+0x70/0x70 [ 6224.380851] ret_from_fork+0x1f/0x30 [ 6232.056266] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [ 6232.066767] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6232.127921] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6232.136797] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6232.147070] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6232.151287] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6232.183761] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6232.192107] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 6232.205352] RDX: 000000000000379f RSI: 0000000000000003 RDI: ffff96848811cae0 [ 6232.215500] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 6232.223006] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 6232.235609] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6232.252476] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 6232.261029] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6232.271284] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 6232.276239] Call Trace: [ 6232.280262] [ 6232.281412] ? show_regs.cold.9+0x22/0x2f [ 6232.287796] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6232.292858] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6232.299717] ? watchdog+0x40/0x40 [ 6232.301249] ? hrtimer_interrupt+0x128/0x2c0 [ 6232.306041] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6232.309711] ? apic_timer_interrupt+0xf/0x20 [ 6232.321756] [ 6232.323895] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6232.327295] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6232.331347] do_raw_spin_lock+0x177/0x190 [ 6232.335837] _raw_spin_lock+0x21/0x30 [ 6232.337487] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6232.344294] obd_mod_free+0x46/0x140 [ptlrpc] [ 6232.348489] mdc_commit_open+0x91/0xb0 [mdc] [ 6232.354363] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6232.358443] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6232.370326] after_reply+0xe2c/0x1780 [ptlrpc] [ 6232.375240] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6232.381491] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6232.388153] ? do_wait_intr+0xf0/0xf0 [ 6232.392243] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6232.397722] kthread+0x1d1/0x200 [ 6232.407333] ? set_kthread_struct+0x70/0x70 [ 6232.410552] ret_from_fork+0x1f/0x30 [ 6236.065837] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 6236.072690] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6236.116671] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6236.123597] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6236.130504] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6236.135717] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6236.159103] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6236.167639] RAX: 0000000000007f86 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 6236.177627] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 6236.188974] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6236.191497] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 6236.202671] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 6236.212048] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 6236.222501] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6236.224844] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 6236.237755] Call Trace: [ 6236.240372] [ 6236.241437] ? show_regs.cold.9+0x22/0x2f [ 6236.243419] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6236.246148] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6236.248850] ? watchdog+0x40/0x40 [ 6236.250308] ? hrtimer_interrupt+0x128/0x2c0 [ 6236.253511] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6236.254737] ? apic_timer_interrupt+0xf/0x20 [ 6236.255658] [ 6236.260736] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6236.262466] do_raw_spin_lock+0x177/0x190 [ 6236.263455] _raw_spin_lock+0x21/0x30 [ 6236.266370] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6236.271595] ? do_raw_spin_unlock+0x75/0x190 [ 6236.274444] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6236.280523] ? do_wait_intr+0xf0/0xf0 [ 6236.281805] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6236.285328] mdc_close+0x53b/0xd70 [mdc] [ 6236.286714] lmv_close+0x18d/0x490 [lmv] [ 6236.289130] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 6236.294793] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6236.300167] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6236.306879] ll_md_close+0x274/0x800 [lustre] [ 6236.323537] ll_file_release+0x128/0xc50 [lustre] [ 6236.329883] __fput+0x10d/0x440 [ 6236.334181] ____fput+0x12/0x20 [ 6236.339657] task_work_run+0xc6/0x110 [ 6236.342546] exit_to_usermode_loop+0x1dd/0x1f0 [ 6236.347993] do_syscall_64+0x3d6/0x440 [ 6236.349468] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6236.357309] RIP: 0033:0x7ff54af45cd8 [ 6236.358612] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6236.379708] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6236.385442] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 6236.388710] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 6236.406163] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 6236.413495] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 6236.420052] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6237.156485] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4278s! [ 6237.175490] Showing busy workqueues and worker pools: [ 6237.184093] workqueue events: flags=0x0 [ 6237.187507] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6237.199294] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 6237.213895] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6237.221244] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 6237.233396] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6237.245426] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 6237.253677] workqueue events_long: flags=0x0 [ 6237.257049] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6237.262470] pending: lu_objects_flush [obdclass] [ 6237.267615] workqueue events_unbound: flags=0x2 [ 6237.272039] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6237.279118] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6237.286598] workqueue events_power_efficient: flags=0x80 [ 6237.292198] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6237.296901] pending: neigh_periodic_work [ 6237.298424] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6237.309101] pending: do_cache_clean, check_lifetime [ 6237.315192] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6237.324183] pending: neigh_periodic_work [ 6237.325190] workqueue writeback: flags=0x4a [ 6237.326193] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6237.332207] pending: wb_workfn, wb_workfn [ 6237.333800] workqueue xprtiod: flags=0xa [ 6237.344059] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6237.348848] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 6237.355683] workqueue ipv6_addrconf: flags=0x40008 [ 6237.359253] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6237.373047] pending: addrconf_verify_work [ 6237.377023] workqueue knbd0-recv: flags=0x1a [ 6237.379109] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6237.382453] in-flight: 657:recv_work [ 6237.386175] workqueue ptlrpc_pinger: flags=0x2 [ 6237.390336] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6237.400467] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 6237.416034] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4278s workers=4 idle: 20831 12642 28859 [ 6237.427387] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6227s workers=2 idle: 659 [ 6244.061436] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [ 6244.076521] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6244.130272] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6244.139973] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6244.152005] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6244.162237] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6244.184186] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6244.190581] RAX: 000000000000204a RBX: ffff96848811cae0 RCX: 0000000000000001 [ 6244.196311] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 6244.203318] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6244.211576] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 6244.213821] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 6244.220810] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 6244.230337] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6244.238271] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 6244.242664] Call Trace: [ 6244.244514] [ 6244.247200] ? show_regs.cold.9+0x22/0x2f [ 6244.251241] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6244.254902] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6244.259316] ? watchdog+0x40/0x40 [ 6244.260769] ? hrtimer_interrupt+0x128/0x2c0 [ 6244.262551] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6244.270256] ? apic_timer_interrupt+0xf/0x20 [ 6244.272515] [ 6244.275284] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6244.282682] do_raw_spin_lock+0x177/0x190 [ 6244.290144] _raw_spin_lock+0x21/0x30 [ 6244.298855] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6244.311631] ? do_raw_spin_unlock+0x75/0x190 [ 6244.316980] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6244.324164] ? do_wait_intr+0xf0/0xf0 [ 6244.327673] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6244.339183] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6244.349459] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6244.356972] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6244.361451] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6244.365506] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6244.367159] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6244.368636] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6244.383256] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6244.393273] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6244.398601] lmv_intent_open+0x338/0x1150 [lmv] [ 6244.404318] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6244.409287] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6244.413518] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6244.416193] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6244.418409] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6244.422934] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6244.424647] ? do_raw_spin_unlock+0x75/0x190 [ 6244.429662] ? _raw_spin_unlock+0x12/0x30 [ 6244.432278] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6244.435419] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6244.438400] ll_atomic_open+0x36a/0xd70 [lustre] [ 6244.445448] lookup_open+0x356/0xb40 [ 6244.446594] do_last+0x313/0xfc0 [ 6244.447676] ? nd_jump_root+0xe5/0x160 [ 6244.452432] ? path_init+0x437/0x520 [ 6244.453554] path_openat+0xf7/0x500 [ 6244.457476] ? do_raw_spin_unlock+0x75/0x190 [ 6244.460168] do_filp_open+0x99/0x140 [ 6244.461544] ? getname_flags+0x6e/0x330 [ 6244.463990] ? __check_object_size+0xff/0x256 [ 6244.465482] ? do_raw_spin_unlock+0x75/0x190 [ 6244.468926] ? _raw_spin_unlock+0x12/0x30 [ 6244.474278] do_sys_openat2+0x2b4/0x410 [ 6244.475958] do_sys_open+0x73/0xa0 [ 6244.477205] __x64_sys_openat+0x24/0x30 [ 6244.478644] do_syscall_64+0xc1/0x440 [ 6244.480519] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6244.483202] RIP: 0033:0x7ff54af45332 [ 6244.484748] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6244.494891] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6244.497604] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 6244.500255] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 6244.503408] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 6244.505936] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 6244.508762] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 6252.059243] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 6252.065122] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6252.109733] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6252.113297] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6252.117500] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6252.121935] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6252.124156] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6252.144277] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6252.150510] RAX: 0000000000001e8d RBX: ffff96848811cae0 RCX: 0000000000000001 [ 6252.159518] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 6252.163246] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 6252.169549] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 6252.174006] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 6252.184091] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 6252.192121] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6252.194361] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 6252.196386] Call Trace: [ 6252.199599] [ 6252.202166] ? show_regs.cold.9+0x22/0x2f [ 6252.203406] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6252.207619] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6252.209054] ? watchdog+0x40/0x40 [ 6252.211627] ? hrtimer_interrupt+0x128/0x2c0 [ 6252.214035] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6252.219599] ? apic_timer_interrupt+0xf/0x20 [ 6252.221949] [ 6252.227049] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6252.232345] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6252.236295] do_raw_spin_lock+0x177/0x190 [ 6252.238812] _raw_spin_lock+0x21/0x30 [ 6252.241212] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6252.249297] process_one_work+0x2c8/0x700 [ 6252.254186] worker_thread+0x296/0x6e0 [ 6252.260489] ? rescuer_thread+0x570/0x570 [ 6252.262819] kthread+0x1d1/0x200 [ 6252.266766] ? set_kthread_struct+0x70/0x70 [ 6252.273892] ret_from_fork+0x1f/0x30 [ 6264.055020] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 6264.061274] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6264.064841] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 6264.133154] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6264.133168] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6264.133172] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6264.133186] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6264.133191] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6264.133197] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 6264.133200] RDX: 000000000000724f RSI: 0000000000000003 RDI: ffff96848811cae0 [ 6264.133203] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 6264.133206] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 6264.133209] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6264.133212] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 6264.133215] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6264.133218] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 6264.133226] Call Trace: [ 6264.133232] [ 6264.133237] ? show_regs.cold.9+0x22/0x2f [ 6264.133246] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6264.133253] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6264.133257] ? watchdog+0x40/0x40 [ 6264.133262] ? hrtimer_interrupt+0x128/0x2c0 [ 6264.133267] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6264.133275] ? apic_timer_interrupt+0xf/0x20 [ 6264.136814] [ 6264.136824] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6264.136832] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6264.136838] do_raw_spin_lock+0x177/0x190 [ 6264.136845] _raw_spin_lock+0x21/0x30 [ 6264.136853] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6264.137160] obd_mod_free+0x46/0x140 [ptlrpc] [ 6264.137370] mdc_commit_open+0x91/0xb0 [mdc] [ 6264.137412] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6264.159871] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6264.159954] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6264.159964] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6264.159967] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6264.159982] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6264.159987] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6264.159992] RAX: 000000000000749f RBX: ffff96848811cae0 RCX: 0000000000000001 [ 6264.159994] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 6264.159997] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6264.159999] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 6264.160001] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 6264.160004] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 6264.160007] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6264.160009] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 6264.160018] Call Trace: [ 6264.160023] [ 6264.160030] ? show_regs.cold.9+0x22/0x2f [ 6264.160041] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6264.160048] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6264.160053] ? watchdog+0x40/0x40 [ 6264.160057] ? hrtimer_interrupt+0x128/0x2c0 [ 6264.160061] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6264.160069] ? apic_timer_interrupt+0xf/0x20 [ 6264.160074] [ 6264.160077] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6264.160084] do_raw_spin_lock+0x177/0x190 [ 6264.160088] _raw_spin_lock+0x21/0x30 [ 6264.160094] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6264.172656] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6264.173007] after_reply+0xe2c/0x1780 [ptlrpc] [ 6264.174816] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6264.175131] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6264.175595] ? do_wait_intr+0xf0/0xf0 [ 6264.175608] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6264.176999] kthread+0x1d1/0x200 [ 6264.177011] ? set_kthread_struct+0x70/0x70 [ 6264.177029] ret_from_fork+0x1f/0x30 [ 6264.529806] ? do_raw_spin_unlock+0x75/0x190 [ 6264.533407] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6264.537368] ? do_wait_intr+0xf0/0xf0 [ 6264.538845] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6264.543908] mdc_close+0x53b/0xd70 [mdc] [ 6264.546106] lmv_close+0x18d/0x490 [lmv] [ 6264.548645] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 6264.552211] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6264.555793] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6264.558258] ll_md_close+0x274/0x800 [lustre] [ 6264.560649] ll_file_release+0x128/0xc50 [lustre] [ 6264.563105] __fput+0x10d/0x440 [ 6264.564721] ____fput+0x12/0x20 [ 6264.566902] task_work_run+0xc6/0x110 [ 6264.569050] exit_to_usermode_loop+0x1dd/0x1f0 [ 6264.571261] do_syscall_64+0x3d6/0x440 [ 6264.573028] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6264.575889] RIP: 0033:0x7ff54af45cd8 [ 6264.577810] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6264.600201] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6264.602359] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 6264.607723] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 6264.613834] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 6264.617773] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 6264.621744] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6267.873381] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4308s! [ 6267.884368] Showing busy workqueues and worker pools: [ 6267.892917] workqueue events: flags=0x0 [ 6267.905495] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6267.921406] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 6267.935803] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6267.940051] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 6267.957883] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6267.967213] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 6267.974764] workqueue events_long: flags=0x0 [ 6267.983232] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6267.993346] pending: lu_objects_flush [obdclass] [ 6267.998463] workqueue events_unbound: flags=0x2 [ 6268.005421] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6268.014317] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6268.024119] workqueue events_power_efficient: flags=0x80 [ 6268.036347] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6268.038602] pending: neigh_periodic_work [ 6268.054803] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6268.060614] pending: do_cache_clean, check_lifetime [ 6268.072061] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6268.080170] pending: neigh_periodic_work [ 6268.088782] workqueue writeback: flags=0x4a [ 6268.093334] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6268.097762] pending: wb_workfn, wb_workfn [ 6268.106203] workqueue xprtiod: flags=0xa [ 6268.112827] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6268.124598] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 6268.132789] workqueue ipv6_addrconf: flags=0x40008 [ 6268.141354] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6268.143593] pending: addrconf_verify_work [ 6268.151891] workqueue knbd0-recv: flags=0x1a [ 6268.153428] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6268.159189] in-flight: 657:recv_work [ 6268.168096] workqueue ptlrpc_pinger: flags=0x2 [ 6268.171690] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6268.174664] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 6268.185503] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4309s workers=4 idle: 20831 12642 28859 [ 6268.193694] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6258s workers=2 idle: 659 [ 6272.061994] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [ 6272.074157] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6272.120145] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6272.136188] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6272.142981] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6272.146683] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6272.162379] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6272.168815] RAX: 0000000000006172 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 6272.177569] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 6272.181308] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6272.184282] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 6272.188315] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 6272.191602] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 6272.193899] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6272.196256] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 6272.199362] Call Trace: [ 6272.200400] [ 6272.201267] ? show_regs.cold.9+0x22/0x2f [ 6272.203296] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6272.205237] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6272.207212] ? watchdog+0x40/0x40 [ 6272.208569] ? hrtimer_interrupt+0x128/0x2c0 [ 6272.210546] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6272.212835] ? apic_timer_interrupt+0xf/0x20 [ 6272.214878] [ 6272.215998] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6272.218207] do_raw_spin_lock+0x177/0x190 [ 6272.220003] _raw_spin_lock+0x21/0x30 [ 6272.221669] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6272.224272] ? do_raw_spin_unlock+0x75/0x190 [ 6272.226367] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6272.228893] ? do_wait_intr+0xf0/0xf0 [ 6272.230904] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6272.233111] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6272.235613] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6272.238600] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6272.241361] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6272.243800] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6272.245943] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6272.248051] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6272.250340] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6272.253648] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6272.256188] lmv_intent_open+0x338/0x1150 [lmv] [ 6272.258477] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6272.261074] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6272.263881] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6272.265967] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6272.268165] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6272.271541] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6272.274386] ? do_raw_spin_unlock+0x75/0x190 [ 6272.276516] ? _raw_spin_unlock+0x12/0x30 [ 6272.278570] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6272.281114] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6272.283834] ll_atomic_open+0x36a/0xd70 [lustre] [ 6272.285958] lookup_open+0x356/0xb40 [ 6272.287659] do_last+0x313/0xfc0 [ 6272.289021] ? nd_jump_root+0xe5/0x160 [ 6272.290261] ? path_init+0x437/0x520 [ 6272.291914] path_openat+0xf7/0x500 [ 6272.293267] ? do_raw_spin_unlock+0x75/0x190 [ 6272.295156] do_filp_open+0x99/0x140 [ 6272.296714] ? getname_flags+0x6e/0x330 [ 6272.298245] ? __check_object_size+0xff/0x256 [ 6272.300795] ? do_raw_spin_unlock+0x75/0x190 [ 6272.302686] ? _raw_spin_unlock+0x12/0x30 [ 6272.304493] do_sys_openat2+0x2b4/0x410 [ 6272.306300] do_sys_open+0x73/0xa0 [ 6272.307953] __x64_sys_openat+0x24/0x30 [ 6272.309825] do_syscall_64+0xc1/0x440 [ 6272.311386] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6272.313732] RIP: 0033:0x7ff54af45332 [ 6272.316782] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6272.324621] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6272.327059] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 6272.330570] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 6272.333575] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 6272.337167] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 6272.340206] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 6280.062785] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 6280.075306] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6280.116605] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6280.123911] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6280.127611] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6280.130599] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6280.135693] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6280.143985] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6280.147859] RAX: 0000000000007c3f RBX: ffff96848811cae0 RCX: 0000000000000001 [ 6280.153629] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 6280.157566] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 6280.159996] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 6280.162984] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 6280.165204] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 6280.170552] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6280.172127] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 6280.174031] Call Trace: [ 6280.174733] [ 6280.175175] ? show_regs.cold.9+0x22/0x2f [ 6280.177042] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6280.179275] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6280.181745] ? watchdog+0x40/0x40 [ 6280.182827] ? hrtimer_interrupt+0x128/0x2c0 [ 6280.183985] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6280.185107] ? apic_timer_interrupt+0xf/0x20 [ 6280.186988] [ 6280.187632] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6280.189082] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6280.190747] do_raw_spin_lock+0x177/0x190 [ 6280.191823] _raw_spin_lock+0x21/0x30 [ 6280.192823] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6280.194738] process_one_work+0x2c8/0x700 [ 6280.198042] worker_thread+0x296/0x6e0 [ 6280.201544] ? rescuer_thread+0x570/0x570 [ 6280.214861] kthread+0x1d1/0x200 [ 6280.216044] ? set_kthread_struct+0x70/0x70 [ 6280.225253] ret_from_fork+0x1f/0x30 [ 6292.062649] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 6292.071950] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6292.111088] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6292.119924] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6292.129880] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6292.134947] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6292.147707] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6292.154866] RAX: 0000000000003414 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 6292.161648] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 6292.167466] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6292.174768] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 6292.182648] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 6292.190995] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 6292.198685] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6292.206459] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 6292.213849] Call Trace: [ 6292.215651] [ 6292.217900] ? show_regs.cold.9+0x22/0x2f [ 6292.220612] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6292.222701] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6292.225992] ? watchdog+0x40/0x40 [ 6292.227550] ? hrtimer_interrupt+0x128/0x2c0 [ 6292.230009] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6292.232324] ? apic_timer_interrupt+0xf/0x20 [ 6292.238038] [ 6292.238960] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6292.242639] do_raw_spin_lock+0x177/0x190 [ 6292.244847] _raw_spin_lock+0x21/0x30 [ 6292.246413] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6292.249153] ? do_raw_spin_unlock+0x75/0x190 [ 6292.250741] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6292.253385] ? do_wait_intr+0xf0/0xf0 [ 6292.255513] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6292.258900] mdc_close+0x53b/0xd70 [mdc] [ 6292.260913] lmv_close+0x18d/0x490 [lmv] [ 6292.264594] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 6292.267748] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6292.277406] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6292.281042] ll_md_close+0x274/0x800 [lustre] [ 6292.288504] ll_file_release+0x128/0xc50 [lustre] [ 6292.291838] __fput+0x10d/0x440 [ 6292.300570] ____fput+0x12/0x20 [ 6292.301928] task_work_run+0xc6/0x110 [ 6292.303189] exit_to_usermode_loop+0x1dd/0x1f0 [ 6292.311390] do_syscall_64+0x3d6/0x440 [ 6292.312699] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6292.317844] RIP: 0033:0x7ff54af45cd8 [ 6292.320778] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6292.336816] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6292.342063] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 6292.349819] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 6292.353967] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 6292.357763] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 6292.363707] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6296.055057] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 6296.059875] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6296.077831] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6296.082836] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6296.085447] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6296.087091] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6296.100981] RSP: 0000:ffffaf9fc1b97bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6296.108086] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 6296.113562] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 6296.118278] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 6296.121930] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 6296.130548] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6296.139399] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 6296.148978] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6296.155640] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 6296.164300] Call Trace: [ 6296.166252] [ 6296.169333] ? show_regs.cold.9+0x22/0x2f [ 6296.173910] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6296.179157] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6296.182383] ? watchdog+0x40/0x40 [ 6296.184932] ? hrtimer_interrupt+0x128/0x2c0 [ 6296.187381] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6296.191060] ? apic_timer_interrupt+0xf/0x20 [ 6296.194416] [ 6296.195570] ? native_safe_halt+0x2e/0x50 [ 6296.198735] kvm_wait+0xcd/0xe0 [ 6296.203432] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6296.207888] do_raw_spin_lock+0x177/0x190 [ 6296.211698] _raw_spin_lock+0x21/0x30 [ 6296.215974] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6296.221068] obd_mod_free+0x46/0x140 [ptlrpc] [ 6296.227995] mdc_commit_open+0x91/0xb0 [mdc] [ 6296.230413] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6296.234325] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6296.241133] after_reply+0xe2c/0x1780 [ptlrpc] [ 6296.246867] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6296.248719] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6296.253944] ? do_wait_intr+0xf0/0xf0 [ 6296.256055] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6296.260405] kthread+0x1d1/0x200 [ 6296.262164] ? set_kthread_struct+0x70/0x70 [ 6296.264061] ret_from_fork+0x1f/0x30 [ 6298.591232] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4339s! [ 6298.595822] Showing busy workqueues and worker pools: [ 6298.606114] workqueue events: flags=0x0 [ 6298.611695] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6298.621829] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 6298.631494] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6298.642805] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 6298.654791] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6298.665079] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 6298.683935] workqueue events_long: flags=0x0 [ 6298.688400] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6298.690706] pending: lu_objects_flush [obdclass] [ 6298.697739] workqueue events_unbound: flags=0x2 [ 6298.704357] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6298.708241] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6298.720100] workqueue events_power_efficient: flags=0x80 [ 6298.723415] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6298.733904] pending: neigh_periodic_work [ 6298.736668] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6298.741788] pending: do_cache_clean, check_lifetime [ 6298.746391] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6298.753074] pending: neigh_periodic_work [ 6298.755520] workqueue writeback: flags=0x4a [ 6298.758681] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6298.761418] pending: wb_workfn, wb_workfn [ 6298.767607] workqueue xprtiod: flags=0xa [ 6298.769857] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6298.773090] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 6298.781637] workqueue ipv6_addrconf: flags=0x40008 [ 6298.783592] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6298.795676] pending: addrconf_verify_work [ 6298.797297] workqueue knbd0-recv: flags=0x1a [ 6298.800524] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6298.807239] in-flight: 657:recv_work [ 6298.812883] workqueue ptlrpc_pinger: flags=0x2 [ 6298.822901] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6298.831567] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 6298.843645] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4339s workers=4 idle: 20831 12642 28859 [ 6298.860712] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6289s workers=2 idle: 659 [ 6300.064590] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 6300.077825] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6300.168712] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6300.178600] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6300.200636] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6300.215411] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6300.259090] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6300.270093] RAX: 0000000000006e09 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 6300.297522] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 6300.305644] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6300.310617] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 6300.319325] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 6300.321172] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 6300.344998] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6300.356380] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 6300.364647] Call Trace: [ 6300.366742] [ 6300.367742] ? show_regs.cold.9+0x22/0x2f [ 6300.374558] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6300.379471] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6300.383439] ? watchdog+0x40/0x40 [ 6300.386011] ? hrtimer_interrupt+0x128/0x2c0 [ 6300.389953] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6300.396611] ? apic_timer_interrupt+0xf/0x20 [ 6300.398444] [ 6300.399074] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6300.403832] do_raw_spin_lock+0x177/0x190 [ 6300.408853] _raw_spin_lock+0x21/0x30 [ 6300.409953] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6300.414070] ? do_raw_spin_unlock+0x75/0x190 [ 6300.415415] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6300.428974] ? do_wait_intr+0xf0/0xf0 [ 6300.432212] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6300.439423] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6300.447088] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6300.452394] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6300.458010] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6300.466634] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6300.470114] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6300.477481] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6300.481912] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6300.488110] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6300.493968] lmv_intent_open+0x338/0x1150 [lmv] [ 6300.498568] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6300.504587] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6300.509175] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6300.514845] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6300.517099] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6300.523185] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6300.527047] ? do_raw_spin_unlock+0x75/0x190 [ 6300.533692] ? _raw_spin_unlock+0x12/0x30 [ 6300.537000] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6300.542159] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6300.548992] ll_atomic_open+0x36a/0xd70 [lustre] [ 6300.553913] lookup_open+0x356/0xb40 [ 6300.557683] do_last+0x313/0xfc0 [ 6300.562106] ? nd_jump_root+0xe5/0x160 [ 6300.566851] ? path_init+0x437/0x520 [ 6300.570202] path_openat+0xf7/0x500 [ 6300.573885] ? do_raw_spin_unlock+0x75/0x190 [ 6300.581695] do_filp_open+0x99/0x140 [ 6300.584136] ? getname_flags+0x6e/0x330 [ 6300.588512] ? __check_object_size+0xff/0x256 [ 6300.595581] ? do_raw_spin_unlock+0x75/0x190 [ 6300.599058] ? _raw_spin_unlock+0x12/0x30 [ 6300.605624] do_sys_openat2+0x2b4/0x410 [ 6300.609083] do_sys_open+0x73/0xa0 [ 6300.613912] __x64_sys_openat+0x24/0x30 [ 6300.618434] do_syscall_64+0xc1/0x440 [ 6300.621952] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6300.627581] RIP: 0033:0x7ff54af45332 [ 6300.631576] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6300.653503] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6300.662579] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 6300.672676] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 6300.679016] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 6300.684774] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 6300.692930] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 6308.059023] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 6308.068545] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6308.126719] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6308.137099] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6308.152405] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6308.164423] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6308.167849] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6308.188693] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6308.200327] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 6308.210878] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 6308.219909] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 6308.222703] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 6308.231621] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 6308.243433] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 6308.250507] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6308.254912] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 6308.262346] Call Trace: [ 6308.267637] [ 6308.268617] ? show_regs.cold.9+0x22/0x2f [ 6308.273597] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6308.280634] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6308.287598] ? watchdog+0x40/0x40 [ 6308.290384] ? hrtimer_interrupt+0x128/0x2c0 [ 6308.295686] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6308.305487] ? apic_timer_interrupt+0xf/0x20 [ 6308.309038] [ 6308.309792] ? native_safe_halt+0x2e/0x50 [ 6308.313984] kvm_wait+0xcd/0xe0 [ 6308.318821] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6308.324512] do_raw_spin_lock+0x177/0x190 [ 6308.328315] _raw_spin_lock+0x21/0x30 [ 6308.330388] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6308.335816] process_one_work+0x2c8/0x700 [ 6308.340583] worker_thread+0x296/0x6e0 [ 6308.344744] ? rescuer_thread+0x570/0x570 [ 6308.346219] kthread+0x1d1/0x200 [ 6308.348213] ? set_kthread_struct+0x70/0x70 [ 6308.352354] ret_from_fork+0x1f/0x30 [ 6320.060145] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [ 6320.067257] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6320.133689] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6320.148419] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6320.161714] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6320.174207] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6320.187366] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6320.193406] RAX: 0000000000006491 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 6320.201365] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 6320.211287] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6320.223801] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 6320.233593] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 6320.239139] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 6320.242543] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6320.251404] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 6320.258502] Call Trace: [ 6320.260438] [ 6320.261016] ? show_regs.cold.9+0x22/0x2f [ 6320.263249] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6320.273828] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6320.278643] ? watchdog+0x40/0x40 [ 6320.286217] ? hrtimer_interrupt+0x128/0x2c0 [ 6320.297977] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6320.299659] ? apic_timer_interrupt+0xf/0x20 [ 6320.311367] [ 6320.313346] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6320.323331] do_raw_spin_lock+0x177/0x190 [ 6320.326418] _raw_spin_lock+0x21/0x30 [ 6320.332335] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6320.334344] ? do_raw_spin_unlock+0x75/0x190 [ 6320.340227] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6320.344172] ? do_wait_intr+0xf0/0xf0 [ 6320.346327] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6320.349460] mdc_close+0x53b/0xd70 [mdc] [ 6320.351468] lmv_close+0x18d/0x490 [lmv] [ 6320.353774] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 6320.360828] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6320.369480] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6320.378587] ll_md_close+0x274/0x800 [lustre] [ 6320.383635] ll_file_release+0x128/0xc50 [lustre] [ 6320.391275] __fput+0x10d/0x440 [ 6320.393250] ____fput+0x12/0x20 [ 6320.394481] task_work_run+0xc6/0x110 [ 6320.399434] exit_to_usermode_loop+0x1dd/0x1f0 [ 6320.404316] do_syscall_64+0x3d6/0x440 [ 6320.406629] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6320.417709] RIP: 0033:0x7ff54af45cd8 [ 6320.422694] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6320.441111] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6320.445277] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 6320.448651] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 6320.454497] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 6320.460664] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 6320.472402] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6324.055030] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 6324.059160] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6324.082371] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6324.089571] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6324.096160] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6324.099474] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6324.110152] RSP: 0000:ffffaf9fc1b97bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6324.120550] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 6324.130461] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 6324.138995] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 6324.146307] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 6324.152532] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6324.162590] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 6324.173476] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6324.177236] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 6324.185423] Call Trace: [ 6324.192134] [ 6324.192794] ? show_regs.cold.9+0x22/0x2f [ 6324.194190] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6324.202197] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6324.207078] ? watchdog+0x40/0x40 [ 6324.212531] ? hrtimer_interrupt+0x128/0x2c0 [ 6324.218060] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6324.224077] ? apic_timer_interrupt+0xf/0x20 [ 6324.229813] [ 6324.232343] ? native_safe_halt+0x2e/0x50 [ 6324.238258] kvm_wait+0xcd/0xe0 [ 6324.241762] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6324.253220] do_raw_spin_lock+0x177/0x190 [ 6324.254912] _raw_spin_lock+0x21/0x30 [ 6324.257683] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6324.264190] obd_mod_free+0x46/0x140 [ptlrpc] [ 6324.268833] mdc_commit_open+0x91/0xb0 [mdc] [ 6324.271737] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6324.274455] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6324.284350] after_reply+0xe2c/0x1780 [ptlrpc] [ 6324.294667] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6324.299324] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6324.301026] ? do_wait_intr+0xf0/0xf0 [ 6324.312567] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6324.324860] kthread+0x1d1/0x200 [ 6324.326056] ? set_kthread_struct+0x70/0x70 [ 6324.329558] ret_from_fork+0x1f/0x30 [ 6328.062290] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 6328.068672] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6328.121416] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6328.131433] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6328.145170] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6328.154458] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6328.169643] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6328.175692] RAX: 00000000000025f2 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 6328.190602] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 6328.211495] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6328.221755] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 6328.227621] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 6328.231837] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 6328.239648] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6328.245565] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 6328.250645] Call Trace: [ 6328.254427] [ 6328.256551] ? show_regs.cold.9+0x22/0x2f [ 6328.263879] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6328.273547] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6328.283495] ? watchdog+0x40/0x40 [ 6328.289784] ? hrtimer_interrupt+0x128/0x2c0 [ 6328.295501] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6328.299102] ? apic_timer_interrupt+0xf/0x20 [ 6328.306556] [ 6328.309222] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6328.320643] do_raw_spin_lock+0x177/0x190 [ 6328.322596] _raw_spin_lock+0x21/0x30 [ 6328.323792] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6328.329138] ? do_raw_spin_unlock+0x75/0x190 [ 6328.334224] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6328.353012] ? do_wait_intr+0xf0/0xf0 [ 6328.357132] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6328.364385] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6328.377616] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6328.380260] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6328.387328] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6328.390773] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6328.394597] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6328.397295] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6328.399674] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6328.402600] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6328.405398] lmv_intent_open+0x338/0x1150 [lmv] [ 6328.408157] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6328.409830] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6328.412299] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6328.414258] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6328.416518] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6328.419354] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6328.421597] ? do_raw_spin_unlock+0x75/0x190 [ 6328.423613] ? _raw_spin_unlock+0x12/0x30 [ 6328.425588] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6328.428121] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6328.434436] ll_atomic_open+0x36a/0xd70 [lustre] [ 6328.437349] lookup_open+0x356/0xb40 [ 6328.439290] do_last+0x313/0xfc0 [ 6328.441775] ? nd_jump_root+0xe5/0x160 [ 6328.444310] ? path_init+0x437/0x520 [ 6328.446655] path_openat+0xf7/0x500 [ 6328.449208] ? do_raw_spin_unlock+0x75/0x190 [ 6328.452458] do_filp_open+0x99/0x140 [ 6328.453688] ? getname_flags+0x6e/0x330 [ 6328.457251] ? __check_object_size+0xff/0x256 [ 6328.458700] ? do_raw_spin_unlock+0x75/0x190 [ 6328.462328] ? _raw_spin_unlock+0x12/0x30 [ 6328.466713] do_sys_openat2+0x2b4/0x410 [ 6328.472190] do_sys_open+0x73/0xa0 [ 6328.475535] __x64_sys_openat+0x24/0x30 [ 6328.477521] do_syscall_64+0xc1/0x440 [ 6328.480136] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6328.489176] RIP: 0033:0x7ff54af45332 [ 6328.493639] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6328.521647] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6328.531567] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 6328.542635] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 6328.553857] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 6328.561573] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 6328.568629] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 6329.311065] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4370s! [ 6329.314757] Showing busy workqueues and worker pools: [ 6329.317825] workqueue events: flags=0x0 [ 6329.322468] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6329.326167] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 6329.329889] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6329.335585] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 6329.341771] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6329.343528] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 6329.346223] workqueue events_long: flags=0x0 [ 6329.348283] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6329.351623] pending: lu_objects_flush [obdclass] [ 6329.354196] workqueue events_unbound: flags=0x2 [ 6329.355587] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6329.358293] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6329.362369] workqueue events_power_efficient: flags=0x80 [ 6329.364134] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6329.366458] pending: neigh_periodic_work [ 6329.368024] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6329.372051] pending: do_cache_clean, check_lifetime [ 6329.374555] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6329.377491] pending: neigh_periodic_work [ 6329.379166] workqueue writeback: flags=0x4a [ 6329.383313] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6329.389457] pending: wb_workfn, wb_workfn [ 6329.392390] workqueue xprtiod: flags=0xa [ 6329.396101] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6329.400688] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 6329.414222] workqueue ipv6_addrconf: flags=0x40008 [ 6329.422314] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6329.433339] pending: addrconf_verify_work [ 6329.434699] workqueue knbd0-recv: flags=0x1a [ 6329.440372] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6329.445565] in-flight: 657:recv_work [ 6329.448885] workqueue ptlrpc_pinger: flags=0x2 [ 6329.453841] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6329.459533] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 6329.468378] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4370s workers=4 idle: 20831 12642 28859 [ 6329.475766] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6319s workers=2 idle: 659 [ 6333.965010] rcu: INFO: rcu_sched self-detected stall on CPU [ 6333.967928] rcu: 0-...!: (2270999 ticks this GP) idle=98a/1/0x4000000000000002 softirq=125154/125154 fqs=959 [ 6333.973338] (t=4380125 jiffies g=253585 q=4281) [ 6333.975935] rcu: rcu_sched kthread starved for 4375085 jiffies! g253585 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [ 6333.981153] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 6333.985680] rcu: RCU grace-period kthread stack dump: [ 6333.988081] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 6333.993168] Call Trace: [ 6333.995036] __schedule+0x351/0xcb0 [ 6333.999929] schedule+0xc0/0x180 [ 6334.004086] schedule_timeout+0xb4/0x190 [ 6334.008199] ? do_raw_spin_unlock+0x75/0x190 [ 6334.012602] ? __next_timer_interrupt+0x160/0x160 [ 6334.017118] ? prepare_to_swait_event+0xf3/0x140 [ 6334.021537] rcu_gp_kthread+0x8dd/0x15b0 [ 6334.025807] ? force_qs_rnp+0x2c0/0x2c0 [ 6334.030747] kthread+0x1d1/0x200 [ 6334.032935] ? set_kthread_struct+0x70/0x70 [ 6334.036365] ret_from_fork+0x1f/0x30 [ 6334.042767] rcu: Stack dump where RCU GP kthread last ran: [ 6334.047466] NMI backtrace for cpu 0 [ 6334.054380] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6334.071176] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6334.082341] Call Trace: [ 6334.084707] [ 6334.089094] ? dump_stack+0xbb/0x10e [ 6334.095367] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 6334.100732] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 6334.105080] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 6334.121056] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 6334.132144] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 6334.137912] ? rcu_check_gp_kthread_starvation+0x16f/0x188 [ 6334.142667] ? rcu_sched_clock_irq.cold.51+0xce/0x307 [ 6334.147927] ? update_process_times+0x6d/0xc0 [ 6334.150781] ? tick_sched_handle.isra.18+0x27/0x90 [ 6334.157049] ? tick_sched_timer+0x53/0xb0 [ 6334.160330] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6334.162407] ? tick_sched_handle.isra.18+0x90/0x90 [ 6334.165152] ? hrtimer_interrupt+0x128/0x2c0 [ 6334.168360] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6334.173124] ? apic_timer_interrupt+0xf/0x20 [ 6334.177153] [ 6334.179627] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6334.183635] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6334.187315] ? do_raw_spin_lock+0x177/0x190 [ 6334.191754] ? _raw_spin_lock+0x21/0x30 [ 6334.193759] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6334.195664] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 6334.201003] ? mdc_commit_open+0x91/0xb0 [mdc] [ 6334.205922] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6334.208598] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6334.220392] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 6334.230424] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6334.237321] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6334.244761] ? do_wait_intr+0xf0/0xf0 [ 6334.246200] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6334.250446] ? kthread+0x1d1/0x200 [ 6334.251663] ? set_kthread_struct+0x70/0x70 [ 6334.253580] ? ret_from_fork+0x1f/0x30 [ 6334.255116] NMI backtrace for cpu 0 [ 6334.256559] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6334.260695] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6334.264571] Call Trace: [ 6334.265575] [ 6334.266440] ? dump_stack+0xbb/0x10e [ 6334.267735] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 6334.269798] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 6334.275297] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 6334.277045] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 6334.278609] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 6334.283679] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 6334.287718] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 6334.302725] ? update_process_times+0x6d/0xc0 [ 6334.314648] ? tick_sched_handle.isra.18+0x27/0x90 [ 6334.317878] ? tick_sched_timer+0x53/0xb0 [ 6334.323328] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6334.332360] ? tick_sched_handle.isra.18+0x90/0x90 [ 6334.337578] ? hrtimer_interrupt+0x128/0x2c0 [ 6334.341739] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6334.345459] ? apic_timer_interrupt+0xf/0x20 [ 6334.346875] [ 6334.347592] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6334.355710] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6334.361213] ? do_raw_spin_lock+0x177/0x190 [ 6334.365267] ? _raw_spin_lock+0x21/0x30 [ 6334.371570] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6334.377738] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 6334.380563] ? mdc_commit_open+0x91/0xb0 [mdc] [ 6334.385707] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6334.391879] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6334.394803] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 6334.397564] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6334.400647] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6334.403266] ? do_wait_intr+0xf0/0xf0 [ 6334.405488] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6334.408301] ? kthread+0x1d1/0x200 [ 6334.411280] ? set_kthread_struct+0x70/0x70 [ 6334.413878] ? ret_from_fork+0x1f/0x30 [ 6336.068532] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 6336.074004] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6336.113330] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6336.118676] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6336.123225] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6336.126390] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6336.128609] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6336.137414] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 6336.141217] RAX: 0000000000006010 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 6336.144636] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 6336.147433] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 6336.151516] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 6336.154807] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 6336.159169] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 6336.162315] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6336.165241] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 6336.169301] Call Trace: [ 6336.171171] [ 6336.172068] ? show_regs.cold.9+0x22/0x2f [ 6336.174110] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6336.175996] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6336.177481] ? watchdog+0x40/0x40 [ 6336.179250] ? hrtimer_interrupt+0x128/0x2c0 [ 6336.181318] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6336.183622] ? apic_timer_interrupt+0xf/0x20 [ 6336.185612] [ 6336.186873] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6336.188372] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6336.189468] do_raw_spin_lock+0x177/0x190 [ 6336.191116] _raw_spin_lock+0x21/0x30 [ 6336.192738] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6336.195988] process_one_work+0x2c8/0x700 [ 6336.197206] worker_thread+0x296/0x6e0 [ 6336.198320] ? rescuer_thread+0x570/0x570 [ 6336.200024] kthread+0x1d1/0x200 [ 6336.201604] ? set_kthread_struct+0x70/0x70 [ 6336.203652] ret_from_fork+0x1f/0x30 [ 6348.058021] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [ 6348.063382] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6348.108302] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6348.119590] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6348.125963] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6348.129211] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6348.144030] RSP: 0018:ffffaf9fc1543ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6348.152325] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 6348.161019] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 6348.165684] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6348.171261] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 6348.177226] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 6348.181538] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 6348.192186] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6348.198359] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 6348.208949] Call Trace: [ 6348.214632] [ 6348.215276] ? show_regs.cold.9+0x22/0x2f [ 6348.221319] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6348.225599] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6348.231278] ? watchdog+0x40/0x40 [ 6348.239195] ? hrtimer_interrupt+0x128/0x2c0 [ 6348.241497] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6348.247702] ? apic_timer_interrupt+0xf/0x20 [ 6348.254252] [ 6348.255986] ? native_safe_halt+0x2e/0x50 [ 6348.260621] kvm_wait+0xcd/0xe0 [ 6348.261750] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6348.263475] do_raw_spin_lock+0x177/0x190 [ 6348.276933] _raw_spin_lock+0x21/0x30 [ 6348.279181] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6348.286076] ? do_raw_spin_unlock+0x75/0x190 [ 6348.289550] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6348.294434] ? do_wait_intr+0xf0/0xf0 [ 6348.298387] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6348.303572] mdc_close+0x53b/0xd70 [mdc] [ 6348.305644] lmv_close+0x18d/0x490 [lmv] [ 6348.309563] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 6348.319085] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6348.324470] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6348.329533] ll_md_close+0x274/0x800 [lustre] [ 6348.334476] ll_file_release+0x128/0xc50 [lustre] [ 6348.339378] __fput+0x10d/0x440 [ 6348.340470] ____fput+0x12/0x20 [ 6348.341595] task_work_run+0xc6/0x110 [ 6348.343890] exit_to_usermode_loop+0x1dd/0x1f0 [ 6348.345471] do_syscall_64+0x3d6/0x440 [ 6348.347542] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6348.352464] RIP: 0033:0x7ff54af45cd8 [ 6348.357672] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6348.387005] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6348.394489] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 6348.402153] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 6348.403750] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 6348.417943] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 6348.422908] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6356.067589] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 6356.074130] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6356.103272] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6356.115304] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6356.131275] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6356.139676] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6356.164464] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6356.172296] RAX: 0000000000007d90 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 6356.177565] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 6356.186837] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6356.195403] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 6356.201274] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 6356.206187] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 6356.211988] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6356.216005] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 6356.220432] Call Trace: [ 6356.224164] [ 6356.226228] ? show_regs.cold.9+0x22/0x2f [ 6356.229139] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6356.232017] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6356.234409] ? watchdog+0x40/0x40 [ 6356.239099] ? hrtimer_interrupt+0x128/0x2c0 [ 6356.244109] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6356.251545] ? apic_timer_interrupt+0xf/0x20 [ 6356.256423] [ 6356.258085] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6356.265047] do_raw_spin_lock+0x177/0x190 [ 6356.270644] _raw_spin_lock+0x21/0x30 [ 6356.272461] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6356.281164] ? do_raw_spin_unlock+0x75/0x190 [ 6356.287295] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6356.290859] ? do_wait_intr+0xf0/0xf0 [ 6356.293138] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6356.302150] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6356.314319] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6356.325414] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6356.332894] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6356.334274] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6356.340896] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6356.342400] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6356.354957] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6356.360939] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6356.365317] lmv_intent_open+0x338/0x1150 [lmv] [ 6356.372450] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6356.376061] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6356.378990] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6356.380367] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6356.386856] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6356.389476] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6356.393871] ? do_raw_spin_unlock+0x75/0x190 [ 6356.395960] ? _raw_spin_unlock+0x12/0x30 [ 6356.398799] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6356.401529] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6356.404000] ll_atomic_open+0x36a/0xd70 [lustre] [ 6356.406243] lookup_open+0x356/0xb40 [ 6356.407972] do_last+0x313/0xfc0 [ 6356.409297] ? nd_jump_root+0xe5/0x160 [ 6356.411218] ? path_init+0x437/0x520 [ 6356.412728] path_openat+0xf7/0x500 [ 6356.414328] ? do_raw_spin_unlock+0x75/0x190 [ 6356.416961] do_filp_open+0x99/0x140 [ 6356.418429] ? getname_flags+0x6e/0x330 [ 6356.420129] ? __check_object_size+0xff/0x256 [ 6356.421948] ? do_raw_spin_unlock+0x75/0x190 [ 6356.423986] ? _raw_spin_unlock+0x12/0x30 [ 6356.425951] do_sys_openat2+0x2b4/0x410 [ 6356.427521] do_sys_open+0x73/0xa0 [ 6356.429259] __x64_sys_openat+0x24/0x30 [ 6356.431161] do_syscall_64+0xc1/0x440 [ 6356.433279] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6356.436099] RIP: 0033:0x7ff54af45332 [ 6356.437325] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6356.444335] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6356.448488] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 6356.452571] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 6356.456264] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 6356.460010] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 6356.463454] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 6359.519326] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4400s! [ 6359.528419] Showing busy workqueues and worker pools: [ 6359.534716] workqueue events: flags=0x0 [ 6359.536618] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6359.546751] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 6359.558403] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6359.564518] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 6359.575996] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6359.580006] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 6359.585016] workqueue events_long: flags=0x0 [ 6359.590947] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6359.596733] pending: lu_objects_flush [obdclass] [ 6359.603637] workqueue events_unbound: flags=0x2 [ 6359.606474] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6359.614609] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6359.626354] workqueue events_power_efficient: flags=0x80 [ 6359.632329] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6359.638654] pending: neigh_periodic_work [ 6359.640303] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6359.655031] pending: do_cache_clean, check_lifetime [ 6359.660436] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6359.667955] pending: neigh_periodic_work [ 6359.672445] workqueue writeback: flags=0x4a [ 6359.674102] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6359.685911] pending: wb_workfn, wb_workfn [ 6359.688385] workqueue xprtiod: flags=0xa [ 6359.691239] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6359.697765] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 6359.707206] workqueue ipv6_addrconf: flags=0x40008 [ 6359.713511] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6359.719983] pending: addrconf_verify_work [ 6359.723310] workqueue knbd0-recv: flags=0x1a [ 6359.733461] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6359.737660] in-flight: 657:recv_work [ 6359.739768] workqueue ptlrpc_pinger: flags=0x2 [ 6359.742412] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6359.745426] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 6359.748414] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4400s workers=4 idle: 20831 12642 28859 [ 6359.752319] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6349s workers=2 idle: 659 [ 6364.059617] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 6364.064189] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6364.116839] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6364.129612] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6364.142558] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6364.150098] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6364.155535] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6364.174213] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6364.178288] RAX: 0000000000000c10 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 6364.183302] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 6364.192390] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 6364.197440] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 6364.224372] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 6364.231399] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 6364.237902] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6364.241385] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 6364.245096] Call Trace: [ 6364.246616] [ 6364.247825] ? show_regs.cold.9+0x22/0x2f [ 6364.250090] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6364.253820] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6364.258558] ? watchdog+0x40/0x40 [ 6364.261648] ? hrtimer_interrupt+0x128/0x2c0 [ 6364.266295] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6364.271716] ? apic_timer_interrupt+0xf/0x20 [ 6364.273537] [ 6364.276825] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6364.282100] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6364.286301] do_raw_spin_lock+0x177/0x190 [ 6364.290711] _raw_spin_lock+0x21/0x30 [ 6364.292487] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6364.295970] process_one_work+0x2c8/0x700 [ 6364.298448] worker_thread+0x296/0x6e0 [ 6364.300270] ? rescuer_thread+0x570/0x570 [ 6364.302380] kthread+0x1d1/0x200 [ 6364.304396] ? set_kthread_struct+0x70/0x70 [ 6364.307073] ret_from_fork+0x1f/0x30 [ 6376.058085] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [ 6376.070925] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6376.132325] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6376.147004] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6376.155904] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6376.167963] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6376.192803] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6376.201409] RAX: 0000000000004659 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 6376.205877] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 6376.209862] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6376.215993] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 6376.225548] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 6376.237253] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 6376.244539] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6376.250412] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 6376.257662] Call Trace: [ 6376.261552] [ 6376.262233] ? show_regs.cold.9+0x22/0x2f [ 6376.265531] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6376.270184] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6376.272522] ? watchdog+0x40/0x40 [ 6376.274357] ? hrtimer_interrupt+0x128/0x2c0 [ 6376.278420] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6376.280020] ? apic_timer_interrupt+0xf/0x20 [ 6376.285835] [ 6376.288617] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6376.292967] do_raw_spin_lock+0x177/0x190 [ 6376.294819] _raw_spin_lock+0x21/0x30 [ 6376.297183] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6376.300891] ? do_raw_spin_unlock+0x75/0x190 [ 6376.303293] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6376.306759] ? do_wait_intr+0xf0/0xf0 [ 6376.308663] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6376.314269] mdc_close+0x53b/0xd70 [mdc] [ 6376.319183] lmv_close+0x18d/0x490 [lmv] [ 6376.323286] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 6376.329863] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6376.336190] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6376.344977] ll_md_close+0x274/0x800 [lustre] [ 6376.352796] ll_file_release+0x128/0xc50 [lustre] [ 6376.356649] __fput+0x10d/0x440 [ 6376.358888] ____fput+0x12/0x20 [ 6376.360848] task_work_run+0xc6/0x110 [ 6376.362962] exit_to_usermode_loop+0x1dd/0x1f0 [ 6376.367887] do_syscall_64+0x3d6/0x440 [ 6376.370595] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6376.372960] RIP: 0033:0x7ff54af45cd8 [ 6376.374902] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6376.383341] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6376.387396] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 6376.391369] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 6376.394971] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 6376.398254] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 6376.401747] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6384.057986] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 6384.061785] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 6384.068802] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6384.068907] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6384.068915] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6384.068918] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6384.068970] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6384.068976] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6384.068980] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 6384.068983] RDX: 00000000000056b7 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 6384.068985] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 6384.068988] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 6384.068991] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6384.068994] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 6384.068997] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6384.069000] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 6384.069009] Call Trace: [ 6384.069024] [ 6384.069031] ? show_regs.cold.9+0x22/0x2f [ 6384.069043] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6384.069060] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6384.069065] ? watchdog+0x40/0x40 [ 6384.069070] ? hrtimer_interrupt+0x128/0x2c0 [ 6384.069075] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6384.069084] ? apic_timer_interrupt+0xf/0x20 [ 6384.069089] [ 6384.069093] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6384.069098] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6384.069103] do_raw_spin_lock+0x177/0x190 [ 6384.069107] _raw_spin_lock+0x21/0x30 [ 6384.069114] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6384.069429] obd_mod_free+0x46/0x140 [ptlrpc] [ 6384.069651] mdc_commit_open+0x91/0xb0 [mdc] [ 6384.069696] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6384.069900] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6384.070125] after_reply+0xe2c/0x1780 [ptlrpc] [ 6384.070329] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6384.070542] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6384.070752] ? do_wait_intr+0xf0/0xf0 [ 6384.070766] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6384.079044] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6384.080058] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6384.080067] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6384.080071] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6384.080089] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6384.080093] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6384.080098] RAX: 00000000000063fd RBX: ffff96848811cae0 RCX: 0000000000000001 [ 6384.080101] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 6384.080103] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6384.080106] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 6384.080109] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 6384.080186] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 6384.080190] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6384.080192] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 6384.080200] Call Trace: [ 6384.080208] [ 6384.080214] ? show_regs.cold.9+0x22/0x2f [ 6384.080227] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6384.080234] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6384.080238] ? watchdog+0x40/0x40 [ 6384.080242] ? hrtimer_interrupt+0x128/0x2c0 [ 6384.080247] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6384.080256] ? apic_timer_interrupt+0xf/0x20 [ 6384.080261] [ 6384.080264] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6384.080272] do_raw_spin_lock+0x177/0x190 [ 6384.080277] _raw_spin_lock+0x21/0x30 [ 6384.080282] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6384.081026] ? do_raw_spin_unlock+0x75/0x190 [ 6384.081033] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6384.118490] kthread+0x1d1/0x200 [ 6384.118523] ? set_kthread_struct+0x70/0x70 [ 6384.118528] ret_from_fork+0x1f/0x30 [ 6384.434762] ? do_wait_intr+0xf0/0xf0 [ 6384.436431] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6384.438781] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6384.441596] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6384.450039] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6384.455036] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6384.459427] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6384.463573] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6384.468759] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6384.472138] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6384.474995] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6384.477420] lmv_intent_open+0x338/0x1150 [lmv] [ 6384.479548] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6384.484827] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6384.488776] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6384.490668] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6384.492732] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6384.495105] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6384.499533] ? do_raw_spin_unlock+0x75/0x190 [ 6384.503076] ? _raw_spin_unlock+0x12/0x30 [ 6384.505738] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6384.511123] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6384.513038] ll_atomic_open+0x36a/0xd70 [lustre] [ 6384.516701] lookup_open+0x356/0xb40 [ 6384.519570] do_last+0x313/0xfc0 [ 6384.521070] ? nd_jump_root+0xe5/0x160 [ 6384.522638] ? path_init+0x437/0x520 [ 6384.524229] path_openat+0xf7/0x500 [ 6384.527596] ? do_raw_spin_unlock+0x75/0x190 [ 6384.529435] do_filp_open+0x99/0x140 [ 6384.534642] ? getname_flags+0x6e/0x330 [ 6384.536320] ? __check_object_size+0xff/0x256 [ 6384.541141] ? do_raw_spin_unlock+0x75/0x190 [ 6384.544214] ? _raw_spin_unlock+0x12/0x30 [ 6384.546030] do_sys_openat2+0x2b4/0x410 [ 6384.549714] do_sys_open+0x73/0xa0 [ 6384.551200] __x64_sys_openat+0x24/0x30 [ 6384.555090] do_syscall_64+0xc1/0x440 [ 6384.558405] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6384.564949] RIP: 0033:0x7ff54af45332 [ 6384.567189] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6384.584512] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6384.587804] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 6384.593121] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 6384.599006] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 6384.605479] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 6384.610167] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 6390.239048] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4431s! [ 6390.242739] Showing busy workqueues and worker pools: [ 6390.245828] workqueue events: flags=0x0 [ 6390.248135] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6390.250891] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 6390.254636] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6390.259531] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 6390.263884] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6390.274452] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 6390.278581] workqueue events_long: flags=0x0 [ 6390.282595] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6390.287476] pending: lu_objects_flush [obdclass] [ 6390.289701] workqueue events_unbound: flags=0x2 [ 6390.291988] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6390.294093] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6390.299638] workqueue events_power_efficient: flags=0x80 [ 6390.301519] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6390.304395] pending: neigh_periodic_work [ 6390.305888] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6390.308218] pending: do_cache_clean, check_lifetime [ 6390.310105] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6390.313033] pending: neigh_periodic_work [ 6390.314691] workqueue writeback: flags=0x4a [ 6390.315855] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6390.317791] pending: wb_workfn, wb_workfn [ 6390.319457] workqueue xprtiod: flags=0xa [ 6390.321012] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6390.322858] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 6390.325489] workqueue ipv6_addrconf: flags=0x40008 [ 6390.330518] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6390.332984] pending: addrconf_verify_work [ 6390.334582] workqueue knbd0-recv: flags=0x1a [ 6390.337728] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6390.345777] in-flight: 657:recv_work [ 6390.354592] workqueue ptlrpc_pinger: flags=0x2 [ 6390.357015] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6390.367307] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 6390.369331] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4431s workers=4 idle: 20831 12642 28859 [ 6390.377525] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6380s workers=2 idle: 659 [ 6392.059049] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 6392.061644] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6392.078985] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6392.084779] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6392.088360] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6392.093534] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6392.100728] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6392.114266] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6392.120035] RAX: 000000000000689a RBX: ffff96848811cae0 RCX: 0000000000000001 [ 6392.123627] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 6392.129875] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 6392.133263] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 6392.138117] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 6392.142940] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 6392.148621] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6392.151606] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 6392.154530] Call Trace: [ 6392.155626] [ 6392.156312] ? show_regs.cold.9+0x22/0x2f [ 6392.164245] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6392.169719] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6392.171768] ? watchdog+0x40/0x40 [ 6392.175762] ? hrtimer_interrupt+0x128/0x2c0 [ 6392.178112] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6392.180338] ? apic_timer_interrupt+0xf/0x20 [ 6392.182028] [ 6392.183187] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6392.186652] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6392.190548] do_raw_spin_lock+0x177/0x190 [ 6392.192111] _raw_spin_lock+0x21/0x30 [ 6392.193744] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6392.196053] process_one_work+0x2c8/0x700 [ 6392.197738] worker_thread+0x296/0x6e0 [ 6392.199096] ? rescuer_thread+0x570/0x570 [ 6392.200763] kthread+0x1d1/0x200 [ 6392.204814] ? set_kthread_struct+0x70/0x70 [ 6392.206623] ret_from_fork+0x1f/0x30 [ 6404.066808] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [ 6404.070914] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6404.147700] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6404.158534] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6404.166816] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6404.170301] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6404.187731] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6404.194865] RAX: 000000000000020d RBX: ffff96848811cae0 RCX: 0000000000000001 [ 6404.197488] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 6404.201494] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6404.206821] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 6404.212571] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 6404.219923] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 6404.223892] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6404.226888] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 6404.229805] Call Trace: [ 6404.231341] [ 6404.232395] ? show_regs.cold.9+0x22/0x2f [ 6404.237788] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6404.240357] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6404.242511] ? watchdog+0x40/0x40 [ 6404.244452] ? hrtimer_interrupt+0x128/0x2c0 [ 6404.249258] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6404.251725] ? apic_timer_interrupt+0xf/0x20 [ 6404.253904] [ 6404.254751] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6404.257452] do_raw_spin_lock+0x177/0x190 [ 6404.259633] _raw_spin_lock+0x21/0x30 [ 6404.262518] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6404.266367] ? do_raw_spin_unlock+0x75/0x190 [ 6404.268775] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6404.270848] ? do_wait_intr+0xf0/0xf0 [ 6404.272521] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6404.275294] mdc_close+0x53b/0xd70 [mdc] [ 6404.276807] lmv_close+0x18d/0x490 [lmv] [ 6404.280298] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 6404.285094] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6404.287588] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6404.292974] ll_md_close+0x274/0x800 [lustre] [ 6404.294796] ll_file_release+0x128/0xc50 [lustre] [ 6404.297800] __fput+0x10d/0x440 [ 6404.299250] ____fput+0x12/0x20 [ 6404.300251] task_work_run+0xc6/0x110 [ 6404.308723] exit_to_usermode_loop+0x1dd/0x1f0 [ 6404.311986] do_syscall_64+0x3d6/0x440 [ 6404.318470] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6404.321864] RIP: 0033:0x7ff54af45cd8 [ 6404.326934] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6404.339981] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6404.343871] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 6404.350667] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 6404.358873] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 6404.368293] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 6404.372597] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6412.060373] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 6412.070407] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6412.092089] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6412.098460] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6412.101703] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6412.103831] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6412.112920] RSP: 0000:ffffaf9fc15234f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6412.116958] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 6412.120478] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 6412.123831] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6412.127351] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 6412.131103] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 6412.134713] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 6412.138907] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6412.141818] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 6412.145482] Call Trace: [ 6412.146405] [ 6412.147147] ? show_regs.cold.9+0x22/0x2f [ 6412.149059] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6412.151394] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6412.153740] ? watchdog+0x40/0x40 [ 6412.155192] ? hrtimer_interrupt+0x128/0x2c0 [ 6412.157164] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6412.158801] ? apic_timer_interrupt+0xf/0x20 [ 6412.160946] [ 6412.161876] ? native_safe_halt+0x2e/0x50 [ 6412.164010] kvm_wait+0xcd/0xe0 [ 6412.165290] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6412.168175] do_raw_spin_lock+0x177/0x190 [ 6412.170075] _raw_spin_lock+0x21/0x30 [ 6412.171789] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6412.174491] ? do_raw_spin_unlock+0x75/0x190 [ 6412.176422] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6412.178749] ? do_wait_intr+0xf0/0xf0 [ 6412.180252] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6412.182889] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6412.185202] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6412.188111] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6412.190205] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6412.193128] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6412.195650] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6412.197727] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6412.200287] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6412.204202] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6412.207301] lmv_intent_open+0x338/0x1150 [lmv] [ 6412.209472] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6412.211991] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6412.214953] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6412.218083] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6412.220427] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6412.223074] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6412.224992] ? do_raw_spin_unlock+0x75/0x190 [ 6412.227362] ? _raw_spin_unlock+0x12/0x30 [ 6412.229904] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6412.232643] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6412.235132] ll_atomic_open+0x36a/0xd70 [lustre] [ 6412.237553] lookup_open+0x356/0xb40 [ 6412.239393] do_last+0x313/0xfc0 [ 6412.241339] ? nd_jump_root+0xe5/0x160 [ 6412.243276] ? path_init+0x437/0x520 [ 6412.246205] path_openat+0xf7/0x500 [ 6412.248129] ? do_raw_spin_unlock+0x75/0x190 [ 6412.250514] do_filp_open+0x99/0x140 [ 6412.252961] ? getname_flags+0x6e/0x330 [ 6412.255862] ? __check_object_size+0xff/0x256 [ 6412.259615] ? do_raw_spin_unlock+0x75/0x190 [ 6412.262150] ? _raw_spin_unlock+0x12/0x30 [ 6412.264515] do_sys_openat2+0x2b4/0x410 [ 6412.266377] do_sys_open+0x73/0xa0 [ 6412.267816] __x64_sys_openat+0x24/0x30 [ 6412.270001] do_syscall_64+0xc1/0x440 [ 6412.272224] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6412.275003] RIP: 0033:0x7ff54af45332 [ 6412.277350] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6412.287108] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6412.292189] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 6412.296286] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 6412.300779] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 6412.305123] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 6412.309173] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 6416.055728] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 6416.062543] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6416.108833] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6416.125438] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6416.138303] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6416.146492] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6416.163370] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6416.168069] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 6416.178173] RDX: 00000000000007d8 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 6416.186185] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 6416.194963] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 6416.208390] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6416.217676] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 6416.230377] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6416.236818] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 6416.249770] Call Trace: [ 6416.255231] [ 6416.256498] ? show_regs.cold.9+0x22/0x2f [ 6416.263481] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6416.270450] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6416.272051] ? watchdog+0x40/0x40 [ 6416.285156] ? hrtimer_interrupt+0x128/0x2c0 [ 6416.291934] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6416.296989] ? apic_timer_interrupt+0xf/0x20 [ 6416.300314] [ 6416.302651] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6416.315426] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6416.317364] do_raw_spin_lock+0x177/0x190 [ 6416.326459] _raw_spin_lock+0x21/0x30 [ 6416.331789] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6416.335723] obd_mod_free+0x46/0x140 [ptlrpc] [ 6416.344595] mdc_commit_open+0x91/0xb0 [mdc] [ 6416.351892] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6416.353915] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6416.368376] after_reply+0xe2c/0x1780 [ptlrpc] [ 6416.376270] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6416.385142] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6416.393221] ? do_wait_intr+0xf0/0xf0 [ 6416.397697] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6416.401460] kthread+0x1d1/0x200 [ 6416.404211] ? set_kthread_struct+0x70/0x70 [ 6416.407678] ret_from_fork+0x1f/0x30 [ 6420.059379] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 6420.073957] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6420.110867] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6420.116563] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6420.122612] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6420.134664] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6420.148134] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6420.161326] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6420.168849] RAX: 0000000000005b7e RBX: ffff96848811cae0 RCX: 0000000000000001 [ 6420.171640] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 6420.175510] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 6420.178318] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 6420.185446] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 6420.188588] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 6420.192940] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6420.194763] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 6420.200647] Call Trace: [ 6420.201867] [ 6420.202718] ? show_regs.cold.9+0x22/0x2f [ 6420.209638] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6420.212291] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6420.216502] ? watchdog+0x40/0x40 [ 6420.222650] ? hrtimer_interrupt+0x128/0x2c0 [ 6420.225811] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6420.228697] ? apic_timer_interrupt+0xf/0x20 [ 6420.230626] [ 6420.232858] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6420.234600] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6420.244418] do_raw_spin_lock+0x177/0x190 [ 6420.245637] _raw_spin_lock+0x21/0x30 [ 6420.246785] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6420.252464] process_one_work+0x2c8/0x700 [ 6420.254168] worker_thread+0x296/0x6e0 [ 6420.255728] ? rescuer_thread+0x570/0x570 [ 6420.257005] kthread+0x1d1/0x200 [ 6420.258735] ? set_kthread_struct+0x70/0x70 [ 6420.260064] ret_from_fork+0x1f/0x30 [ 6420.447125] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4461s! [ 6420.457553] Showing busy workqueues and worker pools: [ 6420.464538] workqueue events: flags=0x0 [ 6420.465926] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6420.477355] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 6420.495892] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6420.503962] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 6420.517414] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6420.525453] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 6420.533537] workqueue events_long: flags=0x0 [ 6420.536308] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6420.553490] pending: lu_objects_flush [obdclass] [ 6420.558632] workqueue events_unbound: flags=0x2 [ 6420.562508] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6420.566515] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6420.580418] workqueue events_power_efficient: flags=0x80 [ 6420.588081] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6420.600499] pending: neigh_periodic_work [ 6420.607106] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6420.617469] pending: do_cache_clean, check_lifetime [ 6420.624843] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6420.635338] pending: neigh_periodic_work [ 6420.637850] workqueue writeback: flags=0x4a [ 6420.641133] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6420.650479] pending: wb_workfn, wb_workfn [ 6420.655017] workqueue xprtiod: flags=0xa [ 6420.663144] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6420.670628] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 6420.679791] workqueue ipv6_addrconf: flags=0x40008 [ 6420.689354] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6420.695140] pending: addrconf_verify_work [ 6420.703250] workqueue knbd0-recv: flags=0x1a [ 6420.709416] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6420.718340] in-flight: 657:recv_work [ 6420.724308] workqueue ptlrpc_pinger: flags=0x2 [ 6420.729281] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6420.735764] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 6420.739858] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4461s workers=4 idle: 20831 12642 28859 [ 6420.750167] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6410s workers=2 idle: 659 [ 6432.058075] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [ 6432.065554] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6432.101520] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6432.121175] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6432.128713] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6432.134701] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6432.145415] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6432.151490] RAX: 0000000000004d9d RBX: ffff96848811cae0 RCX: 0000000000000001 [ 6432.153971] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 6432.161259] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6432.169170] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 6432.174731] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 6432.181151] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 6432.188012] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6432.190749] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 6432.200750] Call Trace: [ 6432.202781] [ 6432.205168] ? show_regs.cold.9+0x22/0x2f [ 6432.208777] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6432.210828] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6432.212539] ? watchdog+0x40/0x40 [ 6432.213624] ? hrtimer_interrupt+0x128/0x2c0 [ 6432.217392] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6432.219631] ? apic_timer_interrupt+0xf/0x20 [ 6432.224325] [ 6432.226780] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6432.230582] do_raw_spin_lock+0x177/0x190 [ 6432.237002] _raw_spin_lock+0x21/0x30 [ 6432.238600] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6432.244122] ? do_raw_spin_unlock+0x75/0x190 [ 6432.245404] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6432.250018] ? do_wait_intr+0xf0/0xf0 [ 6432.255259] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6432.259656] mdc_close+0x53b/0xd70 [mdc] [ 6432.264788] lmv_close+0x18d/0x490 [lmv] [ 6432.266714] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 6432.273125] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6432.279447] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6432.285633] ll_md_close+0x274/0x800 [lustre] [ 6432.288965] ll_file_release+0x128/0xc50 [lustre] [ 6432.291741] __fput+0x10d/0x440 [ 6432.297653] ____fput+0x12/0x20 [ 6432.304622] task_work_run+0xc6/0x110 [ 6432.307345] exit_to_usermode_loop+0x1dd/0x1f0 [ 6432.310142] do_syscall_64+0x3d6/0x440 [ 6432.312141] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6432.321449] RIP: 0033:0x7ff54af45cd8 [ 6432.324153] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6432.339349] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6432.347759] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 6432.351427] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 6432.357054] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 6432.364804] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 6432.373773] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6440.060630] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 6440.064726] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6440.089650] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6440.093471] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6440.095915] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6440.097940] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6440.105274] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6440.108378] RAX: 0000000000006c72 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 6440.110723] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 6440.113367] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6440.116213] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 6440.119267] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 6440.121679] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 6440.125631] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6440.128987] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 6440.132327] Call Trace: [ 6440.133523] [ 6440.134649] ? show_regs.cold.9+0x22/0x2f [ 6440.136561] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6440.138195] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6440.140673] ? watchdog+0x40/0x40 [ 6440.142530] ? hrtimer_interrupt+0x128/0x2c0 [ 6440.144425] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6440.146290] ? apic_timer_interrupt+0xf/0x20 [ 6440.147721] [ 6440.148863] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6440.151103] do_raw_spin_lock+0x177/0x190 [ 6440.153079] _raw_spin_lock+0x21/0x30 [ 6440.154636] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6440.157164] ? do_raw_spin_unlock+0x75/0x190 [ 6440.159069] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6440.161380] ? do_wait_intr+0xf0/0xf0 [ 6440.162741] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6440.169483] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6440.172622] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6440.176325] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6440.178290] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6440.181266] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6440.182896] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6440.185012] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6440.186793] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6440.189322] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6440.191320] lmv_intent_open+0x338/0x1150 [lmv] [ 6440.193140] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6440.195117] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6440.197282] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6440.198918] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6440.200794] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6440.202755] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6440.204798] ? do_raw_spin_unlock+0x75/0x190 [ 6440.206274] ? _raw_spin_unlock+0x12/0x30 [ 6440.207893] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6440.209640] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6440.211477] ll_atomic_open+0x36a/0xd70 [lustre] [ 6440.213091] lookup_open+0x356/0xb40 [ 6440.214436] do_last+0x313/0xfc0 [ 6440.215543] ? nd_jump_root+0xe5/0x160 [ 6440.216750] ? path_init+0x437/0x520 [ 6440.218516] path_openat+0xf7/0x500 [ 6440.219590] ? do_raw_spin_unlock+0x75/0x190 [ 6440.222888] do_filp_open+0x99/0x140 [ 6440.223773] ? getname_flags+0x6e/0x330 [ 6440.224719] ? __check_object_size+0xff/0x256 [ 6440.225850] ? do_raw_spin_unlock+0x75/0x190 [ 6440.228119] ? _raw_spin_unlock+0x12/0x30 [ 6440.229410] do_sys_openat2+0x2b4/0x410 [ 6440.231171] do_sys_open+0x73/0xa0 [ 6440.232101] __x64_sys_openat+0x24/0x30 [ 6440.233113] do_syscall_64+0xc1/0x440 [ 6440.234310] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6440.236282] RIP: 0033:0x7ff54af45332 [ 6440.237768] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6440.246432] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6440.252496] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 6440.261886] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 6440.270908] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 6440.273347] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 6440.286933] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 6448.057296] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [ 6448.067463] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 6448.072120] Modules linked in: [ 6448.088040] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6448.088124] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6448.088132] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6448.088139] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6448.089276] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6448.089307] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6448.089311] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 6448.089316] RAX: 0000000000007f87 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 6448.089319] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 6448.089322] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 6448.089324] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 6448.089326] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 6448.089330] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 6448.089333] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6448.089335] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 6448.089343] Call Trace: [ 6448.089349] [ 6448.089355] ? show_regs.cold.9+0x22/0x2f [ 6448.089365] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6448.089372] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6448.089376] ? watchdog+0x40/0x40 [ 6448.089380] ? hrtimer_interrupt+0x128/0x2c0 [ 6448.089385] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6448.089394] ? apic_timer_interrupt+0xf/0x20 [ 6448.089398] [ 6448.089401] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6448.089407] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6448.089412] do_raw_spin_lock+0x177/0x190 [ 6448.089418] _raw_spin_lock+0x21/0x30 [ 6448.089424] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6448.095017] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) [ 6448.124116] process_one_work+0x2c8/0x700 [ 6448.124135] worker_thread+0x296/0x6e0 [ 6448.124138] ? rescuer_thread+0x570/0x570 [ 6448.124141] kthread+0x1d1/0x200 [ 6448.124145] ? set_kthread_struct+0x70/0x70 [ 6448.124147] ret_from_fork+0x1f/0x30 [ 6448.334335] ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6448.370777] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6448.377056] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6448.383187] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6448.388608] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6448.406636] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6448.413163] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 6448.421459] RDX: 0000000000006c08 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 6448.427563] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 6448.436991] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 6448.445973] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6448.452016] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 6448.454659] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6448.464166] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 6448.467865] Call Trace: [ 6448.470683] [ 6448.476526] ? show_regs.cold.9+0x22/0x2f [ 6448.480110] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6448.485484] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6448.487779] ? watchdog+0x40/0x40 [ 6448.488979] ? hrtimer_interrupt+0x128/0x2c0 [ 6448.495091] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6448.501644] ? apic_timer_interrupt+0xf/0x20 [ 6448.504598] [ 6448.505571] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6448.511907] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6448.513759] do_raw_spin_lock+0x177/0x190 [ 6448.519742] _raw_spin_lock+0x21/0x30 [ 6448.522243] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6448.533313] obd_mod_free+0x46/0x140 [ptlrpc] [ 6448.537642] mdc_commit_open+0x91/0xb0 [mdc] [ 6448.539253] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6448.552239] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6448.562672] after_reply+0xe2c/0x1780 [ptlrpc] [ 6448.568025] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6448.575479] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6448.577130] ? do_wait_intr+0xf0/0xf0 [ 6448.584446] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6448.595179] kthread+0x1d1/0x200 [ 6448.599477] ? set_kthread_struct+0x70/0x70 [ 6448.604978] ret_from_fork+0x1f/0x30 [ 6451.167087] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4492s! [ 6451.172907] Showing busy workqueues and worker pools: [ 6451.175609] workqueue events: flags=0x0 [ 6451.177582] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6451.181604] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 6451.186656] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6451.190682] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 6451.197464] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6451.205888] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 6451.209583] workqueue events_long: flags=0x0 [ 6451.213484] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6451.219435] pending: lu_objects_flush [obdclass] [ 6451.223996] workqueue events_unbound: flags=0x2 [ 6451.225960] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6451.228902] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6451.233793] workqueue events_power_efficient: flags=0x80 [ 6451.236580] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6451.242294] pending: neigh_periodic_work [ 6451.250175] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6451.257372] pending: do_cache_clean, check_lifetime [ 6451.261636] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6451.273810] pending: neigh_periodic_work [ 6451.279202] workqueue writeback: flags=0x4a [ 6451.281703] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6451.292669] pending: wb_workfn, wb_workfn [ 6451.301562] workqueue xprtiod: flags=0xa [ 6451.307840] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6451.317281] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 6451.328507] workqueue ipv6_addrconf: flags=0x40008 [ 6451.339875] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6451.351301] pending: addrconf_verify_work [ 6451.356909] workqueue knbd0-recv: flags=0x1a [ 6451.362222] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6451.367639] in-flight: 657:recv_work [ 6451.372954] workqueue ptlrpc_pinger: flags=0x2 [ 6451.377694] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6451.387322] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 6451.393241] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4492s workers=4 idle: 20831 12642 28859 [ 6451.407897] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6441s workers=2 idle: 659 [ 6460.063806] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 6460.069131] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6460.114974] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6460.129089] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6460.138954] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6460.142220] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6460.166135] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6460.178146] RAX: 0000000000007ea2 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 6460.183223] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 6460.188207] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6460.194870] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 6460.200857] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 6460.208417] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 6460.215425] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6460.220019] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 6460.233770] Call Trace: [ 6460.234814] [ 6460.235637] ? show_regs.cold.9+0x22/0x2f [ 6460.240109] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6460.247117] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6460.252912] ? watchdog+0x40/0x40 [ 6460.257065] ? hrtimer_interrupt+0x128/0x2c0 [ 6460.261895] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6460.264987] ? apic_timer_interrupt+0xf/0x20 [ 6460.269980] [ 6460.274822] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6460.279023] do_raw_spin_lock+0x177/0x190 [ 6460.281699] _raw_spin_lock+0x21/0x30 [ 6460.284089] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6460.291139] ? do_raw_spin_unlock+0x75/0x190 [ 6460.294973] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6460.299839] ? do_wait_intr+0xf0/0xf0 [ 6460.302036] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6460.307880] mdc_close+0x53b/0xd70 [mdc] [ 6460.314710] lmv_close+0x18d/0x490 [lmv] [ 6460.317378] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 6460.321989] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6460.326779] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6460.332106] ll_md_close+0x274/0x800 [lustre] [ 6460.336090] ll_file_release+0x128/0xc50 [lustre] [ 6460.339122] __fput+0x10d/0x440 [ 6460.343168] ____fput+0x12/0x20 [ 6460.346875] task_work_run+0xc6/0x110 [ 6460.348936] exit_to_usermode_loop+0x1dd/0x1f0 [ 6460.358992] do_syscall_64+0x3d6/0x440 [ 6460.362113] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6460.371087] RIP: 0033:0x7ff54af45cd8 [ 6460.375705] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6460.391296] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6460.397068] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 6460.406638] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 6460.417025] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 6460.425779] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 6460.432016] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6468.061242] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 6468.065864] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6468.096305] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6468.101346] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6468.104327] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6468.106842] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6468.118378] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6468.120819] RAX: 000000000000527f RBX: ffff96848811cae0 RCX: 0000000000000001 [ 6468.123372] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 6468.125248] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6468.127338] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 6468.130329] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 6468.133681] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 6468.137412] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6468.139764] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 6468.143589] Call Trace: [ 6468.144931] [ 6468.145978] ? show_regs.cold.9+0x22/0x2f [ 6468.148958] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6468.151418] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6468.153999] ? watchdog+0x40/0x40 [ 6468.155786] ? hrtimer_interrupt+0x128/0x2c0 [ 6468.158208] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6468.160833] ? apic_timer_interrupt+0xf/0x20 [ 6468.162970] [ 6468.164065] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6468.166699] do_raw_spin_lock+0x177/0x190 [ 6468.169053] _raw_spin_lock+0x21/0x30 [ 6468.170303] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6468.175260] ? do_raw_spin_unlock+0x75/0x190 [ 6468.177886] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6468.181059] ? do_wait_intr+0xf0/0xf0 [ 6468.182832] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6468.186236] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6468.187952] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6468.190292] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6468.192326] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6468.194458] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6468.196338] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6468.197871] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6468.199853] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6468.202376] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6468.204626] lmv_intent_open+0x338/0x1150 [lmv] [ 6468.206258] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6468.208422] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6468.210831] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6468.213260] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6468.214821] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6468.219330] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6468.224223] ? do_raw_spin_unlock+0x75/0x190 [ 6468.226951] ? _raw_spin_unlock+0x12/0x30 [ 6468.228271] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6468.231451] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6468.234012] ll_atomic_open+0x36a/0xd70 [lustre] [ 6468.236394] lookup_open+0x356/0xb40 [ 6468.238178] do_last+0x313/0xfc0 [ 6468.239774] ? nd_jump_root+0xe5/0x160 [ 6468.241885] ? path_init+0x437/0x520 [ 6468.243741] path_openat+0xf7/0x500 [ 6468.245391] ? do_raw_spin_unlock+0x75/0x190 [ 6468.248662] do_filp_open+0x99/0x140 [ 6468.250697] ? getname_flags+0x6e/0x330 [ 6468.254304] ? __check_object_size+0xff/0x256 [ 6468.256746] ? do_raw_spin_unlock+0x75/0x190 [ 6468.258670] ? _raw_spin_unlock+0x12/0x30 [ 6468.260802] do_sys_openat2+0x2b4/0x410 [ 6468.262738] do_sys_open+0x73/0xa0 [ 6468.264330] __x64_sys_openat+0x24/0x30 [ 6468.266419] do_syscall_64+0xc1/0x440 [ 6468.269209] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6468.273303] RIP: 0033:0x7ff54af45332 [ 6468.275179] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6468.287301] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6468.292164] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 6468.298137] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 6468.301901] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 6468.306264] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 6468.312569] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 6476.055313] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [ 6476.059133] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 6476.061941] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6476.062346] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6476.062353] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6476.062356] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6476.062369] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6476.062374] RSP: 0000:ffffaf9fc1b97bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6476.062378] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 6476.062381] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 6476.062383] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 6476.062386] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 6476.062388] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6476.062391] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 6476.062394] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6476.062396] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 6476.062404] Call Trace: [ 6476.062410] [ 6476.062416] ? show_regs.cold.9+0x22/0x2f [ 6476.062427] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6476.062432] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6476.062497] ? watchdog+0x40/0x40 [ 6476.062501] ? hrtimer_interrupt+0x128/0x2c0 [ 6476.062505] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6476.062512] ? apic_timer_interrupt+0xf/0x20 [ 6476.062517] [ 6476.062520] ? native_safe_halt+0x2e/0x50 [ 6476.062524] kvm_wait+0xcd/0xe0 [ 6476.062532] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6476.062541] do_raw_spin_lock+0x177/0x190 [ 6476.062546] _raw_spin_lock+0x21/0x30 [ 6476.062552] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6476.063208] obd_mod_free+0x46/0x140 [ptlrpc] [ 6476.063334] mdc_commit_open+0x91/0xb0 [mdc] [ 6476.063362] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6476.067969] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6476.068413] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6476.068421] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6476.068428] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6476.068806] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6476.068816] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6476.068821] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6476.068825] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 6476.068828] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 6476.068830] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 6476.068832] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 6476.068834] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 6476.068837] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 6476.068840] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6476.068842] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 6476.068852] Call Trace: [ 6476.068859] [ 6476.068865] ? show_regs.cold.9+0x22/0x2f [ 6476.068875] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6476.068883] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6476.068887] ? watchdog+0x40/0x40 [ 6476.068891] ? hrtimer_interrupt+0x128/0x2c0 [ 6476.068896] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6476.068902] ? apic_timer_interrupt+0xf/0x20 [ 6476.068907] [ 6476.068910] ? native_safe_halt+0x2e/0x50 [ 6476.068914] kvm_wait+0xcd/0xe0 [ 6476.068921] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6476.068929] do_raw_spin_lock+0x177/0x190 [ 6476.068934] _raw_spin_lock+0x21/0x30 [ 6476.068938] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6476.069277] process_one_work+0x2c8/0x700 [ 6476.069285] worker_thread+0x296/0x6e0 [ 6476.069289] ? rescuer_thread+0x570/0x570 [ 6476.069293] kthread+0x1d1/0x200 [ 6476.069297] ? set_kthread_struct+0x70/0x70 [ 6476.069301] ret_from_fork+0x1f/0x30 [ 6476.379801] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6476.382807] after_reply+0xe2c/0x1780 [ptlrpc] [ 6476.385894] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6476.388485] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6476.390975] ? do_wait_intr+0xf0/0xf0 [ 6476.392923] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6476.395421] kthread+0x1d1/0x200 [ 6476.396585] ? set_kthread_struct+0x70/0x70 [ 6476.398771] ret_from_fork+0x1f/0x30 [ 6481.889090] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4523s! [ 6481.895751] Showing busy workqueues and worker pools: [ 6481.898050] workqueue events: flags=0x0 [ 6481.905072] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6481.913766] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 6481.921086] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6481.931794] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 6481.949760] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6481.955403] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 6481.960225] workqueue events_long: flags=0x0 [ 6481.962622] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6481.976873] pending: lu_objects_flush [obdclass] [ 6481.985612] workqueue events_unbound: flags=0x2 [ 6481.990391] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6481.996668] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6482.005931] workqueue events_power_efficient: flags=0x80 [ 6482.013345] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6482.019895] pending: neigh_periodic_work [ 6482.021861] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6482.031756] pending: do_cache_clean, check_lifetime [ 6482.038058] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6482.046878] pending: neigh_periodic_work [ 6482.048450] workqueue writeback: flags=0x4a [ 6482.055300] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6482.058130] pending: wb_workfn, wb_workfn [ 6482.062463] workqueue xprtiod: flags=0xa [ 6482.065758] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6482.071903] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 6482.075913] workqueue ipv6_addrconf: flags=0x40008 [ 6482.077837] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6482.097501] pending: addrconf_verify_work [ 6482.099115] workqueue knbd0-recv: flags=0x1a [ 6482.106265] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6482.108468] in-flight: 657:recv_work [ 6482.120593] workqueue ptlrpc_pinger: flags=0x2 [ 6482.131371] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6482.133525] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 6482.142516] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4523s workers=4 idle: 20831 12642 28859 [ 6482.154284] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6472s workers=2 idle: 659 [ 6488.058114] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 6488.062680] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6488.108168] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6488.128868] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6488.145125] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6488.151055] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6488.169717] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6488.173776] RAX: 00000000000006e8 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 6488.177920] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 6488.181613] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6488.185465] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 6488.189766] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 6488.193851] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 6488.200984] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6488.207812] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 6488.221306] Call Trace: [ 6488.224446] [ 6488.226821] ? show_regs.cold.9+0x22/0x2f [ 6488.233142] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6488.241898] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6488.251967] ? watchdog+0x40/0x40 [ 6488.258715] ? hrtimer_interrupt+0x128/0x2c0 [ 6488.265012] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6488.272568] ? apic_timer_interrupt+0xf/0x20 [ 6488.279604] [ 6488.281805] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6488.291273] do_raw_spin_lock+0x177/0x190 [ 6488.297053] _raw_spin_lock+0x21/0x30 [ 6488.301738] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6488.308665] ? do_raw_spin_unlock+0x75/0x190 [ 6488.313818] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6488.321142] ? do_wait_intr+0xf0/0xf0 [ 6488.327778] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6488.334775] mdc_close+0x53b/0xd70 [mdc] [ 6488.337601] lmv_close+0x18d/0x490 [lmv] [ 6488.341560] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 6488.347810] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6488.353874] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6488.358248] ll_md_close+0x274/0x800 [lustre] [ 6488.361007] ll_file_release+0x128/0xc50 [lustre] [ 6488.363924] __fput+0x10d/0x440 [ 6488.367572] ____fput+0x12/0x20 [ 6488.369834] task_work_run+0xc6/0x110 [ 6488.373138] exit_to_usermode_loop+0x1dd/0x1f0 [ 6488.376604] do_syscall_64+0x3d6/0x440 [ 6488.380531] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6488.382929] RIP: 0033:0x7ff54af45cd8 [ 6488.385140] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6488.402737] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6488.408951] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 6488.413146] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 6488.416889] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 6488.420611] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 6488.424331] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6496.062170] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 6496.066099] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6496.095495] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6496.116642] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6496.120489] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6496.123145] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6496.131575] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6496.135149] RAX: 00000000000040fc RBX: ffff96848811cae0 RCX: 0000000000000001 [ 6496.138810] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 6496.142101] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6496.145169] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 6496.148212] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 6496.151095] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 6496.154804] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6496.157312] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 6496.160548] Call Trace: [ 6496.161657] [ 6496.162583] ? show_regs.cold.9+0x22/0x2f [ 6496.164462] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6496.166766] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6496.170669] ? watchdog+0x40/0x40 [ 6496.172110] ? hrtimer_interrupt+0x128/0x2c0 [ 6496.177834] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6496.181743] ? apic_timer_interrupt+0xf/0x20 [ 6496.185921] [ 6496.188460] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6496.192896] do_raw_spin_lock+0x177/0x190 [ 6496.196719] _raw_spin_lock+0x21/0x30 [ 6496.199106] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6496.201660] ? do_raw_spin_unlock+0x75/0x190 [ 6496.203321] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6496.206664] ? do_wait_intr+0xf0/0xf0 [ 6496.210304] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6496.215307] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6496.220153] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6496.224279] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6496.227018] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6496.230667] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6496.236484] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6496.238534] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6496.241971] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6496.251508] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6496.257758] lmv_intent_open+0x338/0x1150 [lmv] [ 6496.261138] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6496.264384] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6496.266658] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6496.270196] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6496.272768] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6496.275654] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6496.282906] ? do_raw_spin_unlock+0x75/0x190 [ 6496.287226] ? _raw_spin_unlock+0x12/0x30 [ 6496.292610] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6496.295504] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6496.297205] ll_atomic_open+0x36a/0xd70 [lustre] [ 6496.301637] lookup_open+0x356/0xb40 [ 6496.305854] do_last+0x313/0xfc0 [ 6496.308206] ? nd_jump_root+0xe5/0x160 [ 6496.311133] ? path_init+0x437/0x520 [ 6496.313194] path_openat+0xf7/0x500 [ 6496.315167] ? do_raw_spin_unlock+0x75/0x190 [ 6496.317082] do_filp_open+0x99/0x140 [ 6496.318747] ? getname_flags+0x6e/0x330 [ 6496.320700] ? __check_object_size+0xff/0x256 [ 6496.323174] ? do_raw_spin_unlock+0x75/0x190 [ 6496.324860] ? _raw_spin_unlock+0x12/0x30 [ 6496.326794] do_sys_openat2+0x2b4/0x410 [ 6496.328702] do_sys_open+0x73/0xa0 [ 6496.330362] __x64_sys_openat+0x24/0x30 [ 6496.332369] do_syscall_64+0xc1/0x440 [ 6496.333561] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6496.335809] RIP: 0033:0x7ff54af45332 [ 6496.337707] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6496.348179] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6496.355687] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 6496.362632] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 6496.368121] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 6496.375189] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 6496.381933] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 6504.059054] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 6504.067847] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6504.108498] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6504.115729] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6504.119928] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6504.126581] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6504.131662] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6504.143109] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 6504.146118] RAX: 0000000000001d56 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 6504.149190] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 6504.153552] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 6504.159641] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 6504.161880] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 6504.165350] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 6504.170094] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6504.172582] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 6504.176867] Call Trace: [ 6504.177919] [ 6504.178863] ? show_regs.cold.9+0x22/0x2f [ 6504.181823] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6504.185446] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6504.188572] ? watchdog+0x40/0x40 [ 6504.189775] ? hrtimer_interrupt+0x128/0x2c0 [ 6504.192381] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6504.194280] ? apic_timer_interrupt+0xf/0x20 [ 6504.196420] [ 6504.198222] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6504.201427] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6504.204384] do_raw_spin_lock+0x177/0x190 [ 6504.206145] _raw_spin_lock+0x21/0x30 [ 6504.210633] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6504.214887] process_one_work+0x2c8/0x700 [ 6504.217237] worker_thread+0x296/0x6e0 [ 6504.220541] ? rescuer_thread+0x570/0x570 [ 6504.222715] kthread+0x1d1/0x200 [ 6504.224097] ? set_kthread_struct+0x70/0x70 [ 6504.226942] ret_from_fork+0x1f/0x30 [ 6508.055200] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [ 6508.066325] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6508.106111] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6508.113712] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6508.117555] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6508.123835] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6508.148719] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6508.159571] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 6508.167279] RDX: 0000000000004056 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 6508.177793] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 6508.190177] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 6508.200765] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6508.208563] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 6508.216020] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6508.219940] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 6508.230688] Call Trace: [ 6508.233205] [ 6508.233898] ? show_regs.cold.9+0x22/0x2f [ 6508.236817] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6508.240061] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6508.242815] ? watchdog+0x40/0x40 [ 6508.245619] ? hrtimer_interrupt+0x128/0x2c0 [ 6508.248815] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6508.252502] ? apic_timer_interrupt+0xf/0x20 [ 6508.255558] [ 6508.256964] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6508.260884] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6508.270634] do_raw_spin_lock+0x177/0x190 [ 6508.276541] _raw_spin_lock+0x21/0x30 [ 6508.278416] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6508.282867] obd_mod_free+0x46/0x140 [ptlrpc] [ 6508.286006] mdc_commit_open+0x91/0xb0 [mdc] [ 6508.291292] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6508.301342] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6508.307773] after_reply+0xe2c/0x1780 [ptlrpc] [ 6508.317521] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6508.321541] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6508.329313] ? do_wait_intr+0xf0/0xf0 [ 6508.331670] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6508.334877] kthread+0x1d1/0x200 [ 6508.337835] ? set_kthread_struct+0x70/0x70 [ 6508.340014] ret_from_fork+0x1f/0x30 [ 6512.607044] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4553s! [ 6512.611499] Showing busy workqueues and worker pools: [ 6512.613938] workqueue events: flags=0x0 [ 6512.615839] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6512.619912] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 6512.625516] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6512.630615] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 6512.635416] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6512.638621] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 6512.644528] workqueue events_long: flags=0x0 [ 6512.651505] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6512.655037] pending: lu_objects_flush [obdclass] [ 6512.657380] workqueue events_unbound: flags=0x2 [ 6512.659286] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6512.661771] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6512.666298] workqueue events_power_efficient: flags=0x80 [ 6512.670637] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6512.673010] pending: neigh_periodic_work [ 6512.675603] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6512.680395] pending: do_cache_clean, check_lifetime [ 6512.687037] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6512.692240] pending: neigh_periodic_work [ 6512.697406] workqueue writeback: flags=0x4a [ 6512.699829] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6512.706240] pending: wb_workfn, wb_workfn [ 6512.707848] workqueue xprtiod: flags=0xa [ 6512.712524] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6512.715685] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 6512.726795] workqueue ipv6_addrconf: flags=0x40008 [ 6512.729654] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6512.735665] pending: addrconf_verify_work [ 6512.737926] workqueue knbd0-recv: flags=0x1a [ 6512.743887] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6512.751817] in-flight: 657:recv_work [ 6512.754625] workqueue ptlrpc_pinger: flags=0x2 [ 6512.759648] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6512.761838] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 6512.766943] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4553s workers=4 idle: 20831 12642 28859 [ 6512.769514] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6502s workers=2 idle: 659 [ 6513.976505] rcu: INFO: rcu_sched self-detected stall on CPU [ 6513.980017] rcu: 0-...!: (2365349 ticks this GP) idle=98a/1/0x4000000000000002 softirq=125154/125154 fqs=959 [ 6513.987324] (t=4560136 jiffies g=253585 q=4313) [ 6513.990507] rcu: rcu_sched kthread starved for 4555096 jiffies! g253585 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [ 6513.995714] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 6514.001084] rcu: RCU grace-period kthread stack dump: [ 6514.003141] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 6514.007199] Call Trace: [ 6514.008002] __schedule+0x351/0xcb0 [ 6514.009756] schedule+0xc0/0x180 [ 6514.011292] schedule_timeout+0xb4/0x190 [ 6514.013237] ? do_raw_spin_unlock+0x75/0x190 [ 6514.014769] ? __next_timer_interrupt+0x160/0x160 [ 6514.018151] ? prepare_to_swait_event+0xf3/0x140 [ 6514.022691] rcu_gp_kthread+0x8dd/0x15b0 [ 6514.024978] ? force_qs_rnp+0x2c0/0x2c0 [ 6514.028775] kthread+0x1d1/0x200 [ 6514.030183] ? set_kthread_struct+0x70/0x70 [ 6514.034199] ret_from_fork+0x1f/0x30 [ 6514.037726] rcu: Stack dump where RCU GP kthread last ran: [ 6514.042970] Sending NMI from CPU 0 to CPUs 1: [ 6514.058452] INFO: NMI handler (nmi_cpu_backtrace_handler) took too long to run: 11.460 msecs [ 6514.059017] NMI backtrace for cpu 1 [ 6514.059021] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6514.059023] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6514.059025] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6514.059027] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6514.059028] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 [ 6514.059031] RAX: 0000000000007666 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 6514.059032] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 6514.059033] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6514.059035] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 6514.059036] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 6514.059037] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 6514.059038] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6514.059039] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 6514.059040] Call Trace: [ 6514.059041] [ 6514.059042] ? show_regs.cold.9+0x22/0x2f [ 6514.059043] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 6514.059044] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 6514.059045] ? nmi_handle+0xbd/0x210 [ 6514.059046] ? default_do_nmi+0x4b/0x1b0 [ 6514.059047] ? do_nmi+0x280/0x380 [ 6514.059049] ? end_repeat_nmi+0x16/0x1f [ 6514.059050] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6514.059051] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6514.059052] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6514.059054] [ 6514.059054] do_raw_spin_lock+0x177/0x190 [ 6514.059056] _raw_spin_lock+0x21/0x30 [ 6514.059056] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6514.059058] ? do_raw_spin_unlock+0x75/0x190 [ 6514.059059] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6514.059060] ? do_wait_intr+0xf0/0xf0 [ 6514.059061] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6514.059062] mdc_close+0x53b/0xd70 [mdc] [ 6514.059063] lmv_close+0x18d/0x490 [lmv] [ 6514.059064] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 6514.059065] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6514.059066] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6514.059067] ll_md_close+0x274/0x800 [lustre] [ 6514.059068] ll_file_release+0x128/0xc50 [lustre] [ 6514.059069] __fput+0x10d/0x440 [ 6514.059070] ____fput+0x12/0x20 [ 6514.059071] task_work_run+0xc6/0x110 [ 6514.059072] exit_to_usermode_loop+0x1dd/0x1f0 [ 6514.059073] do_syscall_64+0x3d6/0x440 [ 6514.059074] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6514.059075] RIP: 0033:0x7ff54af45cd8 [ 6514.059076] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6514.059078] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6514.059081] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 6514.059082] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 6514.059084] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 6514.059086] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 6514.059087] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6514.059092] NMI backtrace for cpu 0 [ 6514.283015] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6514.287806] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6514.291820] Call Trace: [ 6514.292738] [ 6514.295573] ? dump_stack+0xbb/0x10e [ 6514.297045] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 6514.300698] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 6514.303345] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 6514.309668] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 6514.313056] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 6514.314771] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 6514.320833] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 6514.325672] ? update_process_times+0x6d/0xc0 [ 6514.329664] ? tick_sched_handle.isra.18+0x27/0x90 [ 6514.335026] ? tick_sched_timer+0x53/0xb0 [ 6514.341760] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6514.347571] ? tick_sched_handle.isra.18+0x90/0x90 [ 6514.349784] ? hrtimer_interrupt+0x128/0x2c0 [ 6514.352281] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6514.355661] ? apic_timer_interrupt+0xf/0x20 [ 6514.356990] [ 6514.357682] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6514.361835] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6514.367543] ? do_raw_spin_lock+0x177/0x190 [ 6514.370574] ? _raw_spin_lock+0x21/0x30 [ 6514.376356] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6514.377744] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 6514.378984] ? mdc_commit_open+0x91/0xb0 [mdc] [ 6514.384613] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6514.387469] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6514.390739] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 6514.393447] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6514.398565] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6514.403440] ? do_wait_intr+0xf0/0xf0 [ 6514.412628] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6514.417537] ? kthread+0x1d1/0x200 [ 6514.419716] ? set_kthread_struct+0x70/0x70 [ 6514.425209] ? ret_from_fork+0x1f/0x30 [ 6524.061441] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [ 6524.066211] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6524.097498] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6524.106166] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6524.111140] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6524.117302] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6524.132961] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6524.137938] RAX: 0000000000007c0b RBX: ffff96848811cae0 RCX: 0000000000000001 [ 6524.149455] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 6524.152668] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6524.155700] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 6524.159305] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 6524.164801] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 6524.169735] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6524.172076] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 6524.175415] Call Trace: [ 6524.176407] [ 6524.177101] ? show_regs.cold.9+0x22/0x2f [ 6524.178397] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6524.179759] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6524.181310] ? watchdog+0x40/0x40 [ 6524.183075] ? hrtimer_interrupt+0x128/0x2c0 [ 6524.184465] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6524.187126] ? apic_timer_interrupt+0xf/0x20 [ 6524.190230] [ 6524.191316] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6524.194317] do_raw_spin_lock+0x177/0x190 [ 6524.196887] _raw_spin_lock+0x21/0x30 [ 6524.198888] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6524.202130] ? do_raw_spin_unlock+0x75/0x190 [ 6524.204503] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6524.211568] ? do_wait_intr+0xf0/0xf0 [ 6524.212802] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6524.222069] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6524.226304] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6524.231137] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6524.236011] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6524.240453] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6524.244180] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6524.248110] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6524.252156] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6524.258532] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6524.264343] lmv_intent_open+0x338/0x1150 [lmv] [ 6524.268420] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6524.271298] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6524.274060] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6524.280384] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6524.284465] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6524.287346] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6524.291772] ? do_raw_spin_unlock+0x75/0x190 [ 6524.294086] ? _raw_spin_unlock+0x12/0x30 [ 6524.299291] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6524.302814] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6524.306087] ll_atomic_open+0x36a/0xd70 [lustre] [ 6524.308170] lookup_open+0x356/0xb40 [ 6524.309906] do_last+0x313/0xfc0 [ 6524.315036] ? nd_jump_root+0xe5/0x160 [ 6524.318020] ? path_init+0x437/0x520 [ 6524.318872] path_openat+0xf7/0x500 [ 6524.319819] ? do_raw_spin_unlock+0x75/0x190 [ 6524.320995] do_filp_open+0x99/0x140 [ 6524.322407] ? getname_flags+0x6e/0x330 [ 6524.323763] ? __check_object_size+0xff/0x256 [ 6524.332133] ? do_raw_spin_unlock+0x75/0x190 [ 6524.338330] ? _raw_spin_unlock+0x12/0x30 [ 6524.340300] do_sys_openat2+0x2b4/0x410 [ 6524.346752] do_sys_open+0x73/0xa0 [ 6524.347964] __x64_sys_openat+0x24/0x30 [ 6524.358908] do_syscall_64+0xc1/0x440 [ 6524.360856] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6524.366059] RIP: 0033:0x7ff54af45332 [ 6524.367292] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6524.382943] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6524.388676] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 6524.393358] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 6524.398329] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 6524.403514] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 6524.406710] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 6532.059033] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 6532.070671] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6532.107434] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6532.112784] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6532.116647] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6532.126808] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6532.129724] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6532.147134] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6532.153838] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 6532.166055] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 6532.176315] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 6532.186668] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 6532.198549] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 6532.207330] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 6532.219019] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6532.223303] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 6532.230875] Call Trace: [ 6532.232238] [ 6532.233159] ? show_regs.cold.9+0x22/0x2f [ 6532.235228] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6532.237547] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6532.239938] ? watchdog+0x40/0x40 [ 6532.240827] ? hrtimer_interrupt+0x128/0x2c0 [ 6532.242941] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6532.245444] ? apic_timer_interrupt+0xf/0x20 [ 6532.247130] [ 6532.248011] ? native_safe_halt+0x2e/0x50 [ 6532.249769] kvm_wait+0xcd/0xe0 [ 6532.251318] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6532.253629] do_raw_spin_lock+0x177/0x190 [ 6532.255465] _raw_spin_lock+0x21/0x30 [ 6532.257223] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6532.259979] process_one_work+0x2c8/0x700 [ 6532.264440] worker_thread+0x296/0x6e0 [ 6532.266640] ? rescuer_thread+0x570/0x570 [ 6532.267563] kthread+0x1d1/0x200 [ 6532.271951] ? set_kthread_struct+0x70/0x70 [ 6532.276514] ret_from_fork+0x1f/0x30 [ 6540.058265] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 6540.058973] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 6540.064580] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata [ 6540.076544] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) [ 6540.097787] dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6540.097810] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6540.097818] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6540.097821] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6540.097834] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6540.097838] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6540.097843] RAX: 0000000000003090 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 6540.097845] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 6540.097848] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6540.097851] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 6540.097853] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 6540.097857] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 6540.097861] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6540.097874] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 6540.097882] Call Trace: [ 6540.097888] [ 6540.097894] ? show_regs.cold.9+0x22/0x2f [ 6540.097905] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6540.097912] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6540.097917] ? watchdog+0x40/0x40 [ 6540.097921] ? hrtimer_interrupt+0x128/0x2c0 [ 6540.097925] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6540.097934] ? apic_timer_interrupt+0xf/0x20 [ 6540.097938] [ 6540.097941] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6540.097949] do_raw_spin_lock+0x177/0x190 [ 6540.097953] _raw_spin_lock+0x21/0x30 [ 6540.097958] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6540.098202] ? do_raw_spin_unlock+0x75/0x190 [ 6540.098206] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6540.098370] ? do_wait_intr+0xf0/0xf0 [ 6540.098378] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6540.098574] mdc_close+0x53b/0xd70 [mdc] [ 6540.098616] lmv_close+0x18d/0x490 [lmv] [ 6540.098645] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 6540.098809] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6540.098925] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6540.099018] ll_md_close+0x274/0x800 [lustre] [ 6540.108412] mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6540.108487] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6540.108494] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6540.108498] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6540.108511] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6540.108516] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6540.108520] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 6540.108523] RDX: 00000000000023a4 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 6540.108525] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 6540.108528] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 6540.108531] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6540.108534] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 6540.108537] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6540.108540] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 6540.108548] Call Trace: [ 6540.108553] [ 6540.108559] ? show_regs.cold.9+0x22/0x2f [ 6540.108570] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6540.108577] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6540.108581] ? watchdog+0x40/0x40 [ 6540.108586] ? hrtimer_interrupt+0x128/0x2c0 [ 6540.108590] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6540.108600] ? apic_timer_interrupt+0xf/0x20 [ 6540.108604] [ 6540.108607] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6540.108612] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6540.108617] do_raw_spin_lock+0x177/0x190 [ 6540.108622] _raw_spin_lock+0x21/0x30 [ 6540.108626] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6540.109662] obd_mod_free+0x46/0x140 [ptlrpc] [ 6540.109980] mdc_commit_open+0x91/0xb0 [mdc] [ 6540.110092] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6540.110429] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6540.110626] after_reply+0xe2c/0x1780 [ptlrpc] [ 6540.120383] ll_file_release+0x128/0xc50 [lustre] [ 6540.120568] __fput+0x10d/0x440 [ 6540.120579] ____fput+0x12/0x20 [ 6540.120584] task_work_run+0xc6/0x110 [ 6540.120593] exit_to_usermode_loop+0x1dd/0x1f0 [ 6540.120601] do_syscall_64+0x3d6/0x440 [ 6540.120606] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6540.120615] RIP: 0033:0x7ff54af45cd8 [ 6540.120624] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6540.120628] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6540.120633] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 6540.120636] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 6540.120638] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 6540.120640] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 6540.120642] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6540.559560] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6540.573165] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6540.578243] ? do_wait_intr+0xf0/0xf0 [ 6540.579368] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6540.585214] kthread+0x1d1/0x200 [ 6540.587132] ? set_kthread_struct+0x70/0x70 [ 6540.591682] ret_from_fork+0x1f/0x30 [ 6542.818259] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4583s! [ 6542.829767] Showing busy workqueues and worker pools: [ 6542.837600] workqueue events: flags=0x0 [ 6542.841266] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6542.846928] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 6542.856523] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6542.866407] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 6542.873458] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6542.882900] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 6542.889071] workqueue events_long: flags=0x0 [ 6542.899005] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6542.915253] pending: lu_objects_flush [obdclass] [ 6542.921977] workqueue events_unbound: flags=0x2 [ 6542.927699] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6542.937502] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6542.949344] workqueue events_power_efficient: flags=0x80 [ 6542.956720] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6542.961292] pending: neigh_periodic_work [ 6542.963902] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6542.973317] pending: do_cache_clean, check_lifetime [ 6542.980373] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6542.986234] pending: neigh_periodic_work [ 6542.988602] workqueue writeback: flags=0x4a [ 6542.991228] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6542.996071] pending: wb_workfn, wb_workfn [ 6542.999486] workqueue xprtiod: flags=0xa [ 6543.001949] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6543.006091] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 6543.010771] workqueue ipv6_addrconf: flags=0x40008 [ 6543.013538] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6543.017729] pending: addrconf_verify_work [ 6543.019295] workqueue knbd0-recv: flags=0x1a [ 6543.021552] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6543.023885] in-flight: 657:recv_work [ 6543.025933] workqueue ptlrpc_pinger: flags=0x2 [ 6543.030082] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6543.033574] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 6543.038108] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4584s workers=4 idle: 20831 12642 28859 [ 6543.043330] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6533s workers=2 idle: 659 [ 6552.060193] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [ 6552.066514] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6552.098719] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6552.104299] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6552.108630] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6552.112009] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6552.121932] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 6552.125948] RAX: 000000000000520e RBX: ffff96848811cae0 RCX: 0000000000000001 [ 6552.129309] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 6552.132470] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6552.137332] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 6552.139884] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 6552.143459] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 6552.148775] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6552.152317] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 6552.156314] Call Trace: [ 6552.157975] [ 6552.159129] ? show_regs.cold.9+0x22/0x2f [ 6552.161799] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6552.165857] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6552.168380] ? watchdog+0x40/0x40 [ 6552.170845] ? hrtimer_interrupt+0x128/0x2c0 [ 6552.173907] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6552.178342] ? apic_timer_interrupt+0xf/0x20 [ 6552.181043] [ 6552.182526] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6552.188312] do_raw_spin_lock+0x177/0x190 [ 6552.191310] _raw_spin_lock+0x21/0x30 [ 6552.195275] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6552.199328] ? do_raw_spin_unlock+0x75/0x190 [ 6552.202445] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6552.207632] ? do_wait_intr+0xf0/0xf0 [ 6552.210717] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6552.214976] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6552.218782] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6552.224739] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6552.228011] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6552.231750] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6552.235362] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6552.239927] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6552.246041] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6552.252050] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6552.258190] lmv_intent_open+0x338/0x1150 [lmv] [ 6552.262252] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6552.266612] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6552.272388] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6552.276376] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6552.280341] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6552.284261] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6552.288944] ? do_raw_spin_unlock+0x75/0x190 [ 6552.292588] ? _raw_spin_unlock+0x12/0x30 [ 6552.295112] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6552.298601] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6552.302565] ll_atomic_open+0x36a/0xd70 [lustre] [ 6552.306797] lookup_open+0x356/0xb40 [ 6552.309277] do_last+0x313/0xfc0 [ 6552.311813] ? nd_jump_root+0xe5/0x160 [ 6552.313967] ? path_init+0x437/0x520 [ 6552.316958] path_openat+0xf7/0x500 [ 6552.319305] ? do_raw_spin_unlock+0x75/0x190 [ 6552.322897] do_filp_open+0x99/0x140 [ 6552.325419] ? getname_flags+0x6e/0x330 [ 6552.328212] ? __check_object_size+0xff/0x256 [ 6552.331464] ? do_raw_spin_unlock+0x75/0x190 [ 6552.334479] ? _raw_spin_unlock+0x12/0x30 [ 6552.338652] do_sys_openat2+0x2b4/0x410 [ 6552.342972] do_sys_open+0x73/0xa0 [ 6552.345277] __x64_sys_openat+0x24/0x30 [ 6552.347991] do_syscall_64+0xc1/0x440 [ 6552.350918] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6552.354950] RIP: 0033:0x7ff54af45332 [ 6552.359969] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6552.378756] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6552.383649] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 6552.388558] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 6552.393072] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 6552.397607] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 6552.401985] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 6560.059023] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 6560.061645] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6560.082596] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6560.088456] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6560.093453] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6560.097391] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6560.099872] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6560.109439] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6560.113259] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 6560.116870] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 6560.120698] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 6560.124397] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 6560.128139] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 6560.131472] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 6560.136088] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6560.140846] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 6560.144981] Call Trace: [ 6560.146427] [ 6560.147431] ? show_regs.cold.9+0x22/0x2f [ 6560.148970] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6560.151731] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6560.153970] ? watchdog+0x40/0x40 [ 6560.155576] ? hrtimer_interrupt+0x128/0x2c0 [ 6560.157810] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6560.160296] ? apic_timer_interrupt+0xf/0x20 [ 6560.161888] [ 6560.162955] ? native_safe_halt+0x2e/0x50 [ 6560.165170] kvm_wait+0xcd/0xe0 [ 6560.166938] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6560.170519] do_raw_spin_lock+0x177/0x190 [ 6560.173439] _raw_spin_lock+0x21/0x30 [ 6560.175345] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6560.178604] process_one_work+0x2c8/0x700 [ 6560.180871] worker_thread+0x296/0x6e0 [ 6560.182760] ? rescuer_thread+0x570/0x570 [ 6560.184786] kthread+0x1d1/0x200 [ 6560.186660] ? set_kthread_struct+0x70/0x70 [ 6560.189185] ret_from_fork+0x1f/0x30 [ 6568.056907] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 6568.061826] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 6568.063104] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6568.063195] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6568.063203] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6568.063206] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6568.063220] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6568.063223] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6568.063228] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 6568.063231] RDX: 00000000000061ea RSI: 0000000000000003 RDI: ffff96848811cae0 [ 6568.063233] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 6568.063235] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 6568.063238] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6568.063241] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 6568.063244] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6568.063246] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 6568.063253] Call Trace: [ 6568.063259] [ 6568.063265] ? show_regs.cold.9+0x22/0x2f [ 6568.063274] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6568.063281] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6568.063285] ? watchdog+0x40/0x40 [ 6568.063289] ? hrtimer_interrupt+0x128/0x2c0 [ 6568.063294] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6568.063303] ? apic_timer_interrupt+0xf/0x20 [ 6568.063309] [ 6568.063312] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6568.063317] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6568.063322] do_raw_spin_lock+0x177/0x190 [ 6568.063327] _raw_spin_lock+0x21/0x30 [ 6568.063332] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6568.065950] obd_mod_free+0x46/0x140 [ptlrpc] [ 6568.067616] mdc_commit_open+0x91/0xb0 [mdc] [ 6568.067653] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6568.067900] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6568.068091] after_reply+0xe2c/0x1780 [ptlrpc] [ 6568.068302] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6568.068499] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6568.068707] ? do_wait_intr+0xf0/0xf0 [ 6568.068727] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6568.068919] kthread+0x1d1/0x200 [ 6568.068926] ? set_kthread_struct+0x70/0x70 [ 6568.068930] ret_from_fork+0x1f/0x30 [ 6568.411246] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6568.436301] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6568.442178] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6568.450413] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6568.453396] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6568.468454] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6568.477041] RAX: 0000000000002913 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 6568.487275] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 6568.494735] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6568.497322] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 6568.502457] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 6568.507410] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 6568.516131] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6568.520351] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 6568.522525] Call Trace: [ 6568.526873] [ 6568.528065] ? show_regs.cold.9+0x22/0x2f [ 6568.532377] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6568.537390] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6568.545667] ? watchdog+0x40/0x40 [ 6568.550668] ? hrtimer_interrupt+0x128/0x2c0 [ 6568.556167] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6568.560365] ? apic_timer_interrupt+0xf/0x20 [ 6568.568026] [ 6568.568973] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6568.574251] do_raw_spin_lock+0x177/0x190 [ 6568.577687] _raw_spin_lock+0x21/0x30 [ 6568.578977] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6568.593531] ? do_raw_spin_unlock+0x75/0x190 [ 6568.599997] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6568.606038] ? do_wait_intr+0xf0/0xf0 [ 6568.607991] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6568.612474] mdc_close+0x53b/0xd70 [mdc] [ 6568.617315] lmv_close+0x18d/0x490 [lmv] [ 6568.620881] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 6568.629600] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6568.640889] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6568.642703] ll_md_close+0x274/0x800 [lustre] [ 6568.647102] ll_file_release+0x128/0xc50 [lustre] [ 6568.650918] __fput+0x10d/0x440 [ 6568.652451] ____fput+0x12/0x20 [ 6568.654708] task_work_run+0xc6/0x110 [ 6568.657006] exit_to_usermode_loop+0x1dd/0x1f0 [ 6568.658901] do_syscall_64+0x3d6/0x440 [ 6568.662617] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6568.666863] RIP: 0033:0x7ff54af45cd8 [ 6568.669175] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6568.683699] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6568.693360] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 6568.706416] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 6568.712379] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 6568.719352] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 6568.724974] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6573.536970] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4614s! [ 6573.550426] Showing busy workqueues and worker pools: [ 6573.556328] workqueue events: flags=0x0 [ 6573.560102] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6573.567188] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 6573.574742] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6573.585212] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 6573.595741] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6573.602634] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 6573.608197] workqueue events_long: flags=0x0 [ 6573.612248] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6573.622697] pending: lu_objects_flush [obdclass] [ 6573.628609] workqueue events_unbound: flags=0x2 [ 6573.630486] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6573.640562] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6573.648141] workqueue events_power_efficient: flags=0x80 [ 6573.654871] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6573.660838] pending: neigh_periodic_work [ 6573.662301] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6573.672949] pending: do_cache_clean, check_lifetime [ 6573.677076] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6573.683195] pending: neigh_periodic_work [ 6573.687860] workqueue writeback: flags=0x4a [ 6573.689793] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6573.692706] pending: wb_workfn, wb_workfn [ 6573.695248] workqueue xprtiod: flags=0xa [ 6573.696977] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6573.699713] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 6573.704157] workqueue ipv6_addrconf: flags=0x40008 [ 6573.706881] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6573.711043] pending: addrconf_verify_work [ 6573.713826] workqueue knbd0-recv: flags=0x1a [ 6573.716915] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6573.720202] in-flight: 657:recv_work [ 6573.723013] workqueue ptlrpc_pinger: flags=0x2 [ 6573.725193] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6573.730424] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 6573.733188] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4614s workers=4 idle: 20831 12642 28859 [ 6573.740373] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6563s workers=2 idle: 659 [ 6580.060023] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [ 6580.062992] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6580.081094] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6580.085380] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6580.088928] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6580.091275] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6580.098355] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6580.101756] RAX: 000000000000004d RBX: ffff96848811cae0 RCX: 0000000000000001 [ 6580.104313] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 6580.107311] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6580.110841] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 6580.112730] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 6580.115076] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 6580.118082] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6580.120402] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 6580.123916] Call Trace: [ 6580.124769] [ 6580.125427] ? show_regs.cold.9+0x22/0x2f [ 6580.127084] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6580.128794] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6580.130765] ? watchdog+0x40/0x40 [ 6580.132040] ? hrtimer_interrupt+0x128/0x2c0 [ 6580.133721] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6580.135533] ? apic_timer_interrupt+0xf/0x20 [ 6580.137432] [ 6580.138202] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6580.140020] do_raw_spin_lock+0x177/0x190 [ 6580.141955] _raw_spin_lock+0x21/0x30 [ 6580.143778] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6580.146161] ? do_raw_spin_unlock+0x75/0x190 [ 6580.148119] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6580.150885] ? do_wait_intr+0xf0/0xf0 [ 6580.152608] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6580.155239] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6580.157860] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6580.160545] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6580.163287] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6580.165684] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6580.167594] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6580.169705] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6580.172204] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6580.175175] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6580.177663] lmv_intent_open+0x338/0x1150 [lmv] [ 6580.179519] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6580.181773] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6580.184331] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6580.186164] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6580.188189] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6580.190509] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6580.192529] ? do_raw_spin_unlock+0x75/0x190 [ 6580.194061] ? _raw_spin_unlock+0x12/0x30 [ 6580.195716] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6580.197940] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6580.200217] ll_atomic_open+0x36a/0xd70 [lustre] [ 6580.202395] lookup_open+0x356/0xb40 [ 6580.203988] do_last+0x313/0xfc0 [ 6580.205367] ? nd_jump_root+0xe5/0x160 [ 6580.207014] ? path_init+0x437/0x520 [ 6580.208443] path_openat+0xf7/0x500 [ 6580.209905] ? do_raw_spin_unlock+0x75/0x190 [ 6580.212081] do_filp_open+0x99/0x140 [ 6580.213326] ? getname_flags+0x6e/0x330 [ 6580.215183] ? __check_object_size+0xff/0x256 [ 6580.216836] ? do_raw_spin_unlock+0x75/0x190 [ 6580.218661] ? _raw_spin_unlock+0x12/0x30 [ 6580.219942] do_sys_openat2+0x2b4/0x410 [ 6580.221351] do_sys_open+0x73/0xa0 [ 6580.222870] __x64_sys_openat+0x24/0x30 [ 6580.224747] do_syscall_64+0xc1/0x440 [ 6580.226320] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6580.228229] RIP: 0033:0x7ff54af45332 [ 6580.229797] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6580.237066] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6580.240046] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 6580.243005] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 6580.245757] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 6580.248718] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 6580.251637] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 6588.066197] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 6588.076816] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6588.128326] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6588.136862] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6588.149888] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6588.157700] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6588.167637] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6588.184331] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 6588.193748] RAX: 0000000000007e0c RBX: ffff96848811cae0 RCX: 0000000000000001 [ 6588.197141] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 6588.201696] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 6588.209763] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 6588.218131] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 6588.222976] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 6588.227564] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6588.229588] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 6588.237043] Call Trace: [ 6588.239539] [ 6588.241121] ? show_regs.cold.9+0x22/0x2f [ 6588.242888] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6588.245616] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6588.247906] ? watchdog+0x40/0x40 [ 6588.249734] ? hrtimer_interrupt+0x128/0x2c0 [ 6588.252488] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6588.256672] ? apic_timer_interrupt+0xf/0x20 [ 6588.260154] [ 6588.260904] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6588.268433] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6588.271827] do_raw_spin_lock+0x177/0x190 [ 6588.273793] _raw_spin_lock+0x21/0x30 [ 6588.276253] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6588.278296] process_one_work+0x2c8/0x700 [ 6588.281501] worker_thread+0x296/0x6e0 [ 6588.286030] ? rescuer_thread+0x570/0x570 [ 6588.289740] kthread+0x1d1/0x200 [ 6588.294113] ? set_kthread_struct+0x70/0x70 [ 6588.302082] ret_from_fork+0x1f/0x30 [ 6596.061387] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [ 6596.072394] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6596.097066] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6596.102121] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6596.105956] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6596.108039] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6596.114365] RSP: 0018:ffffaf9fc1543ae0 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 6596.117480] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 6596.120481] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 6596.123774] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6596.127128] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 6596.130669] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 6596.133613] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 6596.137072] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6596.139580] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 6596.142553] Call Trace: [ 6596.143525] [ 6596.144409] ? show_regs.cold.9+0x22/0x2f [ 6596.146153] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6596.148231] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6596.150469] ? watchdog+0x40/0x40 [ 6596.152143] ? hrtimer_interrupt+0x128/0x2c0 [ 6596.154434] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6596.156830] ? apic_timer_interrupt+0xf/0x20 [ 6596.159094] [ 6596.160158] ? native_safe_halt+0x2e/0x50 [ 6596.164672] kvm_wait+0xcd/0xe0 [ 6596.166046] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6596.169637] do_raw_spin_lock+0x177/0x190 [ 6596.172071] _raw_spin_lock+0x21/0x30 [ 6596.175116] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6596.179656] ? do_raw_spin_unlock+0x75/0x190 [ 6596.181807] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6596.184535] ? do_wait_intr+0xf0/0xf0 [ 6596.186129] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6596.189644] mdc_close+0x53b/0xd70 [mdc] [ 6596.192505] lmv_close+0x18d/0x490 [lmv] [ 6596.195043] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 6596.199658] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6596.203505] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6596.206517] ll_md_close+0x274/0x800 [lustre] [ 6596.210547] ll_file_release+0x128/0xc50 [lustre] [ 6596.216581] __fput+0x10d/0x440 [ 6596.221186] ____fput+0x12/0x20 [ 6596.224772] task_work_run+0xc6/0x110 [ 6596.229917] exit_to_usermode_loop+0x1dd/0x1f0 [ 6596.232166] do_syscall_64+0x3d6/0x440 [ 6596.234180] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6596.236997] RIP: 0033:0x7ff54af45cd8 [ 6596.238977] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6596.249714] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6596.255761] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 6596.264825] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 6596.273153] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 6596.276114] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 6596.279221] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6600.055091] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 6600.067056] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6600.120829] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6600.134518] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6600.139494] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6600.144622] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6600.162673] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6600.166812] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 6600.174096] RDX: 0000000000004e69 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 6600.183077] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 6600.189693] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 6600.205689] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6600.214690] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 6600.224086] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6600.227472] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 6600.231493] Call Trace: [ 6600.232353] [ 6600.233135] ? show_regs.cold.9+0x22/0x2f [ 6600.237417] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6600.240693] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6600.243134] ? watchdog+0x40/0x40 [ 6600.251474] ? hrtimer_interrupt+0x128/0x2c0 [ 6600.252889] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6600.264030] ? apic_timer_interrupt+0xf/0x20 [ 6600.268371] [ 6600.269590] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6600.276437] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6600.283126] do_raw_spin_lock+0x177/0x190 [ 6600.289345] _raw_spin_lock+0x21/0x30 [ 6600.297751] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6600.305301] obd_mod_free+0x46/0x140 [ptlrpc] [ 6600.314797] mdc_commit_open+0x91/0xb0 [mdc] [ 6600.317547] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6600.320990] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6600.325037] after_reply+0xe2c/0x1780 [ptlrpc] [ 6600.327013] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6600.333989] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6600.340200] ? do_wait_intr+0xf0/0xf0 [ 6600.343992] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6600.349799] kthread+0x1d1/0x200 [ 6600.353494] ? set_kthread_struct+0x70/0x70 [ 6600.357632] ret_from_fork+0x1f/0x30 [ 6604.255118] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4645s! [ 6604.259026] Showing busy workqueues and worker pools: [ 6604.266963] workqueue events: flags=0x0 [ 6604.272157] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6604.274318] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 6604.280515] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6604.285146] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 6604.296156] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6604.303597] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 6604.308281] workqueue events_long: flags=0x0 [ 6604.309565] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6604.312062] pending: lu_objects_flush [obdclass] [ 6604.316064] workqueue events_unbound: flags=0x2 [ 6604.318832] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6604.322067] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6604.327185] workqueue events_power_efficient: flags=0x80 [ 6604.331403] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6604.340230] pending: neigh_periodic_work [ 6604.347535] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6604.351502] pending: do_cache_clean, check_lifetime [ 6604.354428] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6604.367603] pending: neigh_periodic_work [ 6604.376146] workqueue writeback: flags=0x4a [ 6604.385554] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6604.391229] pending: wb_workfn, wb_workfn [ 6604.395948] workqueue xprtiod: flags=0xa [ 6604.397687] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6604.409948] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 6604.419457] workqueue ipv6_addrconf: flags=0x40008 [ 6604.420722] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6604.427750] pending: addrconf_verify_work [ 6604.436677] workqueue knbd0-recv: flags=0x1a [ 6604.442506] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6604.453590] in-flight: 657:recv_work [ 6604.455482] workqueue ptlrpc_pinger: flags=0x2 [ 6604.464349] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6604.465877] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 6604.479620] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4645s workers=4 idle: 20831 12642 28859 [ 6604.482255] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6594s workers=2 idle: 659 [ 6608.060043] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [ 6608.062831] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6608.081014] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6608.086591] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6608.089859] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6608.091145] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6608.100753] RSP: 0000:ffffaf9fc15234f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6608.103553] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 6608.105892] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 6608.109119] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6608.111774] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 6608.120520] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 6608.123763] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 6608.128822] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6608.133100] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 6608.136452] Call Trace: [ 6608.137173] [ 6608.137968] ? show_regs.cold.9+0x22/0x2f [ 6608.139801] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6608.141439] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6608.142514] ? watchdog+0x40/0x40 [ 6608.143308] ? hrtimer_interrupt+0x128/0x2c0 [ 6608.149618] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6608.153051] ? apic_timer_interrupt+0xf/0x20 [ 6608.158143] [ 6608.162595] ? native_safe_halt+0x2e/0x50 [ 6608.166086] kvm_wait+0xcd/0xe0 [ 6608.171878] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6608.176998] do_raw_spin_lock+0x177/0x190 [ 6608.179592] _raw_spin_lock+0x21/0x30 [ 6608.180983] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6608.183793] ? do_raw_spin_unlock+0x75/0x190 [ 6608.185686] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6608.189344] ? do_wait_intr+0xf0/0xf0 [ 6608.192503] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6608.196879] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6608.200991] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6608.206884] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6608.209809] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6608.212029] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6608.217899] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6608.221174] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6608.227563] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6608.235189] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6608.241453] lmv_intent_open+0x338/0x1150 [lmv] [ 6608.242564] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6608.243931] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6608.251650] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6608.254010] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6608.256061] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6608.258734] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6608.260649] ? do_raw_spin_unlock+0x75/0x190 [ 6608.262788] ? _raw_spin_unlock+0x12/0x30 [ 6608.264107] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6608.266819] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6608.269859] ll_atomic_open+0x36a/0xd70 [lustre] [ 6608.272967] lookup_open+0x356/0xb40 [ 6608.274915] do_last+0x313/0xfc0 [ 6608.275932] ? nd_jump_root+0xe5/0x160 [ 6608.277070] ? path_init+0x437/0x520 [ 6608.279137] path_openat+0xf7/0x500 [ 6608.280461] ? do_raw_spin_unlock+0x75/0x190 [ 6608.282810] do_filp_open+0x99/0x140 [ 6608.286766] ? getname_flags+0x6e/0x330 [ 6608.287942] ? __check_object_size+0xff/0x256 [ 6608.290214] ? do_raw_spin_unlock+0x75/0x190 [ 6608.293012] ? _raw_spin_unlock+0x12/0x30 [ 6608.294647] do_sys_openat2+0x2b4/0x410 [ 6608.297144] do_sys_open+0x73/0xa0 [ 6608.300408] __x64_sys_openat+0x24/0x30 [ 6608.301583] do_syscall_64+0xc1/0x440 [ 6608.303295] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6608.310624] RIP: 0033:0x7ff54af45332 [ 6608.314989] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6608.325949] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6608.328927] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 6608.332609] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 6608.336653] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 6608.339550] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 6608.346144] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 6616.059541] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 6616.069361] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6616.116531] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6616.126698] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6616.131931] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6616.137190] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6616.139961] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6616.156429] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6616.161461] RAX: 0000000000000220 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 6616.165410] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 6616.168805] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 6616.172900] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 6616.176468] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 6616.180421] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 6616.187991] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6616.193927] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 6616.198575] Call Trace: [ 6616.200541] [ 6616.203631] ? show_regs.cold.9+0x22/0x2f [ 6616.206579] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6616.210296] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6616.213071] ? watchdog+0x40/0x40 [ 6616.217193] ? hrtimer_interrupt+0x128/0x2c0 [ 6616.219814] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6616.222426] ? apic_timer_interrupt+0xf/0x20 [ 6616.225227] [ 6616.226814] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6616.230500] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6616.233807] do_raw_spin_lock+0x177/0x190 [ 6616.236485] _raw_spin_lock+0x21/0x30 [ 6616.240332] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6616.246502] process_one_work+0x2c8/0x700 [ 6616.249626] worker_thread+0x296/0x6e0 [ 6616.251279] ? rescuer_thread+0x570/0x570 [ 6616.253511] kthread+0x1d1/0x200 [ 6616.255955] ? set_kthread_struct+0x70/0x70 [ 6616.259628] ret_from_fork+0x1f/0x30 [ 6624.058371] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [ 6624.076332] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6624.106299] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6624.113238] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6624.117774] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6624.120663] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6624.130825] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6624.136960] RAX: 000000000000421f RBX: ffff96848811cae0 RCX: 0000000000000001 [ 6624.143894] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 6624.149694] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6624.154461] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 6624.157316] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 6624.160572] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 6624.165000] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6624.166628] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 6624.168582] Call Trace: [ 6624.169670] [ 6624.170594] ? show_regs.cold.9+0x22/0x2f [ 6624.172749] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6624.175490] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6624.177425] ? watchdog+0x40/0x40 [ 6624.178752] ? hrtimer_interrupt+0x128/0x2c0 [ 6624.180865] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6624.182841] ? apic_timer_interrupt+0xf/0x20 [ 6624.184780] [ 6624.186039] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6624.189240] do_raw_spin_lock+0x177/0x190 [ 6624.190851] _raw_spin_lock+0x21/0x30 [ 6624.192462] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6624.195756] ? do_raw_spin_unlock+0x75/0x190 [ 6624.197727] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6624.200741] ? do_wait_intr+0xf0/0xf0 [ 6624.202815] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6624.209111] mdc_close+0x53b/0xd70 [mdc] [ 6624.211976] lmv_close+0x18d/0x490 [lmv] [ 6624.216556] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 6624.219507] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6624.222143] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6624.229869] ll_md_close+0x274/0x800 [lustre] [ 6624.236731] ll_file_release+0x128/0xc50 [lustre] [ 6624.241727] __fput+0x10d/0x440 [ 6624.243565] ____fput+0x12/0x20 [ 6624.245585] task_work_run+0xc6/0x110 [ 6624.247532] exit_to_usermode_loop+0x1dd/0x1f0 [ 6624.249969] do_syscall_64+0x3d6/0x440 [ 6624.251785] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6624.256435] RIP: 0033:0x7ff54af45cd8 [ 6624.257752] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6624.269912] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6624.274716] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 6624.277560] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 6624.281170] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 6624.284804] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 6624.287914] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6632.055207] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 6632.064507] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6632.106748] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6632.120418] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6632.127791] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6632.131421] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6632.154442] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6632.161912] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 6632.169044] RDX: 0000000000005299 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 6632.178231] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 6632.181479] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 6632.189676] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6632.200486] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 6632.212449] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6632.224079] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 6632.225794] Call Trace: [ 6632.228254] [ 6632.229912] ? show_regs.cold.9+0x22/0x2f [ 6632.237365] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6632.242217] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6632.248238] ? watchdog+0x40/0x40 [ 6632.250031] ? hrtimer_interrupt+0x128/0x2c0 [ 6632.252968] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6632.254084] ? apic_timer_interrupt+0xf/0x20 [ 6632.255035] [ 6632.255760] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6632.270959] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6632.280860] do_raw_spin_lock+0x177/0x190 [ 6632.289247] _raw_spin_lock+0x21/0x30 [ 6632.295136] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6632.298728] obd_mod_free+0x46/0x140 [ptlrpc] [ 6632.300669] mdc_commit_open+0x91/0xb0 [mdc] [ 6632.306867] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6632.309345] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6632.312050] after_reply+0xe2c/0x1780 [ptlrpc] [ 6632.314471] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6632.317021] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6632.318710] ? do_wait_intr+0xf0/0xf0 [ 6632.319829] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6632.322236] kthread+0x1d1/0x200 [ 6632.323397] ? set_kthread_struct+0x70/0x70 [ 6632.325366] ret_from_fork+0x1f/0x30 [ 6634.977965] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4676s! [ 6634.988849] Showing busy workqueues and worker pools: [ 6634.992667] workqueue events: flags=0x0 [ 6635.000625] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6635.004807] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 6635.016713] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6635.020886] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 6635.028589] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6635.035456] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 6635.039970] workqueue events_long: flags=0x0 [ 6635.045509] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6635.049182] pending: lu_objects_flush [obdclass] [ 6635.053996] workqueue events_unbound: flags=0x2 [ 6635.056338] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6635.059054] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6635.066977] workqueue events_power_efficient: flags=0x80 [ 6635.071983] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6635.077598] pending: neigh_periodic_work [ 6635.079393] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6635.084331] pending: do_cache_clean, check_lifetime [ 6635.089278] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6635.092866] pending: neigh_periodic_work [ 6635.095333] workqueue writeback: flags=0x4a [ 6635.098876] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6635.104812] pending: wb_workfn, wb_workfn [ 6635.110379] workqueue xprtiod: flags=0xa [ 6635.114798] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6635.117180] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 6635.133506] workqueue ipv6_addrconf: flags=0x40008 [ 6635.139710] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6635.151855] pending: addrconf_verify_work [ 6635.162354] workqueue knbd0-recv: flags=0x1a [ 6635.168391] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6635.179385] in-flight: 657:recv_work [ 6635.184718] workqueue ptlrpc_pinger: flags=0x2 [ 6635.195721] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6635.207708] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 6635.217155] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4676s workers=4 idle: 20831 12642 28859 [ 6635.226191] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6625s workers=2 idle: 659 [ 6636.061085] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [ 6636.065342] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6636.091858] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6636.098078] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6636.102617] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6636.105713] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6636.115233] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6636.119218] RAX: 0000000000007c07 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 6636.122719] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 6636.126281] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6636.131113] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 6636.137492] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 6636.141386] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 6636.150321] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6636.155026] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 6636.160988] Call Trace: [ 6636.162817] [ 6636.164799] ? show_regs.cold.9+0x22/0x2f [ 6636.166916] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6636.174261] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6636.185194] ? watchdog+0x40/0x40 [ 6636.191104] ? hrtimer_interrupt+0x128/0x2c0 [ 6636.193413] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6636.199996] ? apic_timer_interrupt+0xf/0x20 [ 6636.205750] [ 6636.206791] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6636.213537] do_raw_spin_lock+0x177/0x190 [ 6636.218164] _raw_spin_lock+0x21/0x30 [ 6636.223291] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6636.230182] ? do_raw_spin_unlock+0x75/0x190 [ 6636.233256] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6636.236849] ? do_wait_intr+0xf0/0xf0 [ 6636.238881] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6636.242459] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6636.245807] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6636.249792] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6636.253368] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6636.256190] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6636.258709] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6636.261533] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6636.264887] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6636.268078] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6636.270641] lmv_intent_open+0x338/0x1150 [lmv] [ 6636.272736] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6636.275622] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6636.277997] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6636.280015] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6636.282249] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6636.285188] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6636.287470] ? do_raw_spin_unlock+0x75/0x190 [ 6636.289239] ? _raw_spin_unlock+0x12/0x30 [ 6636.290960] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6636.292394] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6636.294284] ll_atomic_open+0x36a/0xd70 [lustre] [ 6636.296395] lookup_open+0x356/0xb40 [ 6636.297950] do_last+0x313/0xfc0 [ 6636.299129] ? nd_jump_root+0xe5/0x160 [ 6636.300867] ? path_init+0x437/0x520 [ 6636.302805] path_openat+0xf7/0x500 [ 6636.304404] ? do_raw_spin_unlock+0x75/0x190 [ 6636.306251] do_filp_open+0x99/0x140 [ 6636.307962] ? getname_flags+0x6e/0x330 [ 6636.309695] ? __check_object_size+0xff/0x256 [ 6636.311696] ? do_raw_spin_unlock+0x75/0x190 [ 6636.313697] ? _raw_spin_unlock+0x12/0x30 [ 6636.315529] do_sys_openat2+0x2b4/0x410 [ 6636.317329] do_sys_open+0x73/0xa0 [ 6636.319084] __x64_sys_openat+0x24/0x30 [ 6636.320821] do_syscall_64+0xc1/0x440 [ 6636.322544] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6636.324928] RIP: 0033:0x7ff54af45332 [ 6636.326863] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6636.334636] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6636.338202] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 6636.341382] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 6636.344247] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 6636.347498] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 6636.350820] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 6644.059170] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 6644.067440] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6644.107651] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6644.122967] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6644.129343] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6644.135974] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6644.142659] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6644.168416] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6644.173666] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 6644.180779] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 6644.189491] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 6644.198726] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 6644.206326] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 6644.208921] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 6644.225031] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6644.228494] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 6644.232790] Call Trace: [ 6644.236527] [ 6644.237615] ? show_regs.cold.9+0x22/0x2f [ 6644.240664] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6644.246422] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6644.251470] ? watchdog+0x40/0x40 [ 6644.253575] ? hrtimer_interrupt+0x128/0x2c0 [ 6644.255732] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6644.259802] ? apic_timer_interrupt+0xf/0x20 [ 6644.264629] [ 6644.267659] ? native_safe_halt+0x2e/0x50 [ 6644.272796] kvm_wait+0xcd/0xe0 [ 6644.274545] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6644.277276] do_raw_spin_lock+0x177/0x190 [ 6644.279327] _raw_spin_lock+0x21/0x30 [ 6644.281085] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6644.283947] process_one_work+0x2c8/0x700 [ 6644.286062] worker_thread+0x296/0x6e0 [ 6644.287909] ? rescuer_thread+0x570/0x570 [ 6644.290528] kthread+0x1d1/0x200 [ 6644.292276] ? set_kthread_struct+0x70/0x70 [ 6644.294258] ret_from_fork+0x1f/0x30 [ 6652.061519] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [ 6652.067980] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6652.089424] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6652.094174] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6652.098877] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6652.101618] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6652.123016] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6652.128486] RAX: 0000000000002585 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 6652.134652] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 6652.142359] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6652.145313] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 6652.151293] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 6652.158444] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 6652.164663] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6652.167874] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 6652.177409] Call Trace: [ 6652.178374] [ 6652.179127] ? show_regs.cold.9+0x22/0x2f [ 6652.184686] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6652.189516] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6652.195171] ? watchdog+0x40/0x40 [ 6652.197029] ? hrtimer_interrupt+0x128/0x2c0 [ 6652.201689] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6652.204352] ? apic_timer_interrupt+0xf/0x20 [ 6652.208206] [ 6652.210844] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6652.214362] do_raw_spin_lock+0x177/0x190 [ 6652.218436] _raw_spin_lock+0x21/0x30 [ 6652.224206] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6652.227972] ? do_raw_spin_unlock+0x75/0x190 [ 6652.231574] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6652.234452] ? do_wait_intr+0xf0/0xf0 [ 6652.238107] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6652.247485] mdc_close+0x53b/0xd70 [mdc] [ 6652.251820] lmv_close+0x18d/0x490 [lmv] [ 6652.255441] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 6652.260917] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6652.271659] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6652.279124] ll_md_close+0x274/0x800 [lustre] [ 6652.284608] ll_file_release+0x128/0xc50 [lustre] [ 6652.292321] __fput+0x10d/0x440 [ 6652.299338] ____fput+0x12/0x20 [ 6652.303997] task_work_run+0xc6/0x110 [ 6652.309593] exit_to_usermode_loop+0x1dd/0x1f0 [ 6652.314521] do_syscall_64+0x3d6/0x440 [ 6652.318246] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6652.326221] RIP: 0033:0x7ff54af45cd8 [ 6652.330436] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6652.350172] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6652.353486] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 6652.371000] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 6652.383954] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 6652.392454] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 6652.402199] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6660.055130] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 6660.060369] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6660.099116] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6660.109194] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6660.113764] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6660.116091] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6660.130558] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6660.134190] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 6660.144023] RDX: 0000000000000c7b RSI: 0000000000000003 RDI: ffff96848811cae0 [ 6660.151839] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 6660.159468] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 6660.165081] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6660.169316] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 6660.181764] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6660.188236] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 6660.192176] Call Trace: [ 6660.193498] [ 6660.194528] ? show_regs.cold.9+0x22/0x2f [ 6660.196524] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6660.199393] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6660.202135] ? watchdog+0x40/0x40 [ 6660.208171] ? hrtimer_interrupt+0x128/0x2c0 [ 6660.210419] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6660.212463] ? apic_timer_interrupt+0xf/0x20 [ 6660.213817] [ 6660.218862] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6660.221499] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6660.230116] do_raw_spin_lock+0x177/0x190 [ 6660.234114] _raw_spin_lock+0x21/0x30 [ 6660.235608] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6660.239846] obd_mod_free+0x46/0x140 [ptlrpc] [ 6660.242622] mdc_commit_open+0x91/0xb0 [mdc] [ 6660.244740] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6660.247553] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6660.253094] after_reply+0xe2c/0x1780 [ptlrpc] [ 6660.255380] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6660.258329] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6660.264525] ? do_wait_intr+0xf0/0xf0 [ 6660.267009] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6660.276181] kthread+0x1d1/0x200 [ 6660.282384] ? set_kthread_struct+0x70/0x70 [ 6660.286607] ret_from_fork+0x1f/0x30 [ 6664.061835] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 6664.065470] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6664.115095] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6664.126184] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6664.135082] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6664.142137] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6664.164149] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6664.170816] RAX: 0000000000003b05 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 6664.181847] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 6664.191307] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6664.195186] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 6664.207396] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 6664.221036] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 6664.229706] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6664.240347] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 6664.254463] Call Trace: [ 6664.259115] [ 6664.260254] ? show_regs.cold.9+0x22/0x2f [ 6664.262413] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6664.267855] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6664.272954] ? watchdog+0x40/0x40 [ 6664.277772] ? hrtimer_interrupt+0x128/0x2c0 [ 6664.279187] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6664.282453] ? apic_timer_interrupt+0xf/0x20 [ 6664.288208] [ 6664.291756] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6664.294440] do_raw_spin_lock+0x177/0x190 [ 6664.299438] _raw_spin_lock+0x21/0x30 [ 6664.303575] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6664.310578] ? do_raw_spin_unlock+0x75/0x190 [ 6664.314027] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6664.315769] ? do_wait_intr+0xf0/0xf0 [ 6664.316878] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6664.324336] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6664.336890] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6664.340594] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6664.345018] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6664.348029] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6664.353831] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6664.359014] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6664.363393] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6664.369927] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6664.373094] lmv_intent_open+0x338/0x1150 [lmv] [ 6664.378688] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6664.381741] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6664.385404] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6664.389074] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6664.392887] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6664.401384] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6664.415704] ? do_raw_spin_unlock+0x75/0x190 [ 6664.417275] ? _raw_spin_unlock+0x12/0x30 [ 6664.418511] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6664.425697] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6664.430910] ll_atomic_open+0x36a/0xd70 [lustre] [ 6664.435876] lookup_open+0x356/0xb40 [ 6664.440327] do_last+0x313/0xfc0 [ 6664.442771] ? nd_jump_root+0xe5/0x160 [ 6664.445797] ? path_init+0x437/0x520 [ 6664.450392] path_openat+0xf7/0x500 [ 6664.451448] ? do_raw_spin_unlock+0x75/0x190 [ 6664.455986] do_filp_open+0x99/0x140 [ 6664.458525] ? getname_flags+0x6e/0x330 [ 6664.468118] ? __check_object_size+0xff/0x256 [ 6664.471037] ? do_raw_spin_unlock+0x75/0x190 [ 6664.478803] ? _raw_spin_unlock+0x12/0x30 [ 6664.483771] do_sys_openat2+0x2b4/0x410 [ 6664.488842] do_sys_open+0x73/0xa0 [ 6664.490100] __x64_sys_openat+0x24/0x30 [ 6664.491457] do_syscall_64+0xc1/0x440 [ 6664.494793] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6664.502466] RIP: 0033:0x7ff54af45332 [ 6664.508024] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6664.521178] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6664.531936] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 6664.535459] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 6664.542889] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 6664.553141] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 6664.560579] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 6665.695032] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4706s! [ 6665.717590] Showing busy workqueues and worker pools: [ 6665.728170] workqueue events: flags=0x0 [ 6665.734209] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6665.740748] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 6665.764100] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6665.775781] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 6665.788373] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6665.800617] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 6665.813572] workqueue events_long: flags=0x0 [ 6665.815163] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6665.841167] pending: lu_objects_flush [obdclass] [ 6665.849009] workqueue events_unbound: flags=0x2 [ 6665.850668] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6665.862924] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6665.873376] workqueue events_power_efficient: flags=0x80 [ 6665.886666] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6665.888220] pending: neigh_periodic_work [ 6665.902289] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6665.914776] pending: do_cache_clean, check_lifetime [ 6665.920529] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6665.934255] pending: neigh_periodic_work [ 6665.945367] workqueue writeback: flags=0x4a [ 6665.946957] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6665.958275] pending: wb_workfn, wb_workfn [ 6665.967532] workqueue xprtiod: flags=0xa [ 6665.974433] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6665.984268] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 6665.995931] workqueue ipv6_addrconf: flags=0x40008 [ 6666.010023] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6666.019137] pending: addrconf_verify_work [ 6666.027479] workqueue knbd0-recv: flags=0x1a [ 6666.034946] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6666.047426] in-flight: 657:recv_work [ 6666.054021] workqueue ptlrpc_pinger: flags=0x2 [ 6666.059070] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6666.065497] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 6666.068839] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4707s workers=4 idle: 20831 12642 28859 [ 6666.074950] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6656s workers=2 idle: 659 [ 6672.059025] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 6672.067253] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6672.111888] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6672.120213] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6672.129204] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6672.136753] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6672.139344] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6672.161958] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6672.168751] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 6672.172830] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 6672.178063] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 6672.182113] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 6672.189983] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 6672.197357] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 6672.202118] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6672.207126] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 6672.216057] Call Trace: [ 6672.219365] [ 6672.220181] ? show_regs.cold.9+0x22/0x2f [ 6672.222314] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6672.227943] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6672.231009] ? watchdog+0x40/0x40 [ 6672.232744] ? hrtimer_interrupt+0x128/0x2c0 [ 6672.234937] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6672.240949] ? apic_timer_interrupt+0xf/0x20 [ 6672.246119] [ 6672.248169] ? native_safe_halt+0x2e/0x50 [ 6672.250886] kvm_wait+0xcd/0xe0 [ 6672.252965] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6672.255338] do_raw_spin_lock+0x177/0x190 [ 6672.258333] _raw_spin_lock+0x21/0x30 [ 6672.261986] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6672.265206] process_one_work+0x2c8/0x700 [ 6672.267151] worker_thread+0x296/0x6e0 [ 6672.269195] ? rescuer_thread+0x570/0x570 [ 6672.271288] kthread+0x1d1/0x200 [ 6672.274302] ? set_kthread_struct+0x70/0x70 [ 6672.276284] ret_from_fork+0x1f/0x30 [ 6680.059654] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [ 6680.065836] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6680.105707] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6680.120724] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6680.129073] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6680.134898] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6680.148565] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6680.152078] RAX: 0000000000006538 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 6680.156048] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 6680.163273] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6680.167909] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 6680.171490] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 6680.173563] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 6680.182847] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6680.185621] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 6680.192531] Call Trace: [ 6680.193326] [ 6680.196585] ? show_regs.cold.9+0x22/0x2f [ 6680.204721] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6680.206694] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6680.209605] ? watchdog+0x40/0x40 [ 6680.212191] ? hrtimer_interrupt+0x128/0x2c0 [ 6680.214186] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6680.219203] ? apic_timer_interrupt+0xf/0x20 [ 6680.220442] [ 6680.221022] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6680.222541] do_raw_spin_lock+0x177/0x190 [ 6680.229002] _raw_spin_lock+0x21/0x30 [ 6680.232858] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6680.237868] ? do_raw_spin_unlock+0x75/0x190 [ 6680.240210] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6680.242697] ? do_wait_intr+0xf0/0xf0 [ 6680.248071] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6680.250859] mdc_close+0x53b/0xd70 [mdc] [ 6680.255186] lmv_close+0x18d/0x490 [lmv] [ 6680.256979] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 6680.259688] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6680.262021] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6680.264545] ll_md_close+0x274/0x800 [lustre] [ 6680.266431] ll_file_release+0x128/0xc50 [lustre] [ 6680.268610] __fput+0x10d/0x440 [ 6680.269986] ____fput+0x12/0x20 [ 6680.271171] task_work_run+0xc6/0x110 [ 6680.272716] exit_to_usermode_loop+0x1dd/0x1f0 [ 6680.274581] do_syscall_64+0x3d6/0x440 [ 6680.276070] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6680.278205] RIP: 0033:0x7ff54af45cd8 [ 6680.279492] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6680.287638] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6680.290782] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 6680.293830] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 6680.301686] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 6680.304478] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 6680.306735] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6692.061431] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 6692.061909] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 6692.070661] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6692.071675] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6692.071695] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6692.071698] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6692.071713] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6692.071718] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6692.071723] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 6692.071725] RDX: 00000000000067ec RSI: 0000000000000003 RDI: ffff96848811cae0 [ 6692.071727] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 6692.071729] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 6692.071732] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6692.071736] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 6692.071738] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6692.071741] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 6692.071749] Call Trace: [ 6692.071755] [ 6692.071761] ? show_regs.cold.9+0x22/0x2f [ 6692.071771] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6692.071778] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6692.071781] ? watchdog+0x40/0x40 [ 6692.071785] ? hrtimer_interrupt+0x128/0x2c0 [ 6692.071789] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6692.071797] ? apic_timer_interrupt+0xf/0x20 [ 6692.071801] [ 6692.071803] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6692.071808] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6692.071812] do_raw_spin_lock+0x177/0x190 [ 6692.071816] _raw_spin_lock+0x21/0x30 [ 6692.071822] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6692.076094] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6692.076179] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6692.076189] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6692.076192] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6692.076207] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6692.076212] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6692.076216] RAX: 0000000000006c91 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 6692.076219] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 6692.076221] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6692.076223] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 6692.076226] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 6692.076230] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 6692.076233] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6692.076250] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 6692.076259] Call Trace: [ 6692.076267] [ 6692.076281] ? show_regs.cold.9+0x22/0x2f [ 6692.076292] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6692.076298] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6692.076302] ? watchdog+0x40/0x40 [ 6692.076305] ? hrtimer_interrupt+0x128/0x2c0 [ 6692.076310] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6692.076319] ? apic_timer_interrupt+0xf/0x20 [ 6692.076323] [ 6692.076327] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6692.076334] do_raw_spin_lock+0x177/0x190 [ 6692.077645] _raw_spin_lock+0x21/0x30 [ 6692.077668] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6692.078101] ? do_raw_spin_unlock+0x75/0x190 [ 6692.078109] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6692.078304] ? do_wait_intr+0xf0/0xf0 [ 6692.078313] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6692.079167] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6692.079388] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6692.079609] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6692.079739] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6692.080021] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6692.080048] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6692.080074] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6692.080159] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6692.080492] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6692.080534] lmv_intent_open+0x338/0x1150 [lmv] [ 6692.080572] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6692.080659] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6692.112311] obd_mod_free+0x46/0x140 [ptlrpc] [ 6692.112679] mdc_commit_open+0x91/0xb0 [mdc] [ 6692.112723] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6692.117879] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6692.117934] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6692.118059] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6692.118150] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6692.118237] ? do_raw_spin_unlock+0x75/0x190 [ 6692.118246] ? _raw_spin_unlock+0x12/0x30 [ 6692.118252] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6692.118281] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6692.118301] ll_atomic_open+0x36a/0xd70 [lustre] [ 6692.118383] lookup_open+0x356/0xb40 [ 6692.118392] do_last+0x313/0xfc0 [ 6692.118399] ? nd_jump_root+0xe5/0x160 [ 6692.118403] ? path_init+0x437/0x520 [ 6692.118408] path_openat+0xf7/0x500 [ 6692.118412] ? do_raw_spin_unlock+0x75/0x190 [ 6692.118417] do_filp_open+0x99/0x140 [ 6692.118423] ? getname_flags+0x6e/0x330 [ 6692.118427] ? __check_object_size+0xff/0x256 [ 6692.118434] ? do_raw_spin_unlock+0x75/0x190 [ 6692.118438] ? _raw_spin_unlock+0x12/0x30 [ 6692.118442] do_sys_openat2+0x2b4/0x410 [ 6692.118446] do_sys_open+0x73/0xa0 [ 6692.118451] __x64_sys_openat+0x24/0x30 [ 6692.118454] do_syscall_64+0xc1/0x440 [ 6692.118461] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6692.118466] RIP: 0033:0x7ff54af45332 [ 6692.118473] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6692.118477] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6692.118483] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 6692.118485] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 6692.118487] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 6692.118489] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 6692.118491] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 6692.686317] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6692.688610] after_reply+0xe2c/0x1780 [ptlrpc] [ 6692.690746] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6692.699705] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6692.705593] ? do_wait_intr+0xf0/0xf0 [ 6692.706998] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6692.711159] kthread+0x1d1/0x200 [ 6692.712675] ? set_kthread_struct+0x70/0x70 [ 6692.714910] ret_from_fork+0x1f/0x30 [ 6693.979173] rcu: INFO: rcu_sched self-detected stall on CPU [ 6693.981600] rcu: 0-...!: (2459306 ticks this GP) idle=98a/1/0x4000000000000002 softirq=125154/125154 fqs=959 [ 6693.986744] (t=4740139 jiffies g=253585 q=4348) [ 6693.988491] rcu: rcu_sched kthread starved for 4735099 jiffies! g253585 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [ 6693.998814] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 6694.005754] rcu: RCU grace-period kthread stack dump: [ 6694.010721] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 6694.022564] Call Trace: [ 6694.030021] __schedule+0x351/0xcb0 [ 6694.034956] schedule+0xc0/0x180 [ 6694.036173] schedule_timeout+0xb4/0x190 [ 6694.042708] ? do_raw_spin_unlock+0x75/0x190 [ 6694.046976] ? __next_timer_interrupt+0x160/0x160 [ 6694.051217] ? prepare_to_swait_event+0xf3/0x140 [ 6694.052764] rcu_gp_kthread+0x8dd/0x15b0 [ 6694.059116] ? force_qs_rnp+0x2c0/0x2c0 [ 6694.061964] kthread+0x1d1/0x200 [ 6694.067973] ? set_kthread_struct+0x70/0x70 [ 6694.071062] ret_from_fork+0x1f/0x30 [ 6694.072231] rcu: Stack dump where RCU GP kthread last ran: [ 6694.078810] Sending NMI from CPU 0 to CPUs 1: [ 6694.087230] NMI backtrace for cpu 1 [ 6694.087239] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6694.087241] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6694.087243] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6694.087245] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6694.087247] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 [ 6694.087251] RAX: 0000000000007510 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 6694.087253] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 6694.087254] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6694.087255] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 6694.087257] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 6694.087259] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 6694.087260] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6694.087262] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 6694.087263] Call Trace: [ 6694.087264] [ 6694.087265] ? show_regs.cold.9+0x22/0x2f [ 6694.087267] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 6694.087268] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 6694.087269] ? nmi_handle+0xbd/0x210 [ 6694.087270] ? default_do_nmi+0x4b/0x1b0 [ 6694.087271] ? do_nmi+0x280/0x380 [ 6694.087272] ? end_repeat_nmi+0x16/0x1f [ 6694.087273] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6694.087275] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6694.087276] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6694.087277] [ 6694.087278] do_raw_spin_lock+0x177/0x190 [ 6694.087280] _raw_spin_lock+0x21/0x30 [ 6694.087281] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6694.087282] ? do_raw_spin_unlock+0x75/0x190 [ 6694.087967] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6694.087969] ? do_wait_intr+0xf0/0xf0 [ 6694.087970] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6694.087971] mdc_close+0x53b/0xd70 [mdc] [ 6694.087972] lmv_close+0x18d/0x490 [lmv] [ 6694.087973] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 6694.087974] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6694.087976] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6694.087977] ll_md_close+0x274/0x800 [lustre] [ 6694.087978] ll_file_release+0x128/0xc50 [lustre] [ 6694.087979] __fput+0x10d/0x440 [ 6694.087980] ____fput+0x12/0x20 [ 6694.087981] task_work_run+0xc6/0x110 [ 6694.087982] exit_to_usermode_loop+0x1dd/0x1f0 [ 6694.087983] do_syscall_64+0x3d6/0x440 [ 6694.087985] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6694.087986] RIP: 0033:0x7ff54af45cd8 [ 6694.087987] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6694.087990] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6694.087993] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 6694.087995] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 6694.087996] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 6694.087997] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 6694.087999] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6694.088522] NMI backtrace for cpu 0 [ 6694.469122] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6694.480684] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6694.496156] Call Trace: [ 6694.501209] [ 6694.503752] ? dump_stack+0xbb/0x10e [ 6694.510355] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 6694.512251] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 6694.516792] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 6694.523215] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 6694.533138] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 6694.535055] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 6694.540258] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 6694.544078] ? update_process_times+0x6d/0xc0 [ 6694.549999] ? tick_sched_handle.isra.18+0x27/0x90 [ 6694.551673] ? tick_sched_timer+0x53/0xb0 [ 6694.559592] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6694.562220] ? tick_sched_handle.isra.18+0x90/0x90 [ 6694.567255] ? hrtimer_interrupt+0x128/0x2c0 [ 6694.574967] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6694.577732] ? apic_timer_interrupt+0xf/0x20 [ 6694.583644] [ 6694.587164] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6694.589088] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6694.602652] ? do_raw_spin_lock+0x177/0x190 [ 6694.610623] ? _raw_spin_lock+0x21/0x30 [ 6694.617137] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6694.625743] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 6694.639298] ? mdc_commit_open+0x91/0xb0 [mdc] [ 6694.644675] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6694.652204] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6694.663780] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 6694.673245] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6694.687150] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6694.692295] ? do_wait_intr+0xf0/0xf0 [ 6694.694812] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6694.698239] ? kthread+0x1d1/0x200 [ 6694.700440] ? set_kthread_struct+0x70/0x70 [ 6694.702449] ? ret_from_fork+0x1f/0x30 [ 6696.415725] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4737s! [ 6696.420546] Showing busy workqueues and worker pools: [ 6696.423039] workqueue events: flags=0x0 [ 6696.425425] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6696.430178] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 6696.436451] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6696.439507] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 6696.444648] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6696.448132] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 6696.451727] workqueue events_long: flags=0x0 [ 6696.454425] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6696.459437] pending: lu_objects_flush [obdclass] [ 6696.463640] workqueue events_unbound: flags=0x2 [ 6696.466450] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6696.469767] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6696.475767] workqueue events_power_efficient: flags=0x80 [ 6696.480647] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6696.485665] pending: neigh_periodic_work [ 6696.488704] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6696.492612] pending: do_cache_clean, check_lifetime [ 6696.495901] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6696.500215] pending: neigh_periodic_work [ 6696.503495] workqueue writeback: flags=0x4a [ 6696.506374] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6696.510339] pending: wb_workfn, wb_workfn [ 6696.512768] workqueue xprtiod: flags=0xa [ 6696.515916] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6696.519764] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 6696.524048] workqueue ipv6_addrconf: flags=0x40008 [ 6696.526701] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6696.531767] pending: addrconf_verify_work [ 6696.534815] workqueue knbd0-recv: flags=0x1a [ 6696.538147] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6696.544414] in-flight: 657:recv_work [ 6696.548461] workqueue ptlrpc_pinger: flags=0x2 [ 6696.554099] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6696.557803] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 6696.565906] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4737s workers=4 idle: 20831 12642 28859 [ 6696.573870] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6686s workers=2 idle: 659 [ 6700.059586] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 6700.067824] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6700.104705] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6700.111240] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6700.122113] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6700.128135] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6700.135919] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6700.156812] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6700.162831] RAX: 0000000000000332 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 6700.171937] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 6700.178793] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 6700.186774] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 6700.193865] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 6700.199514] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 6700.206000] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6700.209823] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 6700.215616] Call Trace: [ 6700.218027] [ 6700.219015] ? show_regs.cold.9+0x22/0x2f [ 6700.221780] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6700.225248] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6700.227889] ? watchdog+0x40/0x40 [ 6700.230654] ? hrtimer_interrupt+0x128/0x2c0 [ 6700.233568] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6700.236796] ? apic_timer_interrupt+0xf/0x20 [ 6700.239758] [ 6700.241226] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6700.244757] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6700.251176] do_raw_spin_lock+0x177/0x190 [ 6700.253289] _raw_spin_lock+0x21/0x30 [ 6700.255546] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6700.258458] process_one_work+0x2c8/0x700 [ 6700.259718] worker_thread+0x296/0x6e0 [ 6700.261406] ? rescuer_thread+0x570/0x570 [ 6700.263202] kthread+0x1d1/0x200 [ 6700.264421] ? set_kthread_struct+0x70/0x70 [ 6700.266147] ret_from_fork+0x1f/0x30 [ 6720.058116] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 6720.060018] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 6720.063268] Modules linked in: [ 6720.067405] Modules linked in: [ 6720.067408] lustre(O) osp(O) [ 6720.069058] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul [ 6720.070754] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common [ 6720.072015] crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6720.072040] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6720.072046] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6720.072048] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6720.072062] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6720.090851] sb_edac rapl pcspkr [ 6720.102412] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6720.102588] RAX: 00000000000039f8 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 6720.102592] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 6720.102595] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6720.102597] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 6720.102600] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 6720.102603] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 6720.102606] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6720.102609] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 6720.102617] Call Trace: [ 6720.102625] [ 6720.102631] ? show_regs.cold.9+0x22/0x2f [ 6720.102644] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6720.102651] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6720.102655] ? watchdog+0x40/0x40 [ 6720.102660] ? hrtimer_interrupt+0x128/0x2c0 [ 6720.102663] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6720.102671] ? apic_timer_interrupt+0xf/0x20 [ 6720.102676] [ 6720.102679] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6720.102688] do_raw_spin_lock+0x177/0x190 [ 6720.102693] _raw_spin_lock+0x21/0x30 [ 6720.102774] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6720.103175] ? do_raw_spin_unlock+0x75/0x190 [ 6720.103182] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6720.103399] ? do_wait_intr+0xf0/0xf0 [ 6720.103408] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6720.103690] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6720.104308] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6720.104670] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6720.104803] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6720.104844] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6720.104884] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6720.104913] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6720.105004] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6720.105580] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6720.105612] lmv_intent_open+0x338/0x1150 [lmv] [ 6720.105648] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6720.105798] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6720.113546] i2c_piix4 [ 6720.119396] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6720.119569] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6720.123912] squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror [ 6720.126908] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6720.136052] dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6720.136175] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6720.138101] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6720.141360] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6720.146138] ? do_raw_spin_unlock+0x75/0x190 [ 6720.146160] ? _raw_spin_unlock+0x12/0x30 [ 6720.146168] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6720.146203] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6720.146222] ll_atomic_open+0x36a/0xd70 [lustre] [ 6720.146359] lookup_open+0x356/0xb40 [ 6720.146370] do_last+0x313/0xfc0 [ 6720.146375] ? nd_jump_root+0xe5/0x160 [ 6720.146379] ? path_init+0x437/0x520 [ 6720.146384] path_openat+0xf7/0x500 [ 6720.146389] ? do_raw_spin_unlock+0x75/0x190 [ 6720.146394] do_filp_open+0x99/0x140 [ 6720.146400] ? getname_flags+0x6e/0x330 [ 6720.146404] ? __check_object_size+0xff/0x256 [ 6720.146411] ? do_raw_spin_unlock+0x75/0x190 [ 6720.146415] ? _raw_spin_unlock+0x12/0x30 [ 6720.146419] do_sys_openat2+0x2b4/0x410 [ 6720.146424] do_sys_open+0x73/0xa0 [ 6720.146607] __x64_sys_openat+0x24/0x30 [ 6720.146611] do_syscall_64+0xc1/0x440 [ 6720.146618] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6720.146625] RIP: 0033:0x7ff54af45332 [ 6720.146632] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6720.146636] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6720.146641] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 6720.146644] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 6720.146646] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 6720.146648] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 6720.146650] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 6720.473194] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6720.475914] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6720.498450] RSP: 0018:ffffaf9fc1543ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6720.505410] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 6720.508811] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 6720.515469] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6720.518189] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 6720.522408] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 6720.528218] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 6720.532344] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6720.535357] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 6720.540927] Call Trace: [ 6720.541735] [ 6720.542446] ? show_regs.cold.9+0x22/0x2f [ 6720.549460] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6720.554588] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6720.563338] ? watchdog+0x40/0x40 [ 6720.566368] ? hrtimer_interrupt+0x128/0x2c0 [ 6720.571104] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6720.573719] ? apic_timer_interrupt+0xf/0x20 [ 6720.579499] [ 6720.580557] ? native_safe_halt+0x2e/0x50 [ 6720.591423] kvm_wait+0xcd/0xe0 [ 6720.597036] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6720.607452] do_raw_spin_lock+0x177/0x190 [ 6720.613581] _raw_spin_lock+0x21/0x30 [ 6720.620257] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6720.629160] ? do_raw_spin_unlock+0x75/0x190 [ 6720.630750] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6720.640833] ? do_wait_intr+0xf0/0xf0 [ 6720.642017] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6720.653697] mdc_close+0x53b/0xd70 [mdc] [ 6720.661292] lmv_close+0x18d/0x490 [lmv] [ 6720.666156] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 6720.669713] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6720.672712] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6720.676305] ll_md_close+0x274/0x800 [lustre] [ 6720.678675] ll_file_release+0x128/0xc50 [lustre] [ 6720.682741] __fput+0x10d/0x440 [ 6720.684442] ____fput+0x12/0x20 [ 6720.687713] task_work_run+0xc6/0x110 [ 6720.692696] exit_to_usermode_loop+0x1dd/0x1f0 [ 6720.697278] do_syscall_64+0x3d6/0x440 [ 6720.700372] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6720.705944] RIP: 0033:0x7ff54af45cd8 [ 6720.708544] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6720.725487] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6720.736468] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 6720.745379] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 6720.754087] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 6720.758360] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 6720.763504] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6724.055155] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [ 6724.062351] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6724.119736] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6724.128471] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6724.134762] RIP: 0010:load_balance+0x725/0x1090 [ 6724.139022] Code: 48 83 00 01 9c 58 0f 1f 44 00 00 48 83 05 7a 6d dd 02 01 48 83 05 82 b9 dd 02 01 f6 c4 02 0f 85 22 03 00 00 48 8b 3c 24 57 9d <0f> 1f 44 00 00 8b 84 24 98 00 00 00 a8 02 74 23 83 e0 fd 48 83 05 [ 6724.157724] RSP: 0000:ffffaf9fc0003de0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6724.166064] RAX: 0000000000000006 RBX: ffff9684a0d8d030 RCX: ffff9684c2033440 [ 6724.174532] RDX: 0000000000000008 RSI: ffff9684a0d8cf80 RDI: 0000000000000202 [ 6724.181520] RBP: ffffaf9fc0003e90 R08: ffff968482cb4f80 R09: 0000000000000000 [ 6724.184957] R10: ffff9684a0d8d018 R11: 000000000000a9ab R12: ffff9684a0d8cf80 [ 6724.194926] R13: ffff9684c20b3e08 R14: 0000000000000006 R15: ffff9684c20b3380 [ 6724.199990] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 6724.204947] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6724.209539] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 6724.218352] Call Trace: [ 6724.220771] [ 6724.224372] ? show_regs.cold.9+0x22/0x2f [ 6724.226955] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6724.229939] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6724.234710] ? watchdog+0x40/0x40 [ 6724.237911] ? hrtimer_interrupt+0x128/0x2c0 [ 6724.243777] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6724.246923] ? apic_timer_interrupt+0xf/0x20 [ 6724.253920] ? apic_timer_interrupt+0xa/0x20 [ 6724.257697] ? load_balance+0x725/0x1090 [ 6724.263962] rebalance_domains+0x348/0x580 [ 6724.270622] ? kvm_clock_get_cycles+0x2c/0x50 [ 6724.275392] run_rebalance_domains+0x68/0xa0 [ 6724.276626] __do_softirq+0x134/0x4e8 [ 6724.279619] irq_exit_rcu+0x1c6/0x1e0 [ 6724.281171] irq_exit+0x12/0x30 [ 6724.282494] smp_apic_timer_interrupt+0xe0/0x270 [ 6724.284493] apic_timer_interrupt+0xf/0x20 [ 6724.286317] [ 6724.287070] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6724.289771] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6724.295817] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6724.297955] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 6724.299781] RDX: 00000000000071b3 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 6724.301690] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 6724.310249] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 6724.316349] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6724.319245] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6724.320896] do_raw_spin_lock+0x177/0x190 [ 6724.322167] _raw_spin_lock+0x21/0x30 [ 6724.323157] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6724.324993] obd_mod_free+0x46/0x140 [ptlrpc] [ 6724.326729] mdc_commit_open+0x91/0xb0 [mdc] [ 6724.327823] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6724.329617] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6724.331553] after_reply+0xe2c/0x1780 [ptlrpc] [ 6724.333318] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6724.335121] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6724.336598] ? do_wait_intr+0xf0/0xf0 [ 6724.337577] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6724.339583] kthread+0x1d1/0x200 [ 6724.340546] ? set_kthread_struct+0x70/0x70 [ 6724.341798] ret_from_fork+0x1f/0x30 [ 6726.624499] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4767s! [ 6726.629712] Showing busy workqueues and worker pools: [ 6726.632295] workqueue events: flags=0x0 [ 6726.634438] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6726.637800] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 6726.644540] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6726.651699] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 6726.656855] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6726.663518] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 6726.671702] workqueue events_long: flags=0x0 [ 6726.673951] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6726.677773] pending: lu_objects_flush [obdclass] [ 6726.681212] workqueue events_unbound: flags=0x2 [ 6726.683860] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6726.686671] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6726.691434] workqueue events_power_efficient: flags=0x80 [ 6726.693910] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6726.703330] pending: neigh_periodic_work [ 6726.709919] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6726.717472] pending: do_cache_clean, check_lifetime [ 6726.723850] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6726.731386] pending: neigh_periodic_work [ 6726.737811] workqueue writeback: flags=0x4a [ 6726.742412] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6726.746981] pending: wb_workfn, wb_workfn [ 6726.752220] workqueue xprtiod: flags=0xa [ 6726.755747] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6726.759816] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 6726.766250] workqueue ipv6_addrconf: flags=0x40008 [ 6726.768555] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6726.777676] pending: addrconf_verify_work [ 6726.787754] workqueue knbd0-recv: flags=0x1a [ 6726.793820] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6726.797928] in-flight: 657:recv_work [ 6726.800840] workqueue ptlrpc_pinger: flags=0x2 [ 6726.803845] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6726.807323] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 6726.812863] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4767s workers=4 idle: 20831 12642 28859 [ 6726.822890] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6717s workers=2 idle: 659 [ 6728.059024] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 6728.072753] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6728.116796] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6728.136023] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6728.145341] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6728.149354] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6728.152244] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6728.169615] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6728.175215] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 6728.178876] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 6728.183647] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 6728.192710] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 6728.201756] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 6728.209090] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 6728.216466] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6728.223621] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 6728.230618] Call Trace: [ 6728.232800] [ 6728.233993] ? show_regs.cold.9+0x22/0x2f [ 6728.247020] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6728.248732] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6728.253209] ? watchdog+0x40/0x40 [ 6728.257589] ? hrtimer_interrupt+0x128/0x2c0 [ 6728.260643] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6728.263453] ? apic_timer_interrupt+0xf/0x20 [ 6728.265688] [ 6728.266642] ? native_safe_halt+0x2e/0x50 [ 6728.269690] kvm_wait+0xcd/0xe0 [ 6728.272055] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6728.274755] do_raw_spin_lock+0x177/0x190 [ 6728.281474] _raw_spin_lock+0x21/0x30 [ 6728.285320] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6728.289515] process_one_work+0x2c8/0x700 [ 6728.292367] worker_thread+0x296/0x6e0 [ 6728.294409] ? rescuer_thread+0x570/0x570 [ 6728.297466] kthread+0x1d1/0x200 [ 6728.300781] ? set_kthread_struct+0x70/0x70 [ 6728.311706] ret_from_fork+0x1f/0x30 [ 6748.058193] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 6748.060025] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 6748.066120] Modules linked in: [ 6748.070429] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6748.070570] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6748.070578] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6748.070580] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6748.070604] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6748.070609] RSP: 0000:ffffaf9fc15234f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6748.070613] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 6748.070615] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 6748.070618] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6748.070620] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 6748.070623] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 6748.070626] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 6748.070629] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6748.070631] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 6748.070639] Call Trace: [ 6748.070652] [ 6748.070658] ? show_regs.cold.9+0x22/0x2f [ 6748.070669] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6748.070674] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6748.070679] ? watchdog+0x40/0x40 [ 6748.070683] ? hrtimer_interrupt+0x128/0x2c0 [ 6748.070687] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6748.070693] ? apic_timer_interrupt+0xf/0x20 [ 6748.070698] [ 6748.070700] ? native_safe_halt+0x2e/0x50 [ 6748.070704] kvm_wait+0xcd/0xe0 [ 6748.070712] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6748.070720] do_raw_spin_lock+0x177/0x190 [ 6748.070724] _raw_spin_lock+0x21/0x30 [ 6748.070728] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6748.071103] ? do_raw_spin_unlock+0x75/0x190 [ 6748.071109] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6748.071312] ? do_wait_intr+0xf0/0xf0 [ 6748.071321] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6748.071514] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6748.071867] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6748.072151] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6748.072653] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6748.072696] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6748.072726] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6748.072755] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6748.072910] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6748.073263] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6748.073295] lmv_intent_open+0x338/0x1150 [lmv] [ 6748.073330] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6748.073417] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6748.073501] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6748.073525] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6748.073608] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6748.073687] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6748.074575] ? do_raw_spin_unlock+0x75/0x190 [ 6748.074586] ? _raw_spin_unlock+0x12/0x30 [ 6748.074592] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6748.074624] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6748.074642] ll_atomic_open+0x36a/0xd70 [lustre] [ 6748.074733] lookup_open+0x356/0xb40 [ 6748.074742] do_last+0x313/0xfc0 [ 6748.074746] ? nd_jump_root+0xe5/0x160 [ 6748.074751] ? path_init+0x437/0x520 [ 6748.074887] path_openat+0xf7/0x500 [ 6748.074892] ? do_raw_spin_unlock+0x75/0x190 [ 6748.074896] do_filp_open+0x99/0x140 [ 6748.074902] ? getname_flags+0x6e/0x330 [ 6748.074906] ? __check_object_size+0xff/0x256 [ 6748.074913] ? do_raw_spin_unlock+0x75/0x190 [ 6748.074916] ? _raw_spin_unlock+0x12/0x30 [ 6748.074920] do_sys_openat2+0x2b4/0x410 [ 6748.074925] do_sys_open+0x73/0xa0 [ 6748.074930] __x64_sys_openat+0x24/0x30 [ 6748.074933] do_syscall_64+0xc1/0x440 [ 6748.074942] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6748.074949] RIP: 0033:0x7ff54af45332 [ 6748.074957] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6748.074961] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6748.075048] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 6748.075050] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 6748.075053] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 6748.075055] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 6748.075057] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 6748.389314] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6748.407478] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6748.419540] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6748.424488] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6748.427112] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6748.435811] RSP: 0018:ffffaf9fc1543ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6748.438619] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 6748.442132] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 6748.445435] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6748.450629] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 6748.463861] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 6748.473724] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 6748.483014] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6748.490733] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 6748.493028] Call Trace: [ 6748.493882] [ 6748.495140] ? show_regs.cold.9+0x22/0x2f [ 6748.496539] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6748.500046] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6748.502440] ? watchdog+0x40/0x40 [ 6748.504303] ? hrtimer_interrupt+0x128/0x2c0 [ 6748.507602] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6748.510183] ? apic_timer_interrupt+0xf/0x20 [ 6748.511304] [ 6748.511813] ? native_safe_halt+0x2e/0x50 [ 6748.512747] kvm_wait+0xcd/0xe0 [ 6748.513548] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6748.515454] do_raw_spin_lock+0x177/0x190 [ 6748.517207] _raw_spin_lock+0x21/0x30 [ 6748.518324] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6748.521239] ? do_raw_spin_unlock+0x75/0x190 [ 6748.523030] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6748.525942] ? do_wait_intr+0xf0/0xf0 [ 6748.527640] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6748.530688] mdc_close+0x53b/0xd70 [mdc] [ 6748.532626] lmv_close+0x18d/0x490 [lmv] [ 6748.538871] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 6748.545427] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6748.550661] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6748.553218] ll_md_close+0x274/0x800 [lustre] [ 6748.555503] ll_file_release+0x128/0xc50 [lustre] [ 6748.558853] __fput+0x10d/0x440 [ 6748.561326] ____fput+0x12/0x20 [ 6748.563787] task_work_run+0xc6/0x110 [ 6748.566930] exit_to_usermode_loop+0x1dd/0x1f0 [ 6748.569640] do_syscall_64+0x3d6/0x440 [ 6748.571604] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6748.574487] RIP: 0033:0x7ff54af45cd8 [ 6748.576471] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6748.597750] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6748.611474] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 6748.623235] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 6748.630638] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 6748.640502] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 6748.649755] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6752.055032] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [ 6752.059768] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6752.083524] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6752.088178] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6752.091766] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6752.095495] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6752.115376] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6752.124708] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 6752.132814] RDX: 0000000000006cc9 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 6752.140939] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 6752.147536] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 6752.155095] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6752.160400] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 6752.174430] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6752.180746] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 6752.192978] Call Trace: [ 6752.194273] [ 6752.195366] ? show_regs.cold.9+0x22/0x2f [ 6752.202664] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6752.210826] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6752.212321] ? watchdog+0x40/0x40 [ 6752.214130] ? hrtimer_interrupt+0x128/0x2c0 [ 6752.217029] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6752.224032] ? apic_timer_interrupt+0xf/0x20 [ 6752.226047] [ 6752.231406] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6752.233865] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6752.244651] do_raw_spin_lock+0x177/0x190 [ 6752.246468] _raw_spin_lock+0x21/0x30 [ 6752.252123] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6752.254971] obd_mod_free+0x46/0x140 [ptlrpc] [ 6752.258923] mdc_commit_open+0x91/0xb0 [mdc] [ 6752.261945] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6752.266385] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6752.273118] after_reply+0xe2c/0x1780 [ptlrpc] [ 6752.282724] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6752.285481] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6752.294318] ? do_wait_intr+0xf0/0xf0 [ 6752.300256] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6752.309626] kthread+0x1d1/0x200 [ 6752.311990] ? set_kthread_struct+0x70/0x70 [ 6752.317567] ret_from_fork+0x1f/0x30 [ 6756.061434] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 6756.065242] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6756.109953] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6756.117609] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6756.127773] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6756.131571] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6756.134426] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6756.147542] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6756.153101] RAX: 0000000000004b14 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 6756.157188] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 6756.160583] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 6756.164170] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 6756.167252] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 6756.173007] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 6756.176328] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6756.179302] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 6756.185582] Call Trace: [ 6756.186569] [ 6756.188878] ? show_regs.cold.9+0x22/0x2f [ 6756.191007] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6756.194830] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6756.197909] ? watchdog+0x40/0x40 [ 6756.201914] ? hrtimer_interrupt+0x128/0x2c0 [ 6756.204706] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6756.208696] ? apic_timer_interrupt+0xf/0x20 [ 6756.210824] [ 6756.212006] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6756.214571] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6756.217062] do_raw_spin_lock+0x177/0x190 [ 6756.218610] _raw_spin_lock+0x21/0x30 [ 6756.220102] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6756.222450] process_one_work+0x2c8/0x700 [ 6756.225269] worker_thread+0x296/0x6e0 [ 6756.228852] ? rescuer_thread+0x570/0x570 [ 6756.233359] kthread+0x1d1/0x200 [ 6756.235318] ? set_kthread_struct+0x70/0x70 [ 6756.239068] ret_from_fork+0x1f/0x30 [ 6756.834096] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4797s! [ 6756.848199] Showing busy workqueues and worker pools: [ 6756.856793] workqueue events: flags=0x0 [ 6756.861933] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6756.870005] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 6756.879569] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6756.885003] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 6756.896315] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6756.900115] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 6756.908699] workqueue events_long: flags=0x0 [ 6756.912207] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6756.919109] pending: lu_objects_flush [obdclass] [ 6756.923191] workqueue events_unbound: flags=0x2 [ 6756.926979] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6756.935299] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6756.944030] workqueue events_power_efficient: flags=0x80 [ 6756.946578] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6756.956345] pending: neigh_periodic_work [ 6756.959223] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6756.967082] pending: do_cache_clean, check_lifetime [ 6756.969973] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6756.973506] pending: neigh_periodic_work [ 6756.975703] workqueue writeback: flags=0x4a [ 6756.978040] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6756.982130] pending: wb_workfn, wb_workfn [ 6756.984907] workqueue xprtiod: flags=0xa [ 6756.987415] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6756.990975] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 6756.995589] workqueue ipv6_addrconf: flags=0x40008 [ 6756.998097] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6757.000863] pending: addrconf_verify_work [ 6757.003960] workqueue knbd0-recv: flags=0x1a [ 6757.007013] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6757.012068] in-flight: 657:recv_work [ 6757.018518] workqueue ptlrpc_pinger: flags=0x2 [ 6757.023658] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6757.033251] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 6757.047567] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4798s workers=4 idle: 20831 12642 28859 [ 6757.060863] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6747s workers=2 idle: 659 [ 6776.058196] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 6776.063039] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6776.063659] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 6776.108678] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6776.108694] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6776.108706] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6776.108720] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6776.108724] RSP: 0018:ffffaf9fc1543ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6776.108729] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 6776.108731] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 6776.108733] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6776.108736] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 6776.108738] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 6776.108742] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 6776.108745] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6776.108756] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 6776.108763] Call Trace: [ 6776.108769] [ 6776.108775] ? show_regs.cold.9+0x22/0x2f [ 6776.108784] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6776.108790] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6776.108794] ? watchdog+0x40/0x40 [ 6776.108798] ? hrtimer_interrupt+0x128/0x2c0 [ 6776.108806] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6776.108813] ? apic_timer_interrupt+0xf/0x20 [ 6776.108817] [ 6776.108819] ? native_safe_halt+0x2e/0x50 [ 6776.108823] kvm_wait+0xcd/0xe0 [ 6776.108829] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6776.108839] do_raw_spin_lock+0x177/0x190 [ 6776.108843] _raw_spin_lock+0x21/0x30 [ 6776.108846] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6776.109190] ? do_raw_spin_unlock+0x75/0x190 [ 6776.109195] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6776.109714] ? do_wait_intr+0xf0/0xf0 [ 6776.109722] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6776.110095] mdc_close+0x53b/0xd70 [mdc] [ 6776.110210] lmv_close+0x18d/0x490 [lmv] [ 6776.110250] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 6776.110903] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6776.111020] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6776.111095] ll_md_close+0x274/0x800 [lustre] [ 6776.115325] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6776.115398] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6776.115405] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6776.115408] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6776.115645] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6776.115649] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6776.115654] RAX: 00000000000072e1 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 6776.115656] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 6776.115659] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6776.115661] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 6776.115664] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 6776.115667] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 6776.115670] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6776.115673] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 6776.115680] Call Trace: [ 6776.115686] [ 6776.115692] ? show_regs.cold.9+0x22/0x2f [ 6776.115774] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6776.115781] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6776.115790] ? watchdog+0x40/0x40 [ 6776.115795] ? hrtimer_interrupt+0x128/0x2c0 [ 6776.115799] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6776.115807] ? apic_timer_interrupt+0xf/0x20 [ 6776.115815] [ 6776.115818] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6776.115824] do_raw_spin_lock+0x177/0x190 [ 6776.115829] _raw_spin_lock+0x21/0x30 [ 6776.115834] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6776.116187] ? do_raw_spin_unlock+0x75/0x190 [ 6776.116193] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6776.116672] ? do_wait_intr+0xf0/0xf0 [ 6776.116682] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6776.117113] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6776.117716] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6776.124869] ll_file_release+0x128/0xc50 [lustre] [ 6776.125002] __fput+0x10d/0x440 [ 6776.125012] ____fput+0x12/0x20 [ 6776.125016] task_work_run+0xc6/0x110 [ 6776.125025] exit_to_usermode_loop+0x1dd/0x1f0 [ 6776.125034] do_syscall_64+0x3d6/0x440 [ 6776.125040] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6776.125049] RIP: 0033:0x7ff54af45cd8 [ 6776.125106] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6776.125111] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6776.125117] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 6776.125119] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 6776.125122] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 6776.125124] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 6776.125126] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6776.422263] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6776.424883] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6776.426638] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6776.427974] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6776.431292] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6776.433252] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6776.435925] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6776.439046] lmv_intent_open+0x338/0x1150 [lmv] [ 6776.440659] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6776.443886] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6776.445213] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6776.447864] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6776.449955] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6776.452688] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6776.454907] ? do_raw_spin_unlock+0x75/0x190 [ 6776.456962] ? _raw_spin_unlock+0x12/0x30 [ 6776.458960] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6776.461363] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6776.463749] ll_atomic_open+0x36a/0xd70 [lustre] [ 6776.466228] lookup_open+0x356/0xb40 [ 6776.467680] do_last+0x313/0xfc0 [ 6776.469830] ? nd_jump_root+0xe5/0x160 [ 6776.471855] ? path_init+0x437/0x520 [ 6776.472850] path_openat+0xf7/0x500 [ 6776.475052] ? do_raw_spin_unlock+0x75/0x190 [ 6776.477338] do_filp_open+0x99/0x140 [ 6776.478706] ? getname_flags+0x6e/0x330 [ 6776.481679] ? __check_object_size+0xff/0x256 [ 6776.483299] ? do_raw_spin_unlock+0x75/0x190 [ 6776.486462] ? _raw_spin_unlock+0x12/0x30 [ 6776.487651] do_sys_openat2+0x2b4/0x410 [ 6776.489923] do_sys_open+0x73/0xa0 [ 6776.491354] __x64_sys_openat+0x24/0x30 [ 6776.493046] do_syscall_64+0xc1/0x440 [ 6776.495965] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6776.498813] RIP: 0033:0x7ff54af45332 [ 6776.500781] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6776.511867] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6776.516079] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 6776.520346] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 6776.522615] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 6776.525116] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 6776.527825] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 6784.057446] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [ 6784.063165] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 6784.066586] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6784.067071] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6784.067079] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6784.067082] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6784.067098] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6784.067103] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6784.067107] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 6784.067109] RDX: 0000000000000057 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 6784.067112] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 6784.067114] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 6784.067116] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6784.067119] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 6784.067122] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6784.067125] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 6784.067133] Call Trace: [ 6784.067139] [ 6784.067145] ? show_regs.cold.9+0x22/0x2f [ 6784.067157] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6784.067164] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6784.067168] ? watchdog+0x40/0x40 [ 6784.067173] ? hrtimer_interrupt+0x128/0x2c0 [ 6784.067177] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6784.067184] ? apic_timer_interrupt+0xf/0x20 [ 6784.067189] [ 6784.067192] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6784.067196] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6784.067200] do_raw_spin_lock+0x177/0x190 [ 6784.067205] _raw_spin_lock+0x21/0x30 [ 6784.067210] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6784.069965] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6784.070041] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6784.070048] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6784.070053] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6784.070331] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6784.070654] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6784.070658] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6784.070663] RAX: 0000000000000c13 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 6784.070666] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 6784.070668] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 6784.070670] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 6784.070672] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 6784.070675] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 6784.070678] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6784.070680] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 6784.070688] Call Trace: [ 6784.070694] [ 6784.070699] ? show_regs.cold.9+0x22/0x2f [ 6784.070709] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6784.070716] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6784.070720] ? watchdog+0x40/0x40 [ 6784.070724] ? hrtimer_interrupt+0x128/0x2c0 [ 6784.070728] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6784.070738] ? apic_timer_interrupt+0xf/0x20 [ 6784.070743] [ 6784.070746] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6784.070751] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6784.070757] do_raw_spin_lock+0x177/0x190 [ 6784.070762] _raw_spin_lock+0x21/0x30 [ 6784.070767] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6784.070988] process_one_work+0x2c8/0x700 [ 6784.070994] worker_thread+0x296/0x6e0 [ 6784.070999] ? rescuer_thread+0x570/0x570 [ 6784.071002] kthread+0x1d1/0x200 [ 6784.071006] ? set_kthread_struct+0x70/0x70 [ 6784.071010] ret_from_fork+0x1f/0x30 [ 6784.478427] obd_mod_free+0x46/0x140 [ptlrpc] [ 6784.484793] mdc_commit_open+0x91/0xb0 [mdc] [ 6784.488742] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6784.492277] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6784.499346] after_reply+0xe2c/0x1780 [ptlrpc] [ 6784.502441] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6784.507809] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6784.512816] ? do_wait_intr+0xf0/0xf0 [ 6784.516148] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6784.525634] kthread+0x1d1/0x200 [ 6784.528445] ? set_kthread_struct+0x70/0x70 [ 6784.532822] ret_from_fork+0x1f/0x30 [ 6787.551746] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4828s! [ 6787.559526] Showing busy workqueues and worker pools: [ 6787.561281] workqueue events: flags=0x0 [ 6787.570179] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6787.584161] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 6787.592598] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6787.597476] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 6787.609146] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6787.613367] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 6787.616615] workqueue events_long: flags=0x0 [ 6787.622292] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6787.625358] pending: lu_objects_flush [obdclass] [ 6787.628397] workqueue events_unbound: flags=0x2 [ 6787.630930] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6787.633375] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6787.637071] workqueue events_power_efficient: flags=0x80 [ 6787.639400] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6787.648301] pending: neigh_periodic_work [ 6787.649997] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6787.656031] pending: do_cache_clean, check_lifetime [ 6787.659868] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6787.663969] pending: neigh_periodic_work [ 6787.666173] workqueue writeback: flags=0x4a [ 6787.669073] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6787.673194] pending: wb_workfn, wb_workfn [ 6787.677402] workqueue xprtiod: flags=0xa [ 6787.679185] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6787.682816] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 6787.688048] workqueue ipv6_addrconf: flags=0x40008 [ 6787.693864] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6787.701750] pending: addrconf_verify_work [ 6787.706397] workqueue knbd0-recv: flags=0x1a [ 6787.710810] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6787.717836] in-flight: 657:recv_work [ 6787.721269] workqueue ptlrpc_pinger: flags=0x2 [ 6787.727294] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6787.734333] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 6787.741362] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4828s workers=4 idle: 20831 12642 28859 [ 6787.746430] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6777s workers=2 idle: 659 [ 6804.059422] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 6804.060038] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 6804.067075] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6804.067170] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6804.067179] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6804.067183] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6804.067203] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6804.067207] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6804.067212] RAX: 0000000000004991 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 6804.067215] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 6804.067217] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6804.067219] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 6804.067221] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 6804.067225] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 6804.067228] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6804.067230] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 6804.067240] Call Trace: [ 6804.067246] [ 6804.067254] ? show_regs.cold.9+0x22/0x2f [ 6804.067264] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6804.067272] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6804.067276] ? watchdog+0x40/0x40 [ 6804.067280] ? hrtimer_interrupt+0x128/0x2c0 [ 6804.067284] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6804.067296] ? apic_timer_interrupt+0xf/0x20 [ 6804.067301] [ 6804.067304] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6804.067311] do_raw_spin_lock+0x177/0x190 [ 6804.067316] _raw_spin_lock+0x21/0x30 [ 6804.067320] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6804.068257] ? do_raw_spin_unlock+0x75/0x190 [ 6804.068266] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6804.069491] ? do_wait_intr+0xf0/0xf0 [ 6804.069504] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6804.069756] mdc_close+0x53b/0xd70 [mdc] [ 6804.069889] lmv_close+0x18d/0x490 [lmv] [ 6804.069924] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 6804.070117] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6804.070241] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6804.070328] ll_md_close+0x274/0x800 [lustre] [ 6804.070417] ll_file_release+0x128/0xc50 [lustre] [ 6804.071078] __fput+0x10d/0x440 [ 6804.071090] ____fput+0x12/0x20 [ 6804.071095] task_work_run+0xc6/0x110 [ 6804.071104] exit_to_usermode_loop+0x1dd/0x1f0 [ 6804.071113] do_syscall_64+0x3d6/0x440 [ 6804.071118] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6804.071130] RIP: 0033:0x7ff54af45cd8 [ 6804.071140] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6804.071145] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6804.071151] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 6804.071154] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 6804.071156] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 6804.071158] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 6804.071161] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6804.411543] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6804.429583] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6804.434008] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6804.436794] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6804.438633] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6804.446031] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6804.450784] RAX: 0000000000003c14 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 6804.457617] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 6804.460161] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6804.462514] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 6804.464593] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 6804.466496] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 6804.469048] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6804.471544] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 6804.474799] Call Trace: [ 6804.475809] [ 6804.476851] ? show_regs.cold.9+0x22/0x2f [ 6804.478675] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6804.481034] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6804.483221] ? watchdog+0x40/0x40 [ 6804.484692] ? hrtimer_interrupt+0x128/0x2c0 [ 6804.486677] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6804.489215] ? apic_timer_interrupt+0xf/0x20 [ 6804.491239] [ 6804.492420] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6804.494653] do_raw_spin_lock+0x177/0x190 [ 6804.496692] _raw_spin_lock+0x21/0x30 [ 6804.498117] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6804.505490] ? do_raw_spin_unlock+0x75/0x190 [ 6804.509049] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6804.516219] ? do_wait_intr+0xf0/0xf0 [ 6804.519846] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6804.523714] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6804.526626] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6804.528826] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6804.533353] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6804.536145] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6804.541126] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6804.546545] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6804.552137] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6804.559113] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6804.563216] lmv_intent_open+0x338/0x1150 [lmv] [ 6804.565660] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6804.568803] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6804.571676] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6804.573811] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6804.576210] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6804.580223] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6804.582637] ? do_raw_spin_unlock+0x75/0x190 [ 6804.584987] ? _raw_spin_unlock+0x12/0x30 [ 6804.589144] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6804.594804] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6804.606621] ll_atomic_open+0x36a/0xd70 [lustre] [ 6804.611781] lookup_open+0x356/0xb40 [ 6804.614589] do_last+0x313/0xfc0 [ 6804.616800] ? nd_jump_root+0xe5/0x160 [ 6804.618820] ? path_init+0x437/0x520 [ 6804.620579] path_openat+0xf7/0x500 [ 6804.622173] ? do_raw_spin_unlock+0x75/0x190 [ 6804.624817] do_filp_open+0x99/0x140 [ 6804.628529] ? getname_flags+0x6e/0x330 [ 6804.630653] ? __check_object_size+0xff/0x256 [ 6804.635735] ? do_raw_spin_unlock+0x75/0x190 [ 6804.641648] ? _raw_spin_unlock+0x12/0x30 [ 6804.644802] do_sys_openat2+0x2b4/0x410 [ 6804.649664] do_sys_open+0x73/0xa0 [ 6804.651091] __x64_sys_openat+0x24/0x30 [ 6804.652636] do_syscall_64+0xc1/0x440 [ 6804.654784] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6804.659653] RIP: 0033:0x7ff54af45332 [ 6804.663674] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6804.677798] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6804.683223] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 6804.697127] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 6804.703794] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 6804.711750] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 6804.719281] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 6812.060204] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [ 6812.067408] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6812.072366] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 6812.124736] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6812.124752] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6812.124756] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6812.145090] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6812.146000] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6812.146008] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6812.146015] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6812.146617] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6812.146638] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6812.146643] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 6812.146646] RAX: 0000000000007463 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 6812.146648] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 6812.146651] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 6812.146652] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 6812.146654] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 6812.146658] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 6812.146660] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6812.146662] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 6812.147016] Call Trace: [ 6812.147023] [ 6812.147029] ? show_regs.cold.9+0x22/0x2f [ 6812.147040] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6812.147049] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6812.147054] ? watchdog+0x40/0x40 [ 6812.147058] ? hrtimer_interrupt+0x128/0x2c0 [ 6812.147062] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6812.147071] ? apic_timer_interrupt+0xf/0x20 [ 6812.147075] [ 6812.147078] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6812.147084] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6812.147089] do_raw_spin_lock+0x177/0x190 [ 6812.147094] _raw_spin_lock+0x21/0x30 [ 6812.147099] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6812.147384] process_one_work+0x2c8/0x700 [ 6812.147391] worker_thread+0x296/0x6e0 [ 6812.147395] ? rescuer_thread+0x570/0x570 [ 6812.147399] kthread+0x1d1/0x200 [ 6812.147403] ? set_kthread_struct+0x70/0x70 [ 6812.147407] ret_from_fork+0x1f/0x30 [ 6812.397953] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6812.405879] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6812.409029] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 6812.412849] RDX: 0000000000007b41 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 6812.415941] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 6812.419631] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 6812.422963] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6812.426166] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 6812.434717] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6812.437575] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 6812.439791] Call Trace: [ 6812.440776] [ 6812.441838] ? show_regs.cold.9+0x22/0x2f [ 6812.444090] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6812.446616] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6812.449076] ? watchdog+0x40/0x40 [ 6812.450832] ? hrtimer_interrupt+0x128/0x2c0 [ 6812.453255] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6812.454956] ? apic_timer_interrupt+0xf/0x20 [ 6812.463829] [ 6812.464758] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6812.468393] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6812.471862] do_raw_spin_lock+0x177/0x190 [ 6812.473665] _raw_spin_lock+0x21/0x30 [ 6812.476900] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6812.479583] obd_mod_free+0x46/0x140 [ptlrpc] [ 6812.482415] mdc_commit_open+0x91/0xb0 [mdc] [ 6812.484568] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6812.487339] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6812.489916] after_reply+0xe2c/0x1780 [ptlrpc] [ 6812.492999] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6812.502282] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6812.503606] ? do_wait_intr+0xf0/0xf0 [ 6812.504464] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6812.511363] kthread+0x1d1/0x200 [ 6812.513562] ? set_kthread_struct+0x70/0x70 [ 6812.518632] ret_from_fork+0x1f/0x30 [ 6817.761423] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4858s! [ 6817.773714] Showing busy workqueues and worker pools: [ 6817.782266] workqueue events: flags=0x0 [ 6817.792666] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6817.795127] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 6817.800289] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6817.805923] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 6817.825345] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6817.829888] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 6817.838467] workqueue events_long: flags=0x0 [ 6817.842108] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6817.848752] pending: lu_objects_flush [obdclass] [ 6817.852278] workqueue events_unbound: flags=0x2 [ 6817.856650] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6817.866770] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6817.881684] workqueue events_power_efficient: flags=0x80 [ 6817.888545] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6817.894430] pending: neigh_periodic_work [ 6817.897643] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6817.904671] pending: do_cache_clean, check_lifetime [ 6817.909112] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6817.915525] pending: neigh_periodic_work [ 6817.919682] workqueue writeback: flags=0x4a [ 6817.926112] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6817.933406] pending: wb_workfn, wb_workfn [ 6817.936521] workqueue xprtiod: flags=0xa [ 6817.939674] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6817.944061] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 6817.950535] workqueue ipv6_addrconf: flags=0x40008 [ 6817.961685] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6817.967988] pending: addrconf_verify_work [ 6817.970537] workqueue knbd0-recv: flags=0x1a [ 6817.974190] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6817.980095] in-flight: 657:recv_work [ 6817.984595] workqueue ptlrpc_pinger: flags=0x2 [ 6817.992017] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6817.995522] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 6818.003524] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4859s workers=4 idle: 20831 12642 28859 [ 6818.013387] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6808s workers=2 idle: 659 [ 6832.058072] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 6832.060442] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 6832.062650] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6832.062727] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6832.062735] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6832.062739] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6832.062756] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6832.062761] RSP: 0018:ffffaf9fc1543ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6832.062766] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 6832.062776] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 6832.062780] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6832.062782] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 6832.062785] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 6832.062789] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 6832.062792] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6832.062795] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 6832.062803] Call Trace: [ 6832.062809] [ 6832.062815] ? show_regs.cold.9+0x22/0x2f [ 6832.062828] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6832.062835] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6832.062840] ? watchdog+0x40/0x40 [ 6832.062844] ? hrtimer_interrupt+0x128/0x2c0 [ 6832.062848] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6832.062857] ? apic_timer_interrupt+0xf/0x20 [ 6832.062861] [ 6832.062864] ? native_safe_halt+0x2e/0x50 [ 6832.062869] kvm_wait+0xcd/0xe0 [ 6832.062877] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6832.062886] do_raw_spin_lock+0x177/0x190 [ 6832.062891] _raw_spin_lock+0x21/0x30 [ 6832.062896] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6832.069497] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6832.069581] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6832.069589] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6832.069592] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6832.069603] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6832.069607] RSP: 0000:ffffaf9fc15234f0 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 6832.069612] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 6832.069614] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 6832.069616] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6832.069619] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 6832.069621] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 6832.069624] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 6832.069627] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6832.069630] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 6832.069637] Call Trace: [ 6832.069643] [ 6832.069648] ? show_regs.cold.9+0x22/0x2f [ 6832.069656] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6832.069663] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6832.069667] ? watchdog+0x40/0x40 [ 6832.069671] ? hrtimer_interrupt+0x128/0x2c0 [ 6832.069675] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6832.069682] ? apic_timer_interrupt+0xf/0x20 [ 6832.069686] [ 6832.069689] ? native_safe_halt+0x2e/0x50 [ 6832.069693] kvm_wait+0xcd/0xe0 [ 6832.069699] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6832.069706] do_raw_spin_lock+0x177/0x190 [ 6832.069710] _raw_spin_lock+0x21/0x30 [ 6832.069714] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6832.070260] ? do_raw_spin_unlock+0x75/0x190 [ 6832.070266] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6832.070562] ? do_wait_intr+0xf0/0xf0 [ 6832.070570] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6832.070773] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6832.071108] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6832.071326] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6832.071875] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6832.071909] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6832.071934] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6832.071960] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6832.072108] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6832.072346] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6832.072544] lmv_intent_open+0x338/0x1150 [lmv] [ 6832.072579] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6832.072736] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6832.072895] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6832.072920] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6832.073076] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6832.073172] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6832.073256] ? do_raw_spin_unlock+0x75/0x190 [ 6832.073264] ? _raw_spin_unlock+0x12/0x30 [ 6832.073271] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6832.073299] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6832.073320] ll_atomic_open+0x36a/0xd70 [lustre] [ 6832.073491] lookup_open+0x356/0xb40 [ 6832.073499] do_last+0x313/0xfc0 [ 6832.073504] ? nd_jump_root+0xe5/0x160 [ 6832.073508] ? path_init+0x437/0x520 [ 6832.073513] path_openat+0xf7/0x500 [ 6832.073518] ? do_raw_spin_unlock+0x75/0x190 [ 6832.073523] do_filp_open+0x99/0x140 [ 6832.073529] ? getname_flags+0x6e/0x330 [ 6832.073533] ? __check_object_size+0xff/0x256 [ 6832.073540] ? do_raw_spin_unlock+0x75/0x190 [ 6832.073544] ? _raw_spin_unlock+0x12/0x30 [ 6832.073549] do_sys_openat2+0x2b4/0x410 [ 6832.073553] do_sys_open+0x73/0xa0 [ 6832.073557] __x64_sys_openat+0x24/0x30 [ 6832.073561] do_syscall_64+0xc1/0x440 [ 6832.073568] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6832.073575] RIP: 0033:0x7ff54af45332 [ 6832.073583] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6832.073587] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6832.073592] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 6832.073595] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 6832.073598] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 6832.073600] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 6832.073603] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 6832.624751] ? do_raw_spin_unlock+0x75/0x190 [ 6832.627581] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6832.633646] ? do_wait_intr+0xf0/0xf0 [ 6832.637424] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6832.641394] mdc_close+0x53b/0xd70 [mdc] [ 6832.643806] lmv_close+0x18d/0x490 [lmv] [ 6832.644883] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 6832.648579] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6832.651768] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6832.653910] ll_md_close+0x274/0x800 [lustre] [ 6832.655729] ll_file_release+0x128/0xc50 [lustre] [ 6832.656939] __fput+0x10d/0x440 [ 6832.657857] ____fput+0x12/0x20 [ 6832.658909] task_work_run+0xc6/0x110 [ 6832.662223] exit_to_usermode_loop+0x1dd/0x1f0 [ 6832.667111] do_syscall_64+0x3d6/0x440 [ 6832.668645] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6832.671037] RIP: 0033:0x7ff54af45cd8 [ 6832.673611] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6832.689297] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6832.692367] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 6832.696193] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 6832.700178] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 6832.702374] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 6832.705379] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6840.059456] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 6840.062304] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6840.104310] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6840.117242] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6840.122752] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6840.126330] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6840.129139] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6840.140458] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6840.149608] RAX: 0000000000001f81 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 6840.157594] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 6840.169292] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 6840.175268] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 6840.180772] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 6840.186214] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 6840.194754] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6840.200477] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 6840.214006] Call Trace: [ 6840.217796] [ 6840.221999] ? show_regs.cold.9+0x22/0x2f [ 6840.231132] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6840.239160] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6840.240718] ? watchdog+0x40/0x40 [ 6840.241830] ? hrtimer_interrupt+0x128/0x2c0 [ 6840.251495] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6840.255049] ? apic_timer_interrupt+0xf/0x20 [ 6840.257206] [ 6840.262019] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6840.266153] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6840.271157] do_raw_spin_lock+0x177/0x190 [ 6840.272570] _raw_spin_lock+0x21/0x30 [ 6840.273784] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6840.280569] process_one_work+0x2c8/0x700 [ 6840.282589] worker_thread+0x296/0x6e0 [ 6840.285141] ? rescuer_thread+0x570/0x570 [ 6840.287011] kthread+0x1d1/0x200 [ 6840.290503] ? set_kthread_struct+0x70/0x70 [ 6840.293255] ret_from_fork+0x1f/0x30 [ 6844.055979] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [ 6844.058779] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6844.162208] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6844.171855] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6844.180752] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6844.183911] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6844.198282] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6844.205827] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 6844.212450] RDX: 0000000000004698 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 6844.231207] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 6844.240220] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 6844.251685] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6844.257798] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 6844.268486] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6844.273982] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 6844.279245] Call Trace: [ 6844.282514] [ 6844.282961] ? show_regs.cold.9+0x22/0x2f [ 6844.283992] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6844.293970] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6844.295132] ? watchdog+0x40/0x40 [ 6844.295867] ? hrtimer_interrupt+0x128/0x2c0 [ 6844.296862] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6844.304113] ? apic_timer_interrupt+0xf/0x20 [ 6844.305496] [ 6844.306357] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6844.307757] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6844.311605] do_raw_spin_lock+0x177/0x190 [ 6844.313267] _raw_spin_lock+0x21/0x30 [ 6844.314672] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6844.317402] obd_mod_free+0x46/0x140 [ptlrpc] [ 6844.321847] mdc_commit_open+0x91/0xb0 [mdc] [ 6844.326447] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6844.330386] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6844.333032] after_reply+0xe2c/0x1780 [ptlrpc] [ 6844.335413] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6844.337337] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6844.339561] ? do_wait_intr+0xf0/0xf0 [ 6844.341232] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6844.347629] kthread+0x1d1/0x200 [ 6844.349435] ? set_kthread_struct+0x70/0x70 [ 6844.351267] ret_from_fork+0x1f/0x30 [ 6848.481888] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4889s! [ 6848.510559] Showing busy workqueues and worker pools: [ 6848.530072] workqueue events: flags=0x0 [ 6848.540184] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6848.546429] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 6848.552628] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6848.558370] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 6848.565765] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6848.571758] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 6848.578650] workqueue events_long: flags=0x0 [ 6848.581521] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6848.585623] pending: lu_objects_flush [obdclass] [ 6848.589338] workqueue events_unbound: flags=0x2 [ 6848.593421] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6848.602532] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6848.611031] workqueue events_power_efficient: flags=0x80 [ 6848.614252] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6848.618523] pending: neigh_periodic_work [ 6848.622693] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6848.627222] pending: do_cache_clean, check_lifetime [ 6848.630408] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6848.634227] pending: neigh_periodic_work [ 6848.636405] workqueue writeback: flags=0x4a [ 6848.638420] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6848.642127] pending: wb_workfn, wb_workfn [ 6848.644107] workqueue xprtiod: flags=0xa [ 6848.646663] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6848.652254] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 6848.657353] workqueue ipv6_addrconf: flags=0x40008 [ 6848.659760] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6848.664287] pending: addrconf_verify_work [ 6848.668180] workqueue knbd0-recv: flags=0x1a [ 6848.670541] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6848.674791] in-flight: 657:recv_work [ 6848.678730] workqueue ptlrpc_pinger: flags=0x2 [ 6848.682393] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6848.686142] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 6848.690627] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4889s workers=4 idle: 20831 12642 28859 [ 6848.697280] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6838s workers=2 idle: 659 [ 6860.058354] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 6860.061808] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 6860.066346] Modules linked in: [ 6860.068953] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6860.069012] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6860.069017] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6860.069020] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6860.069032] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6860.069035] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6860.069038] RAX: 0000000000002198 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 6860.069040] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 6860.069042] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6860.069043] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 6860.069045] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 6860.069048] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 6860.069050] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6860.069052] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 6860.069059] Call Trace: [ 6860.069066] [ 6860.069071] ? show_regs.cold.9+0x22/0x2f [ 6860.069078] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6860.069083] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6860.069173] ? watchdog+0x40/0x40 [ 6860.069176] ? hrtimer_interrupt+0x128/0x2c0 [ 6860.069179] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6860.069185] ? apic_timer_interrupt+0xf/0x20 [ 6860.069188] [ 6860.069190] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6860.069195] do_raw_spin_lock+0x177/0x190 [ 6860.069199] _raw_spin_lock+0x21/0x30 [ 6860.069203] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6860.069435] ? do_raw_spin_unlock+0x75/0x190 [ 6860.069439] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6860.069617] ? do_wait_intr+0xf0/0xf0 [ 6860.069624] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6860.073151] lustre(O) [ 6860.104681] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6860.115437] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6860.115512] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6860.115520] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6860.115523] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6860.119671] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6860.120131] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6860.120266] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6860.120311] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6860.120342] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6860.120369] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6860.120455] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6860.123130] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6860.123177] lmv_intent_open+0x338/0x1150 [lmv] [ 6860.123211] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6860.123330] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6860.123426] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6860.123450] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6860.123538] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6860.123621] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6860.123704] ? do_raw_spin_unlock+0x75/0x190 [ 6860.123728] ? _raw_spin_unlock+0x12/0x30 [ 6860.123736] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6860.123767] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6860.123787] ll_atomic_open+0x36a/0xd70 [lustre] [ 6860.125004] lookup_open+0x356/0xb40 [ 6860.125017] do_last+0x313/0xfc0 [ 6860.125022] ? nd_jump_root+0xe5/0x160 [ 6860.125027] ? path_init+0x437/0x520 [ 6860.125032] path_openat+0xf7/0x500 [ 6860.125036] ? do_raw_spin_unlock+0x75/0x190 [ 6860.125042] do_filp_open+0x99/0x140 [ 6860.125048] ? getname_flags+0x6e/0x330 [ 6860.125053] ? __check_object_size+0xff/0x256 [ 6860.125059] ? do_raw_spin_unlock+0x75/0x190 [ 6860.125063] ? _raw_spin_unlock+0x12/0x30 [ 6860.125068] do_sys_openat2+0x2b4/0x410 [ 6860.125072] do_sys_open+0x73/0xa0 [ 6860.125076] __x64_sys_openat+0x24/0x30 [ 6860.125079] do_syscall_64+0xc1/0x440 [ 6860.125086] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6860.125092] RIP: 0033:0x7ff54af45332 [ 6860.125101] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6860.125105] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6860.125110] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 6860.125112] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 6860.125114] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 6860.125116] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 6860.125119] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 6860.664677] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6860.680541] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6860.684473] RAX: 000000000000520a RBX: ffff96848811cae0 RCX: 0000000000000001 [ 6860.687958] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 6860.691532] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6860.694558] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 6860.701947] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 6860.711634] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 6860.722711] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6860.725637] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 6860.736686] Call Trace: [ 6860.740379] [ 6860.740907] ? show_regs.cold.9+0x22/0x2f [ 6860.749791] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6860.753543] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6860.762815] ? watchdog+0x40/0x40 [ 6860.767357] ? hrtimer_interrupt+0x128/0x2c0 [ 6860.774690] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6860.780831] ? apic_timer_interrupt+0xf/0x20 [ 6860.786686] [ 6860.791580] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6860.797522] do_raw_spin_lock+0x177/0x190 [ 6860.805505] _raw_spin_lock+0x21/0x30 [ 6860.812527] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6860.822398] ? do_raw_spin_unlock+0x75/0x190 [ 6860.827547] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6860.831483] ? do_wait_intr+0xf0/0xf0 [ 6860.832588] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6860.836925] mdc_close+0x53b/0xd70 [mdc] [ 6860.842823] lmv_close+0x18d/0x490 [lmv] [ 6860.844972] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 6860.846837] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6860.855142] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6860.859486] ll_md_close+0x274/0x800 [lustre] [ 6860.861129] ll_file_release+0x128/0xc50 [lustre] [ 6860.866447] __fput+0x10d/0x440 [ 6860.867951] ____fput+0x12/0x20 [ 6860.871713] task_work_run+0xc6/0x110 [ 6860.874294] exit_to_usermode_loop+0x1dd/0x1f0 [ 6860.876588] do_syscall_64+0x3d6/0x440 [ 6860.878641] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6860.881684] RIP: 0033:0x7ff54af45cd8 [ 6860.885594] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6860.899768] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6860.903743] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 6860.931850] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 6860.946649] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 6860.952475] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 6860.961493] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6868.059283] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 6868.064031] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6868.122490] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6868.129107] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6868.134617] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6868.138270] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6868.141529] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6868.157447] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6868.163184] RAX: 0000000000006f7b RBX: ffff96848811cae0 RCX: 0000000000000001 [ 6868.166519] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 6868.170007] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 6868.172403] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 6868.177376] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 6868.182637] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 6868.187719] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6868.192476] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 6868.195285] Call Trace: [ 6868.197224] [ 6868.198237] ? show_regs.cold.9+0x22/0x2f [ 6868.199597] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6868.202311] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6868.211106] ? watchdog+0x40/0x40 [ 6868.214504] ? hrtimer_interrupt+0x128/0x2c0 [ 6868.218336] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6868.222258] ? apic_timer_interrupt+0xf/0x20 [ 6868.224355] [ 6868.225275] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6868.229392] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6868.235222] do_raw_spin_lock+0x177/0x190 [ 6868.238443] _raw_spin_lock+0x21/0x30 [ 6868.241454] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6868.246117] process_one_work+0x2c8/0x700 [ 6868.248589] worker_thread+0x296/0x6e0 [ 6868.250308] ? rescuer_thread+0x570/0x570 [ 6868.258450] kthread+0x1d1/0x200 [ 6868.262160] ? set_kthread_struct+0x70/0x70 [ 6868.265614] ret_from_fork+0x1f/0x30 [ 6873.982059] rcu: INFO: rcu_sched self-detected stall on CPU [ 6873.996607] rcu: 0-...!: (2552684 ticks this GP) idle=98a/1/0x4000000000000002 softirq=125154/125154 fqs=959 [ 6874.008629] (t=4920142 jiffies g=253585 q=4376) [ 6874.016498] rcu: rcu_sched kthread starved for 4915102 jiffies! g253585 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [ 6874.023431] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 6874.030518] rcu: RCU grace-period kthread stack dump: [ 6874.033943] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 6874.039321] Call Trace: [ 6874.041499] __schedule+0x351/0xcb0 [ 6874.047171] schedule+0xc0/0x180 [ 6874.051165] schedule_timeout+0xb4/0x190 [ 6874.054343] ? do_raw_spin_unlock+0x75/0x190 [ 6874.059412] ? __next_timer_interrupt+0x160/0x160 [ 6874.063403] ? prepare_to_swait_event+0xf3/0x140 [ 6874.068364] rcu_gp_kthread+0x8dd/0x15b0 [ 6874.071331] ? force_qs_rnp+0x2c0/0x2c0 [ 6874.072550] kthread+0x1d1/0x200 [ 6874.074649] ? set_kthread_struct+0x70/0x70 [ 6874.076956] ret_from_fork+0x1f/0x30 [ 6874.079873] rcu: Stack dump where RCU GP kthread last ran: [ 6874.089379] Sending NMI from CPU 0 to CPUs 2: [ 6874.096736] NMI backtrace for cpu 2 [ 6874.096745] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6874.096757] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6874.096758] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6874.096761] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6874.096763] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6874.096767] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 [ 6874.096768] RAX: 0000000000007c0c RBX: ffff96848811cae0 RCX: 0000000000000001 [ 6874.096770] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 6874.096771] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 6874.096772] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 6874.096774] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 6874.096776] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 6874.096777] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6874.096779] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 6874.096780] Call Trace: [ 6874.096781] [ 6874.096782] ? show_regs.cold.9+0x22/0x2f [ 6874.096783] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 6874.096784] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 6874.096786] ? nmi_handle+0xbd/0x210 [ 6874.096787] ? default_do_nmi+0x4b/0x1b0 [ 6874.096788] ? do_nmi+0x280/0x380 [ 6874.096789] ? end_repeat_nmi+0x16/0x1f [ 6874.096791] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6874.096792] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6874.096793] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6874.096795] [ 6874.096796] do_raw_spin_lock+0x177/0x190 [ 6874.096797] _raw_spin_lock+0x21/0x30 [ 6874.096798] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6874.096799] process_one_work+0x2c8/0x700 [ 6874.096800] worker_thread+0x296/0x6e0 [ 6874.096802] ? rescuer_thread+0x570/0x570 [ 6874.096803] kthread+0x1d1/0x200 [ 6874.096804] ? set_kthread_struct+0x70/0x70 [ 6874.096805] ret_from_fork+0x1f/0x30 [ 6874.098904] NMI backtrace for cpu 0 [ 6874.264949] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6874.271967] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6874.279020] Call Trace: [ 6874.280584] [ 6874.281428] ? dump_stack+0xbb/0x10e [ 6874.283292] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 6874.285465] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 6874.289821] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 6874.292798] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 6874.294102] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 6874.297246] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 6874.299515] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 6874.304318] ? update_process_times+0x6d/0xc0 [ 6874.309927] ? tick_sched_handle.isra.18+0x27/0x90 [ 6874.311630] ? tick_sched_timer+0x53/0xb0 [ 6874.315200] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6874.317376] ? tick_sched_handle.isra.18+0x90/0x90 [ 6874.320435] ? hrtimer_interrupt+0x128/0x2c0 [ 6874.322337] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6874.324495] ? apic_timer_interrupt+0xf/0x20 [ 6874.328043] [ 6874.329033] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6874.334152] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6874.343149] ? do_raw_spin_lock+0x177/0x190 [ 6874.344702] ? _raw_spin_lock+0x21/0x30 [ 6874.351349] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6874.355628] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 6874.364926] ? mdc_commit_open+0x91/0xb0 [mdc] [ 6874.369697] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6874.375138] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6874.381450] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 6874.386692] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6874.393001] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6874.395552] ? do_wait_intr+0xf0/0xf0 [ 6874.398675] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6874.401047] ? kthread+0x1d1/0x200 [ 6874.406499] ? set_kthread_struct+0x70/0x70 [ 6874.408730] ? ret_from_fork+0x1f/0x30 [ 6879.199071] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4920s! [ 6879.204150] Showing busy workqueues and worker pools: [ 6879.207306] workqueue events: flags=0x0 [ 6879.209486] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6879.212987] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 6879.226759] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6879.232184] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 6879.241126] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6879.246812] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 6879.251382] workqueue events_long: flags=0x0 [ 6879.259095] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6879.268708] pending: lu_objects_flush [obdclass] [ 6879.280489] workqueue events_unbound: flags=0x2 [ 6879.298127] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6879.305171] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6879.313312] workqueue events_power_efficient: flags=0x80 [ 6879.315953] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6879.323128] pending: neigh_periodic_work [ 6879.327012] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6879.332433] pending: do_cache_clean, check_lifetime [ 6879.338175] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6879.342377] pending: neigh_periodic_work [ 6879.345072] workqueue writeback: flags=0x4a [ 6879.347711] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6879.351062] pending: wb_workfn, wb_workfn [ 6879.353308] workqueue xprtiod: flags=0xa [ 6879.355643] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6879.357753] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 6879.361470] workqueue ipv6_addrconf: flags=0x40008 [ 6879.364089] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6879.369920] pending: addrconf_verify_work [ 6879.373104] workqueue knbd0-recv: flags=0x1a [ 6879.383189] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6879.385926] in-flight: 657:recv_work [ 6879.389459] workqueue ptlrpc_pinger: flags=0x2 [ 6879.390959] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6879.401213] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 6879.408070] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4920s workers=4 idle: 20831 12642 28859 [ 6879.418219] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6869s workers=2 idle: 659 [ 6888.058050] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [ 6888.063133] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [ 6888.069943] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6888.070048] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6888.070059] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6888.070061] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6888.070075] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6888.070079] RSP: 0018:ffffaf9fc1543ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6888.070083] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 6888.070086] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 6888.070089] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6888.070094] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 6888.070097] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 6888.070100] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 6888.070103] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6888.070105] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 6888.070113] Call Trace: [ 6888.070122] [ 6888.070128] ? show_regs.cold.9+0x22/0x2f [ 6888.070138] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6888.070144] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6888.070149] ? watchdog+0x40/0x40 [ 6888.070154] ? hrtimer_interrupt+0x128/0x2c0 [ 6888.070158] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6888.070165] ? apic_timer_interrupt+0xf/0x20 [ 6888.070170] [ 6888.070173] ? native_safe_halt+0x2e/0x50 [ 6888.070177] kvm_wait+0xcd/0xe0 [ 6888.070185] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6888.070193] do_raw_spin_lock+0x177/0x190 [ 6888.070198] _raw_spin_lock+0x21/0x30 [ 6888.070203] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6888.070497] ? do_raw_spin_unlock+0x75/0x190 [ 6888.070504] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6888.070746] ? do_wait_intr+0xf0/0xf0 [ 6888.070754] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6888.070961] mdc_close+0x53b/0xd70 [mdc] [ 6888.071001] lmv_close+0x18d/0x490 [lmv] [ 6888.071030] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 6888.071219] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6888.071354] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6888.071441] ll_md_close+0x274/0x800 [lustre] [ 6888.071529] ll_file_release+0x128/0xc50 [lustre] [ 6888.071631] __fput+0x10d/0x440 [ 6888.071641] ____fput+0x12/0x20 [ 6888.071645] task_work_run+0xc6/0x110 [ 6888.071653] exit_to_usermode_loop+0x1dd/0x1f0 [ 6888.071661] do_syscall_64+0x3d6/0x440 [ 6888.071667] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6888.071674] RIP: 0033:0x7ff54af45cd8 [ 6888.071680] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6888.071684] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6888.071689] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 6888.071691] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 6888.071694] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 6888.071696] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 6888.071698] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6888.263418] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6888.282002] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6888.287701] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6888.291662] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6888.294442] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6888.307212] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6888.311999] RAX: 0000000000007f90 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 6888.318224] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 6888.322372] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6888.325523] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 6888.334737] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 6888.341160] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 6888.351197] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6888.358830] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 6888.363867] Call Trace: [ 6888.365808] [ 6888.367197] ? show_regs.cold.9+0x22/0x2f [ 6888.370752] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6888.375252] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6888.379368] ? watchdog+0x40/0x40 [ 6888.383032] ? hrtimer_interrupt+0x128/0x2c0 [ 6888.386151] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6888.392902] ? apic_timer_interrupt+0xf/0x20 [ 6888.396995] [ 6888.399447] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6888.401935] do_raw_spin_lock+0x177/0x190 [ 6888.405178] _raw_spin_lock+0x21/0x30 [ 6888.409406] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6888.417401] ? do_raw_spin_unlock+0x75/0x190 [ 6888.421829] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6888.425273] ? do_wait_intr+0xf0/0xf0 [ 6888.427885] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6888.432027] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6888.435907] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6888.440582] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6888.445550] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6888.452762] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6888.458517] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6888.461882] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6888.466902] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6888.471551] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6888.475770] lmv_intent_open+0x338/0x1150 [lmv] [ 6888.479398] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6888.484380] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6888.487568] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6888.490090] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6888.492850] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6888.496082] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6888.498881] ? do_raw_spin_unlock+0x75/0x190 [ 6888.501150] ? _raw_spin_unlock+0x12/0x30 [ 6888.503760] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6888.507309] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6888.510023] ll_atomic_open+0x36a/0xd70 [lustre] [ 6888.512806] lookup_open+0x356/0xb40 [ 6888.514979] do_last+0x313/0xfc0 [ 6888.517216] ? nd_jump_root+0xe5/0x160 [ 6888.519417] ? path_init+0x437/0x520 [ 6888.521221] path_openat+0xf7/0x500 [ 6888.523428] ? do_raw_spin_unlock+0x75/0x190 [ 6888.529091] do_filp_open+0x99/0x140 [ 6888.532021] ? getname_flags+0x6e/0x330 [ 6888.534042] ? __check_object_size+0xff/0x256 [ 6888.539015] ? do_raw_spin_unlock+0x75/0x190 [ 6888.541359] ? _raw_spin_unlock+0x12/0x30 [ 6888.546995] do_sys_openat2+0x2b4/0x410 [ 6888.552977] do_sys_open+0x73/0xa0 [ 6888.557949] __x64_sys_openat+0x24/0x30 [ 6888.559905] do_syscall_64+0xc1/0x440 [ 6888.561956] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6888.565081] RIP: 0033:0x7ff54af45332 [ 6888.568284] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6888.586250] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6888.598080] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 6888.603430] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 6888.610123] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 6888.621402] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 6888.634542] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 6900.059032] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 6900.061548] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6900.076813] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6900.080400] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6900.083428] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6900.085728] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6900.087732] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6900.094166] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 6900.096890] RAX: 00000000000035a8 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 6900.099234] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 6900.101763] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 6900.104186] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 6900.110877] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 6900.118423] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 6900.127415] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6900.129958] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 6900.133551] Call Trace: [ 6900.136287] [ 6900.139091] ? show_regs.cold.9+0x22/0x2f [ 6900.141369] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6900.148628] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6900.151970] ? watchdog+0x40/0x40 [ 6900.152930] ? hrtimer_interrupt+0x128/0x2c0 [ 6900.160117] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6900.167758] ? apic_timer_interrupt+0xf/0x20 [ 6900.174593] [ 6900.175429] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6900.179550] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6900.183851] do_raw_spin_lock+0x177/0x190 [ 6900.185579] _raw_spin_lock+0x21/0x30 [ 6900.187268] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6900.192256] process_one_work+0x2c8/0x700 [ 6900.194803] worker_thread+0x296/0x6e0 [ 6900.196024] ? rescuer_thread+0x570/0x570 [ 6900.198800] kthread+0x1d1/0x200 [ 6900.200424] ? set_kthread_struct+0x70/0x70 [ 6900.201899] ret_from_fork+0x1f/0x30 [ 6904.055169] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 6904.058252] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6904.078600] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6904.082544] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6904.085561] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6904.086888] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6904.092867] RSP: 0000:ffffaf9fc1b97bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6904.095577] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 6904.098279] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 6904.101269] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 6904.104745] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 6904.107554] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6904.111261] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 6904.114609] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6904.116798] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 6904.120549] Call Trace: [ 6904.121715] [ 6904.122727] ? show_regs.cold.9+0x22/0x2f [ 6904.124513] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6904.126655] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6904.128536] ? watchdog+0x40/0x40 [ 6904.129713] ? hrtimer_interrupt+0x128/0x2c0 [ 6904.131530] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6904.133515] ? apic_timer_interrupt+0xf/0x20 [ 6904.135542] [ 6904.136552] ? native_safe_halt+0x2e/0x50 [ 6904.138976] kvm_wait+0xcd/0xe0 [ 6904.143480] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6904.151141] do_raw_spin_lock+0x177/0x190 [ 6904.159329] _raw_spin_lock+0x21/0x30 [ 6904.161953] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6904.173602] obd_mod_free+0x46/0x140 [ptlrpc] [ 6904.177645] mdc_commit_open+0x91/0xb0 [mdc] [ 6904.181610] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6904.184719] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6904.187995] after_reply+0xe2c/0x1780 [ptlrpc] [ 6904.192228] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6904.199384] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6904.201841] ? do_wait_intr+0xf0/0xf0 [ 6904.203787] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6904.205586] kthread+0x1d1/0x200 [ 6904.208683] ? set_kthread_struct+0x70/0x70 [ 6904.212764] ret_from_fork+0x1f/0x30 [ 6909.919083] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4951s! [ 6909.923136] Showing busy workqueues and worker pools: [ 6909.926050] workqueue events: flags=0x0 [ 6909.928468] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6909.933803] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 6909.940649] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6909.945467] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 6909.959213] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6909.973920] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 6909.980025] workqueue events_long: flags=0x0 [ 6909.986699] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6909.997510] pending: lu_objects_flush [obdclass] [ 6910.012288] workqueue events_unbound: flags=0x2 [ 6910.020131] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6910.025619] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6910.043152] workqueue events_power_efficient: flags=0x80 [ 6910.053524] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6910.064498] pending: neigh_periodic_work [ 6910.067696] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6910.072003] pending: do_cache_clean, check_lifetime [ 6910.080419] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6910.085185] pending: neigh_periodic_work [ 6910.090764] workqueue writeback: flags=0x4a [ 6910.094521] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6910.098364] pending: wb_workfn, wb_workfn [ 6910.102366] workqueue xprtiod: flags=0xa [ 6910.104444] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6910.107959] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 6910.111924] workqueue ipv6_addrconf: flags=0x40008 [ 6910.113689] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6910.117678] pending: addrconf_verify_work [ 6910.123234] workqueue knbd0-recv: flags=0x1a [ 6910.125344] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6910.131629] in-flight: 657:recv_work [ 6910.135408] workqueue ptlrpc_pinger: flags=0x2 [ 6910.137578] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6910.139774] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 6910.145667] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4951s workers=4 idle: 20831 12642 28859 [ 6910.150585] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6900s workers=2 idle: 659 [ 6916.058878] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [ 6916.060049] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [ 6916.062599] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6916.062678] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6916.062685] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6916.062688] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6916.062703] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6916.062707] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6916.062712] RAX: 0000000000004f86 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 6916.062714] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 6916.062716] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6916.062717] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 6916.062719] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 6916.062722] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 6916.062724] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6916.062725] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 6916.062733] Call Trace: [ 6916.062738] [ 6916.062744] ? show_regs.cold.9+0x22/0x2f [ 6916.062753] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6916.062757] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6916.062760] ? watchdog+0x40/0x40 [ 6916.062763] ? hrtimer_interrupt+0x128/0x2c0 [ 6916.062766] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6916.062774] ? apic_timer_interrupt+0xf/0x20 [ 6916.062779] [ 6916.062782] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6916.062788] do_raw_spin_lock+0x177/0x190 [ 6916.062793] _raw_spin_lock+0x21/0x30 [ 6916.062798] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6916.063016] ? do_raw_spin_unlock+0x75/0x190 [ 6916.063020] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6916.063148] ? do_wait_intr+0xf0/0xf0 [ 6916.063156] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6916.063317] mdc_close+0x53b/0xd70 [mdc] [ 6916.079334] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs [ 6916.114746] lmv_close+0x18d/0x490 [lmv] [ 6916.114980] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 6916.115317] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6916.115445] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6916.115531] ll_md_close+0x274/0x800 [lustre] [ 6916.115621] ll_file_release+0x128/0xc50 [lustre] [ 6916.115707] __fput+0x10d/0x440 [ 6916.115717] ____fput+0x12/0x20 [ 6916.115723] task_work_run+0xc6/0x110 [ 6916.115730] exit_to_usermode_loop+0x1dd/0x1f0 [ 6916.115738] do_syscall_64+0x3d6/0x440 [ 6916.115744] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6916.115753] RIP: 0033:0x7ff54af45cd8 [ 6916.115761] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6916.115765] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6916.125824] ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror [ 6916.139582] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 6916.139593] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 6916.139596] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 6916.139598] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 6916.139601] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6916.423784] dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6916.429516] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6916.437791] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6916.442071] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6916.444962] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6916.452520] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6916.455406] RAX: 0000000000006c64 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 6916.457815] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 6916.460323] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6916.462745] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 6916.465383] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 6916.467961] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 6916.470943] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6916.473126] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 6916.475991] Call Trace: [ 6916.476920] [ 6916.477840] ? show_regs.cold.9+0x22/0x2f [ 6916.479408] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6916.481464] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6916.483057] ? watchdog+0x40/0x40 [ 6916.484456] ? hrtimer_interrupt+0x128/0x2c0 [ 6916.485900] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6916.487847] ? apic_timer_interrupt+0xf/0x20 [ 6916.489550] [ 6916.490852] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6916.492902] do_raw_spin_lock+0x177/0x190 [ 6916.494805] _raw_spin_lock+0x21/0x30 [ 6916.496116] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6916.503264] ? do_raw_spin_unlock+0x75/0x190 [ 6916.505485] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6916.506857] ? do_wait_intr+0xf0/0xf0 [ 6916.507692] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6916.509876] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6916.512660] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6916.517744] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6916.519939] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6916.525251] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6916.528846] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6916.530392] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6916.533469] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6916.537819] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6916.540706] lmv_intent_open+0x338/0x1150 [lmv] [ 6916.543266] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6916.545010] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6916.548394] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6916.554097] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6916.556924] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6916.562769] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6916.565767] ? do_raw_spin_unlock+0x75/0x190 [ 6916.571152] ? _raw_spin_unlock+0x12/0x30 [ 6916.573040] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6916.578043] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6916.581856] ll_atomic_open+0x36a/0xd70 [lustre] [ 6916.590908] lookup_open+0x356/0xb40 [ 6916.593965] do_last+0x313/0xfc0 [ 6916.595655] ? nd_jump_root+0xe5/0x160 [ 6916.598840] ? path_init+0x437/0x520 [ 6916.600431] path_openat+0xf7/0x500 [ 6916.601795] ? do_raw_spin_unlock+0x75/0x190 [ 6916.603880] do_filp_open+0x99/0x140 [ 6916.605510] ? getname_flags+0x6e/0x330 [ 6916.607189] ? __check_object_size+0xff/0x256 [ 6916.609756] ? do_raw_spin_unlock+0x75/0x190 [ 6916.613386] ? _raw_spin_unlock+0x12/0x30 [ 6916.616522] do_sys_openat2+0x2b4/0x410 [ 6916.617567] do_sys_open+0x73/0xa0 [ 6916.620553] __x64_sys_openat+0x24/0x30 [ 6916.621651] do_syscall_64+0xc1/0x440 [ 6916.623639] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6916.627821] RIP: 0033:0x7ff54af45332 [ 6916.629042] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6916.637395] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6916.640280] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 6916.646681] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 6916.649968] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 6916.657068] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 6916.662230] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 6928.059028] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 6928.062144] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6928.137478] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6928.155968] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6928.166433] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6928.173571] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6928.175240] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6928.211269] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6928.217783] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 6928.228334] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 6928.233845] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 6928.244824] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 6928.252483] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 6928.262531] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 6928.275426] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6928.281719] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 6928.291406] Call Trace: [ 6928.293506] [ 6928.297838] ? show_regs.cold.9+0x22/0x2f [ 6928.300763] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6928.307740] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6928.315355] ? watchdog+0x40/0x40 [ 6928.321475] ? hrtimer_interrupt+0x128/0x2c0 [ 6928.328446] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6928.332516] ? apic_timer_interrupt+0xf/0x20 [ 6928.334494] [ 6928.336860] ? native_safe_halt+0x2e/0x50 [ 6928.341862] kvm_wait+0xcd/0xe0 [ 6928.343984] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6928.352679] do_raw_spin_lock+0x177/0x190 [ 6928.354193] _raw_spin_lock+0x21/0x30 [ 6928.356525] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6928.360521] process_one_work+0x2c8/0x700 [ 6928.362117] worker_thread+0x296/0x6e0 [ 6928.363465] ? rescuer_thread+0x570/0x570 [ 6928.368183] kthread+0x1d1/0x200 [ 6928.375219] ? set_kthread_struct+0x70/0x70 [ 6928.380971] ret_from_fork+0x1f/0x30 [ 6936.055010] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 6936.057646] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6936.114237] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6936.120012] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6936.124714] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6936.127955] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6936.142415] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6936.151101] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 6936.153345] RDX: 00000000000016d7 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 6936.157391] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 6936.160681] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 6936.164655] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6936.168425] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 6936.173101] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6936.176306] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 6936.180453] Call Trace: [ 6936.181623] [ 6936.182558] ? show_regs.cold.9+0x22/0x2f [ 6936.184163] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6936.186063] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6936.189547] ? watchdog+0x40/0x40 [ 6936.196138] ? hrtimer_interrupt+0x128/0x2c0 [ 6936.200814] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6936.205328] ? apic_timer_interrupt+0xf/0x20 [ 6936.210304] [ 6936.212253] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6936.215787] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6936.219308] do_raw_spin_lock+0x177/0x190 [ 6936.223752] _raw_spin_lock+0x21/0x30 [ 6936.225817] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6936.231583] obd_mod_free+0x46/0x140 [ptlrpc] [ 6936.235612] mdc_commit_open+0x91/0xb0 [mdc] [ 6936.237298] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6936.242213] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6936.246762] after_reply+0xe2c/0x1780 [ptlrpc] [ 6936.251656] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6936.254925] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6936.258686] ? do_wait_intr+0xf0/0xf0 [ 6936.260361] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6936.263814] kthread+0x1d1/0x200 [ 6936.267627] ? set_kthread_struct+0x70/0x70 [ 6936.271262] ret_from_fork+0x1f/0x30 [ 6940.640272] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4981s! [ 6940.652485] Showing busy workqueues and worker pools: [ 6940.655647] workqueue events: flags=0x0 [ 6940.659979] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6940.661822] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 6940.670304] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6940.676793] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 6940.688830] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6940.696781] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 6940.715247] workqueue events_long: flags=0x0 [ 6940.726438] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6940.731380] pending: lu_objects_flush [obdclass] [ 6940.740088] workqueue events_unbound: flags=0x2 [ 6940.746340] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6940.756267] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6940.764641] workqueue events_power_efficient: flags=0x80 [ 6940.769572] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6940.776915] pending: neigh_periodic_work [ 6940.785956] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6940.792270] pending: do_cache_clean, check_lifetime [ 6940.797811] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6940.803409] pending: neigh_periodic_work [ 6940.808430] workqueue writeback: flags=0x4a [ 6940.817333] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6940.828317] pending: wb_workfn, wb_workfn [ 6940.836276] workqueue xprtiod: flags=0xa [ 6940.840792] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6940.843990] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 6940.855938] workqueue ipv6_addrconf: flags=0x40008 [ 6940.865789] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6940.874372] pending: addrconf_verify_work [ 6940.880096] workqueue knbd0-recv: flags=0x1a [ 6940.881811] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6940.889341] in-flight: 657:recv_work [ 6940.892534] workqueue ptlrpc_pinger: flags=0x2 [ 6940.893824] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6940.897251] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 6940.901480] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4982s workers=4 idle: 20831 12642 28859 [ 6940.906864] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6931s workers=2 idle: 659 [ 6944.060373] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [ 6944.061805] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [ 6944.068219] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6944.068307] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6944.068314] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6944.068317] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6944.068332] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6944.068336] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6944.068341] RAX: 00000000000011c1 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 6944.068344] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 6944.068346] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6944.068349] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 6944.068351] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 6944.068355] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 6944.068357] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6944.068360] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 6944.068368] Call Trace: [ 6944.068373] [ 6944.068378] ? show_regs.cold.9+0x22/0x2f [ 6944.076403] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6944.076487] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6944.076506] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6944.077324] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6944.077343] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6944.077348] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6944.077354] RAX: 000000000000108e RBX: ffff96848811cae0 RCX: 0000000000000001 [ 6944.077356] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 6944.077359] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6944.077361] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 6944.077363] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 6944.077366] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 6944.077371] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6944.077373] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 6944.077382] Call Trace: [ 6944.077387] [ 6944.077393] ? show_regs.cold.9+0x22/0x2f [ 6944.077402] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6944.077408] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6944.077413] ? watchdog+0x40/0x40 [ 6944.077417] ? hrtimer_interrupt+0x128/0x2c0 [ 6944.077421] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6944.077429] ? apic_timer_interrupt+0xf/0x20 [ 6944.077434] [ 6944.077437] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6944.077444] do_raw_spin_lock+0x177/0x190 [ 6944.077448] _raw_spin_lock+0x21/0x30 [ 6944.077453] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6944.119620] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6944.119654] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6944.119660] ? watchdog+0x40/0x40 [ 6944.119665] ? hrtimer_interrupt+0x128/0x2c0 [ 6944.119670] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6944.119680] ? apic_timer_interrupt+0xf/0x20 [ 6944.119686] [ 6944.119692] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6944.119704] do_raw_spin_lock+0x177/0x190 [ 6944.119710] _raw_spin_lock+0x21/0x30 [ 6944.119716] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6944.127456] ? do_raw_spin_unlock+0x75/0x190 [ 6944.127485] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6944.127820] ? do_wait_intr+0xf0/0xf0 [ 6944.127830] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6944.128043] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6944.128289] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6944.128510] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6944.128647] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6944.128698] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6944.128728] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6944.128757] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6944.128842] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6944.129062] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6944.129109] lmv_intent_open+0x338/0x1150 [lmv] [ 6944.129149] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6944.129244] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6944.129330] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6944.129352] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6944.129438] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6944.129526] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6944.129623] ? do_raw_spin_unlock+0x75/0x190 [ 6944.129631] ? _raw_spin_unlock+0x12/0x30 [ 6944.129639] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6944.129674] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6944.129699] ll_atomic_open+0x36a/0xd70 [lustre] [ 6944.129791] lookup_open+0x356/0xb40 [ 6944.129802] do_last+0x313/0xfc0 [ 6944.129807] ? nd_jump_root+0xe5/0x160 [ 6944.129813] ? path_init+0x437/0x520 [ 6944.129817] path_openat+0xf7/0x500 [ 6944.129822] ? do_raw_spin_unlock+0x75/0x190 [ 6944.129827] do_filp_open+0x99/0x140 [ 6944.129834] ? getname_flags+0x6e/0x330 [ 6944.129838] ? __check_object_size+0xff/0x256 [ 6944.129847] ? do_raw_spin_unlock+0x75/0x190 [ 6944.129851] ? _raw_spin_unlock+0x12/0x30 [ 6944.129856] do_sys_openat2+0x2b4/0x410 [ 6944.129861] do_sys_open+0x73/0xa0 [ 6944.129866] __x64_sys_openat+0x24/0x30 [ 6944.129870] do_syscall_64+0xc1/0x440 [ 6944.129877] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6944.129884] RIP: 0033:0x7ff54af45332 [ 6944.129895] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6944.129900] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6944.129906] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 6944.129909] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 6944.129912] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 6944.129914] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 6944.129917] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 6944.600718] ? do_raw_spin_unlock+0x75/0x190 [ 6944.602170] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6944.620703] ? do_wait_intr+0xf0/0xf0 [ 6944.621867] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6944.631446] mdc_close+0x53b/0xd70 [mdc] [ 6944.635374] lmv_close+0x18d/0x490 [lmv] [ 6944.648469] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 6944.652384] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6944.656976] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6944.659946] ll_md_close+0x274/0x800 [lustre] [ 6944.662582] ll_file_release+0x128/0xc50 [lustre] [ 6944.666244] __fput+0x10d/0x440 [ 6944.668872] ____fput+0x12/0x20 [ 6944.670384] task_work_run+0xc6/0x110 [ 6944.671728] exit_to_usermode_loop+0x1dd/0x1f0 [ 6944.674548] do_syscall_64+0x3d6/0x440 [ 6944.676154] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6944.678687] RIP: 0033:0x7ff54af45cd8 [ 6944.680192] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6944.689716] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6944.694258] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 6944.697549] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 6944.701799] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 6944.705636] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 6944.709489] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6956.060901] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 6956.066119] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6956.120917] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6956.134083] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6956.158403] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6956.160812] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6956.163342] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6956.170654] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6956.173658] RAX: 0000000000000c14 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 6956.176374] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 6956.178964] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 6956.182148] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 6956.185119] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 6956.188041] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 6956.191418] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6956.193584] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 6956.196441] Call Trace: [ 6956.197399] [ 6956.198460] ? show_regs.cold.9+0x22/0x2f [ 6956.199752] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6956.201986] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6956.203750] ? watchdog+0x40/0x40 [ 6956.206072] ? hrtimer_interrupt+0x128/0x2c0 [ 6956.208059] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6956.210525] ? apic_timer_interrupt+0xf/0x20 [ 6956.212330] [ 6956.213210] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6956.216470] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6956.219627] do_raw_spin_lock+0x177/0x190 [ 6956.221504] _raw_spin_lock+0x21/0x30 [ 6956.223051] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6956.225568] process_one_work+0x2c8/0x700 [ 6956.227810] worker_thread+0x296/0x6e0 [ 6956.229312] ? rescuer_thread+0x570/0x570 [ 6956.232502] kthread+0x1d1/0x200 [ 6956.233901] ? set_kthread_struct+0x70/0x70 [ 6956.235831] ret_from_fork+0x1f/0x30 [ 6968.056953] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 6968.068145] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6968.094539] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6968.101489] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6968.108898] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6968.111022] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6968.122059] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6968.126675] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 6968.131566] RDX: 00000000000050a9 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 6968.135336] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 6968.139712] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 6968.143527] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6968.148281] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 6968.157648] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6968.161474] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 6968.163890] Call Trace: [ 6968.164624] [ 6968.167575] ? show_regs.cold.9+0x22/0x2f [ 6968.168988] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6968.172137] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6968.174802] ? watchdog+0x40/0x40 [ 6968.176746] ? hrtimer_interrupt+0x128/0x2c0 [ 6968.179062] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6968.181784] ? apic_timer_interrupt+0xf/0x20 [ 6968.184076] [ 6968.185168] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6968.188047] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6968.192020] do_raw_spin_lock+0x177/0x190 [ 6968.196154] _raw_spin_lock+0x21/0x30 [ 6968.198506] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6968.205061] obd_mod_free+0x46/0x140 [ptlrpc] [ 6968.211249] mdc_commit_open+0x91/0xb0 [mdc] [ 6968.216175] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6968.221161] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6968.224702] after_reply+0xe2c/0x1780 [ptlrpc] [ 6968.230267] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6968.235238] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6968.238238] ? do_wait_intr+0xf0/0xf0 [ 6968.240918] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6968.244980] kthread+0x1d1/0x200 [ 6968.246872] ? set_kthread_struct+0x70/0x70 [ 6968.248364] ret_from_fork+0x1f/0x30 [ 6971.362673] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5012s! [ 6971.369406] Showing busy workqueues and worker pools: [ 6971.374321] workqueue events: flags=0x0 [ 6971.379813] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6971.383963] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 6971.389166] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6971.397312] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 6971.404214] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6971.408375] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 6971.413246] workqueue events_long: flags=0x0 [ 6971.415360] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6971.418814] pending: lu_objects_flush [obdclass] [ 6971.420895] workqueue events_unbound: flags=0x2 [ 6971.423012] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6971.427511] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6971.433155] workqueue events_power_efficient: flags=0x80 [ 6971.435759] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6971.438989] pending: neigh_periodic_work [ 6971.441001] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6971.448034] pending: do_cache_clean, check_lifetime [ 6971.453829] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6971.459746] pending: neigh_periodic_work [ 6971.461843] workqueue writeback: flags=0x4a [ 6971.464993] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6971.475204] pending: wb_workfn, wb_workfn [ 6971.480032] workqueue xprtiod: flags=0xa [ 6971.485593] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6971.496786] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 6971.504014] workqueue ipv6_addrconf: flags=0x40008 [ 6971.508624] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6971.514079] pending: addrconf_verify_work [ 6971.516232] workqueue knbd0-recv: flags=0x1a [ 6971.518511] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6971.526692] in-flight: 657:recv_work [ 6971.539921] workqueue ptlrpc_pinger: flags=0x2 [ 6971.555410] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6971.563793] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 6971.574570] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5012s workers=4 idle: 20831 12642 28859 [ 6971.590052] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6961s workers=2 idle: 659 [ 6972.061460] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [ 6972.063424] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [ 6972.068254] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6972.068618] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6972.074955] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6972.075042] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6972.075052] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6972.075055] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6972.075076] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6972.075080] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 6972.075085] RAX: 0000000000006f01 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 6972.075088] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 6972.075090] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6972.075092] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 6972.075094] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 6972.075097] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 6972.075100] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6972.075102] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 6972.075111] Call Trace: [ 6972.075116] [ 6972.075122] ? show_regs.cold.9+0x22/0x2f [ 6972.075131] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6972.075137] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6972.075142] ? watchdog+0x40/0x40 [ 6972.075146] ? hrtimer_interrupt+0x128/0x2c0 [ 6972.075151] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6972.075159] ? apic_timer_interrupt+0xf/0x20 [ 6972.075163] [ 6972.075166] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6972.075173] do_raw_spin_lock+0x177/0x190 [ 6972.076110] _raw_spin_lock+0x21/0x30 [ 6972.076117] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6972.076419] ? do_raw_spin_unlock+0x75/0x190 [ 6972.076425] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6972.076624] ? do_wait_intr+0xf0/0xf0 [ 6972.076632] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6972.076853] mdc_close+0x53b/0xd70 [mdc] [ 6972.076902] lmv_close+0x18d/0x490 [lmv] [ 6972.076933] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 6972.077106] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6972.077232] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6972.077317] ll_md_close+0x274/0x800 [lustre] [ 6972.077416] ll_file_release+0x128/0xc50 [lustre] [ 6972.077496] __fput+0x10d/0x440 [ 6972.077508] ____fput+0x12/0x20 [ 6972.077511] task_work_run+0xc6/0x110 [ 6972.077518] exit_to_usermode_loop+0x1dd/0x1f0 [ 6972.077525] do_syscall_64+0x3d6/0x440 [ 6972.077529] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6972.077537] RIP: 0033:0x7ff54af45cd8 [ 6972.077545] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6972.077549] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6972.077554] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 6972.077556] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 6972.077559] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 6972.077561] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 6972.077564] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6972.546343] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6972.553436] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6972.555511] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6972.574683] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6972.578302] RAX: 000000000000240a RBX: ffff96848811cae0 RCX: 0000000000000001 [ 6972.586734] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 6972.595212] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6972.604471] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 6972.612393] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 6972.619363] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 6972.625379] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6972.634302] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 6972.640372] Call Trace: [ 6972.641268] [ 6972.654166] ? show_regs.cold.9+0x22/0x2f [ 6972.659899] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6972.668103] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6972.669567] ? watchdog+0x40/0x40 [ 6972.680125] ? hrtimer_interrupt+0x128/0x2c0 [ 6972.681263] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6972.686216] ? apic_timer_interrupt+0xf/0x20 [ 6972.688528] [ 6972.691893] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6972.703959] do_raw_spin_lock+0x177/0x190 [ 6972.707881] _raw_spin_lock+0x21/0x30 [ 6972.711261] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6972.717604] ? do_raw_spin_unlock+0x75/0x190 [ 6972.721543] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6972.727004] ? do_wait_intr+0xf0/0xf0 [ 6972.731786] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6972.744169] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6972.747552] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6972.751457] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6972.756371] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6972.760997] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6972.764370] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6972.773435] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6972.776720] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6972.783549] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6972.792000] lmv_intent_open+0x338/0x1150 [lmv] [ 6972.799892] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6972.809255] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6972.811822] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6972.813522] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6972.821175] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6972.824864] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6972.827810] ? do_raw_spin_unlock+0x75/0x190 [ 6972.830462] ? _raw_spin_unlock+0x12/0x30 [ 6972.831779] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6972.837269] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6972.839623] ll_atomic_open+0x36a/0xd70 [lustre] [ 6972.842369] lookup_open+0x356/0xb40 [ 6972.844847] do_last+0x313/0xfc0 [ 6972.846715] ? nd_jump_root+0xe5/0x160 [ 6972.849737] ? path_init+0x437/0x520 [ 6972.857636] path_openat+0xf7/0x500 [ 6972.861299] ? do_raw_spin_unlock+0x75/0x190 [ 6972.865486] do_filp_open+0x99/0x140 [ 6972.866762] ? getname_flags+0x6e/0x330 [ 6972.876365] ? __check_object_size+0xff/0x256 [ 6972.881056] ? do_raw_spin_unlock+0x75/0x190 [ 6972.885174] ? _raw_spin_unlock+0x12/0x30 [ 6972.891313] do_sys_openat2+0x2b4/0x410 [ 6972.900331] do_sys_open+0x73/0xa0 [ 6972.903585] __x64_sys_openat+0x24/0x30 [ 6972.905603] do_syscall_64+0xc1/0x440 [ 6972.907913] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6972.913481] RIP: 0033:0x7ff54af45332 [ 6972.921384] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6972.936964] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6972.941749] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 6972.948124] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 6972.956341] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 6972.964310] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 6972.970705] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 6984.059080] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 6984.065058] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6984.106982] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6984.115436] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6984.123274] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6984.128706] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6984.133970] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6984.152205] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6984.158930] RAX: 0000000000007b1a RBX: ffff96848811cae0 RCX: 0000000000000001 [ 6984.165235] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 6984.170719] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 6984.175904] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 6984.179566] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 6984.184068] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 6984.188654] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6984.192478] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 6984.196759] Call Trace: [ 6984.198280] [ 6984.200076] ? show_regs.cold.9+0x22/0x2f [ 6984.202729] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6984.207806] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6984.212310] ? watchdog+0x40/0x40 [ 6984.214283] ? hrtimer_interrupt+0x128/0x2c0 [ 6984.218078] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6984.221676] ? apic_timer_interrupt+0xf/0x20 [ 6984.224960] [ 6984.226180] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6984.231027] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6984.235004] do_raw_spin_lock+0x177/0x190 [ 6984.238388] _raw_spin_lock+0x21/0x30 [ 6984.241076] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6984.245577] process_one_work+0x2c8/0x700 [ 6984.248437] worker_thread+0x296/0x6e0 [ 6984.251344] ? rescuer_thread+0x570/0x570 [ 6984.254450] kthread+0x1d1/0x200 [ 6984.256863] ? set_kthread_struct+0x70/0x70 [ 6984.261026] ret_from_fork+0x1f/0x30 [ 6996.055542] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 6996.064848] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6996.146260] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6996.175309] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6996.200662] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6996.210231] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6996.237633] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6996.248889] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 6996.256314] RDX: 0000000000000bc7 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 6996.262417] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 6996.271504] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 6996.288220] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6996.293356] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 6996.302088] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6996.309122] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 6996.318278] Call Trace: [ 6996.324086] [ 6996.326110] ? show_regs.cold.9+0x22/0x2f [ 6996.329622] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6996.339292] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6996.342177] ? watchdog+0x40/0x40 [ 6996.344088] ? hrtimer_interrupt+0x128/0x2c0 [ 6996.347856] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6996.358321] ? apic_timer_interrupt+0xf/0x20 [ 6996.362363] [ 6996.368925] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6996.376471] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6996.389369] do_raw_spin_lock+0x177/0x190 [ 6996.391706] _raw_spin_lock+0x21/0x30 [ 6996.400444] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6996.406191] obd_mod_free+0x46/0x140 [ptlrpc] [ 6996.408200] mdc_commit_open+0x91/0xb0 [mdc] [ 6996.414922] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6996.419577] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6996.429982] after_reply+0xe2c/0x1780 [ptlrpc] [ 6996.436170] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6996.439733] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6996.443329] ? do_wait_intr+0xf0/0xf0 [ 6996.445235] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6996.448445] kthread+0x1d1/0x200 [ 6996.451905] ? set_kthread_struct+0x70/0x70 [ 6996.454060] ret_from_fork+0x1f/0x30 [ 7000.059276] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [ 7000.061029] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [ 7000.064236] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7000.064316] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7000.064324] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7000.064327] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7000.064342] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7000.064347] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7000.064620] RAX: 0000000000007f0f RBX: ffff96848811cae0 RCX: 0000000000000001 [ 7000.064624] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 7000.064625] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7000.064627] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 7000.064629] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 7000.064633] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 7000.064696] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7000.064699] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 7000.064707] Call Trace: [ 7000.064714] [ 7000.064720] ? show_regs.cold.9+0x22/0x2f [ 7000.064731] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7000.064738] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7000.064742] ? watchdog+0x40/0x40 [ 7000.064746] ? hrtimer_interrupt+0x128/0x2c0 [ 7000.064750] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7000.064758] ? apic_timer_interrupt+0xf/0x20 [ 7000.064763] [ 7000.064766] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7000.064781] do_raw_spin_lock+0x177/0x190 [ 7000.064786] _raw_spin_lock+0x21/0x30 [ 7000.064799] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7000.082711] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7000.082788] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7000.082797] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7000.082802] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7000.136211] ? do_raw_spin_unlock+0x75/0x190 [ 7000.136858] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7000.142081] ? do_wait_intr+0xf0/0xf0 [ 7000.142095] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7000.144700] mdc_close+0x53b/0xd70 [mdc] [ 7000.144748] lmv_close+0x18d/0x490 [lmv] [ 7000.144787] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 7000.151771] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7000.151787] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7000.151794] RAX: 0000000000003a88 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 7000.151797] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 7000.151799] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7000.151801] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 7000.151804] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 7000.151807] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 7000.151809] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7000.151812] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 7000.151819] Call Trace: [ 7000.151826] [ 7000.151832] ? show_regs.cold.9+0x22/0x2f [ 7000.151846] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7000.151853] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7000.151858] ? watchdog+0x40/0x40 [ 7000.151863] ? hrtimer_interrupt+0x128/0x2c0 [ 7000.151867] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7000.151875] ? apic_timer_interrupt+0xf/0x20 [ 7000.151880] [ 7000.151883] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7000.151892] do_raw_spin_lock+0x177/0x190 [ 7000.151897] _raw_spin_lock+0x21/0x30 [ 7000.151902] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7000.152274] ? do_raw_spin_unlock+0x75/0x190 [ 7000.152280] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7000.152788] ? do_wait_intr+0xf0/0xf0 [ 7000.152803] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7000.153108] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7000.153425] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7000.157429] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7000.157615] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7000.157703] ll_md_close+0x274/0x800 [lustre] [ 7000.157781] ll_file_release+0x128/0xc50 [lustre] [ 7000.157873] __fput+0x10d/0x440 [ 7000.157884] ____fput+0x12/0x20 [ 7000.157888] task_work_run+0xc6/0x110 [ 7000.157896] exit_to_usermode_loop+0x1dd/0x1f0 [ 7000.157905] do_syscall_64+0x3d6/0x440 [ 7000.157910] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7000.157924] RIP: 0033:0x7ff54af45cd8 [ 7000.157932] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7000.157936] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7000.157941] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 7000.157944] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 7000.157946] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 7000.157949] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 7000.157951] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7000.556824] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7000.562791] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7000.575980] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7000.578470] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7000.583054] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7000.587224] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7000.591658] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7000.594700] lmv_intent_open+0x338/0x1150 [lmv] [ 7000.598687] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7000.610916] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7000.613836] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7000.616106] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7000.619132] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7000.622938] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7000.632756] ? do_raw_spin_unlock+0x75/0x190 [ 7000.636443] ? _raw_spin_unlock+0x12/0x30 [ 7000.638061] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7000.644714] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7000.674692] ll_atomic_open+0x36a/0xd70 [lustre] [ 7000.692463] lookup_open+0x356/0xb40 [ 7000.695169] do_last+0x313/0xfc0 [ 7000.697950] ? nd_jump_root+0xe5/0x160 [ 7000.698967] ? path_init+0x437/0x520 [ 7000.705290] path_openat+0xf7/0x500 [ 7000.712125] ? do_raw_spin_unlock+0x75/0x190 [ 7000.723345] do_filp_open+0x99/0x140 [ 7000.724790] ? getname_flags+0x6e/0x330 [ 7000.727727] ? __check_object_size+0xff/0x256 [ 7000.731014] ? do_raw_spin_unlock+0x75/0x190 [ 7000.732262] ? _raw_spin_unlock+0x12/0x30 [ 7000.736773] do_sys_openat2+0x2b4/0x410 [ 7000.738793] do_sys_open+0x73/0xa0 [ 7000.742087] __x64_sys_openat+0x24/0x30 [ 7000.744118] do_syscall_64+0xc1/0x440 [ 7000.745134] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7000.747781] RIP: 0033:0x7ff54af45332 [ 7000.758782] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7000.776489] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7000.784295] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 7000.786272] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 7000.788414] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 7000.790253] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 7000.792051] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 7002.079038] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5043s! [ 7002.082170] Showing busy workqueues and worker pools: [ 7002.084114] workqueue events: flags=0x0 [ 7002.085762] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7002.088688] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 7002.095892] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7002.106869] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 7002.116764] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7002.137748] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 7002.155282] workqueue events_long: flags=0x0 [ 7002.159579] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7002.176578] pending: lu_objects_flush [obdclass] [ 7002.181446] workqueue events_unbound: flags=0x2 [ 7002.187754] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7002.202503] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7002.219290] workqueue events_power_efficient: flags=0x80 [ 7002.224499] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7002.231345] pending: neigh_periodic_work [ 7002.236811] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7002.240887] pending: do_cache_clean, check_lifetime [ 7002.244576] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7002.248767] pending: neigh_periodic_work [ 7002.252618] workqueue writeback: flags=0x4a [ 7002.255685] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7002.257992] pending: wb_workfn, wb_workfn [ 7002.262632] workqueue xprtiod: flags=0xa [ 7002.266528] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7002.269794] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 7002.278971] workqueue ipv6_addrconf: flags=0x40008 [ 7002.284556] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7002.286883] pending: addrconf_verify_work [ 7002.289926] workqueue knbd0-recv: flags=0x1a [ 7002.292025] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7002.296165] in-flight: 657:recv_work [ 7002.299757] workqueue ptlrpc_pinger: flags=0x2 [ 7002.305960] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7002.314195] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 7002.319757] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5043s workers=4 idle: 20831 12642 28859 [ 7002.325108] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6992s workers=2 idle: 659 [ 7012.061170] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 7012.074961] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7012.171224] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7012.192476] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7012.203559] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7012.206549] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7012.209694] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7012.217192] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7012.220187] RAX: 0000000000006c0e RBX: ffff96848811cae0 RCX: 0000000000000001 [ 7012.223251] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 7012.226171] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 7012.229340] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 7012.234198] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 7012.251522] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 7012.257776] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7012.262574] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 7012.270983] Call Trace: [ 7012.274291] [ 7012.275618] ? show_regs.cold.9+0x22/0x2f [ 7012.280328] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7012.283838] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7012.286824] ? watchdog+0x40/0x40 [ 7012.289317] ? hrtimer_interrupt+0x128/0x2c0 [ 7012.292176] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7012.294512] ? apic_timer_interrupt+0xf/0x20 [ 7012.299176] [ 7012.299932] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7012.301877] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7012.306527] do_raw_spin_lock+0x177/0x190 [ 7012.307695] _raw_spin_lock+0x21/0x30 [ 7012.309843] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7012.312906] process_one_work+0x2c8/0x700 [ 7012.318888] worker_thread+0x296/0x6e0 [ 7012.320071] ? rescuer_thread+0x570/0x570 [ 7012.321367] kthread+0x1d1/0x200 [ 7012.322501] ? set_kthread_struct+0x70/0x70 [ 7012.323993] ret_from_fork+0x1f/0x30 [ 7028.055113] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 7028.066478] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 7028.071867] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 7028.071887] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7028.072086] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7028.072095] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7028.072099] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7028.072119] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7028.072123] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7028.072128] RAX: 0000000000007d86 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 7028.072130] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 7028.072133] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7028.072135] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 7028.072138] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 7028.072141] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 7028.072144] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7028.072146] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 7028.072154] Call Trace: [ 7028.072159] [ 7028.072165] ? show_regs.cold.9+0x22/0x2f [ 7028.072175] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7028.072181] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7028.072185] ? watchdog+0x40/0x40 [ 7028.072190] ? hrtimer_interrupt+0x128/0x2c0 [ 7028.072195] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7028.072203] ? apic_timer_interrupt+0xf/0x20 [ 7028.072208] [ 7028.072212] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7028.073503] do_raw_spin_lock+0x177/0x190 [ 7028.073511] _raw_spin_lock+0x21/0x30 [ 7028.073517] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7028.076838] ? do_raw_spin_unlock+0x75/0x190 [ 7028.076849] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7028.078446] ? do_wait_intr+0xf0/0xf0 [ 7028.078456] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7028.080846] mdc_close+0x53b/0xd70 [mdc] [ 7028.080895] lmv_close+0x18d/0x490 [lmv] [ 7028.080944] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 7028.081378] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7028.081474] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7028.081481] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7028.081491] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7028.081504] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7028.081508] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7028.081513] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 7028.081516] RDX: 0000000000007d23 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 7028.081518] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 7028.081520] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 7028.081523] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7028.081526] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 7028.081529] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7028.081531] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 7028.081539] Call Trace: [ 7028.081544] [ 7028.081547] ? show_regs.cold.9+0x22/0x2f [ 7028.081567] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7028.081574] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7028.081580] ? watchdog+0x40/0x40 [ 7028.081584] ? hrtimer_interrupt+0x128/0x2c0 [ 7028.081589] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7028.081598] ? apic_timer_interrupt+0xf/0x20 [ 7028.081603] [ 7028.081606] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7028.081610] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7028.081615] do_raw_spin_lock+0x177/0x190 [ 7028.081619] _raw_spin_lock+0x21/0x30 [ 7028.081625] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7028.081928] obd_mod_free+0x46/0x140 [ptlrpc] [ 7028.082295] mdc_commit_open+0x91/0xb0 [mdc] [ 7028.082327] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7028.083196] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7028.083953] after_reply+0xe2c/0x1780 [ptlrpc] [ 7028.084167] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7028.084395] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7028.085397] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7028.085757] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7028.085846] ll_md_close+0x274/0x800 [lustre] [ 7028.087490] ll_file_release+0x128/0xc50 [lustre] [ 7028.087612] __fput+0x10d/0x440 [ 7028.087624] ____fput+0x12/0x20 [ 7028.087630] task_work_run+0xc6/0x110 [ 7028.087638] exit_to_usermode_loop+0x1dd/0x1f0 [ 7028.087647] do_syscall_64+0x3d6/0x440 [ 7028.087653] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7028.087866] RIP: 0033:0x7ff54af45cd8 [ 7028.087875] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7028.087879] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7028.087885] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 7028.087887] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 7028.087890] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 7028.087892] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 7028.087894] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7028.097512] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7028.098845] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7028.098853] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7028.098856] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7028.098870] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7028.098875] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7028.099010] RAX: 0000000000007d90 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 7028.099013] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 7028.099016] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7028.099018] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 7028.099020] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 7028.099023] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 7028.099026] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7028.099028] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 7028.099036] Call Trace: [ 7028.099041] [ 7028.099047] ? show_regs.cold.9+0x22/0x2f [ 7028.099057] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7028.099064] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7028.099068] ? watchdog+0x40/0x40 [ 7028.099073] ? hrtimer_interrupt+0x128/0x2c0 [ 7028.099078] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7028.100390] ? apic_timer_interrupt+0xf/0x20 [ 7028.100396] [ 7028.100400] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7028.100410] do_raw_spin_lock+0x177/0x190 [ 7028.100416] _raw_spin_lock+0x21/0x30 [ 7028.100421] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7028.101888] ? do_raw_spin_unlock+0x75/0x190 [ 7028.101897] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7028.122431] ? do_wait_intr+0xf0/0xf0 [ 7028.122456] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7028.122797] kthread+0x1d1/0x200 [ 7028.122804] ? set_kthread_struct+0x70/0x70 [ 7028.122808] ret_from_fork+0x1f/0x30 [ 7029.076740] ? do_wait_intr+0xf0/0xf0 [ 7029.079756] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7029.082800] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7029.086411] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7029.091673] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7029.096601] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7029.099389] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7029.102501] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7029.104657] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7029.108674] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7029.112421] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7029.115719] lmv_intent_open+0x338/0x1150 [lmv] [ 7029.117803] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7029.121765] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7029.127703] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7029.129671] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7029.132523] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7029.135396] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7029.138067] ? do_raw_spin_unlock+0x75/0x190 [ 7029.140789] ? _raw_spin_unlock+0x12/0x30 [ 7029.144049] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7029.146972] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7029.149286] ll_atomic_open+0x36a/0xd70 [lustre] [ 7029.151433] lookup_open+0x356/0xb40 [ 7029.154894] do_last+0x313/0xfc0 [ 7029.156492] ? nd_jump_root+0xe5/0x160 [ 7029.158508] ? path_init+0x437/0x520 [ 7029.160591] path_openat+0xf7/0x500 [ 7029.162316] ? do_raw_spin_unlock+0x75/0x190 [ 7029.164664] do_filp_open+0x99/0x140 [ 7029.167345] ? getname_flags+0x6e/0x330 [ 7029.169439] ? __check_object_size+0xff/0x256 [ 7029.172204] ? do_raw_spin_unlock+0x75/0x190 [ 7029.174410] ? _raw_spin_unlock+0x12/0x30 [ 7029.176527] do_sys_openat2+0x2b4/0x410 [ 7029.178504] do_sys_open+0x73/0xa0 [ 7029.180001] __x64_sys_openat+0x24/0x30 [ 7029.181730] do_syscall_64+0xc1/0x440 [ 7029.184074] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7029.185936] RIP: 0033:0x7ff54af45332 [ 7029.187488] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7029.197513] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7029.202865] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 7029.206605] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 7029.209853] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 7029.213397] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 7029.216622] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 7032.799038] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5073s! [ 7032.801559] Showing busy workqueues and worker pools: [ 7032.802996] workqueue events: flags=0x0 [ 7032.804297] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7032.806381] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 7032.809294] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7032.811377] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 7032.815203] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7032.817645] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 7032.819897] workqueue events_long: flags=0x0 [ 7032.821317] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7032.823387] pending: lu_objects_flush [obdclass] [ 7032.825087] workqueue events_unbound: flags=0x2 [ 7032.826751] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7032.828891] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7032.832548] workqueue events_power_efficient: flags=0x80 [ 7032.834382] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7032.836914] pending: neigh_periodic_work [ 7032.838318] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7032.840554] pending: do_cache_clean, check_lifetime [ 7032.842164] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7032.844694] pending: neigh_periodic_work [ 7032.846288] workqueue writeback: flags=0x4a [ 7032.847702] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7032.850092] pending: wb_workfn, wb_workfn [ 7032.851520] workqueue xprtiod: flags=0xa [ 7032.852875] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7032.854722] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 7032.857126] workqueue ipv6_addrconf: flags=0x40008 [ 7032.858575] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7032.860572] pending: addrconf_verify_work [ 7032.861891] workqueue knbd0-recv: flags=0x1a [ 7032.863192] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7032.865601] in-flight: 657:recv_work [ 7032.866825] workqueue ptlrpc_pinger: flags=0x2 [ 7032.868462] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7032.870748] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 7032.873123] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5073s workers=4 idle: 20831 12642 28859 [ 7032.876624] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7023s workers=2 idle: 659 [ 7040.061123] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 7040.077682] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7040.195274] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7040.230276] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7040.244420] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7040.254337] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7040.261745] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7040.287226] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7040.303706] RAX: 0000000000004909 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 7040.318322] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 7040.331378] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 7040.333817] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 7040.345970] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 7040.360679] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 7040.377197] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7040.388705] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 7040.393947] Call Trace: [ 7040.404257] [ 7040.406686] ? show_regs.cold.9+0x22/0x2f [ 7040.415791] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7040.422945] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7040.430815] ? watchdog+0x40/0x40 [ 7040.451078] ? hrtimer_interrupt+0x128/0x2c0 [ 7040.457499] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7040.464618] ? apic_timer_interrupt+0xf/0x20 [ 7040.466024] [ 7040.466735] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7040.491322] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7040.508869] do_raw_spin_lock+0x177/0x190 [ 7040.518186] _raw_spin_lock+0x21/0x30 [ 7040.534258] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7040.541787] process_one_work+0x2c8/0x700 [ 7040.546551] worker_thread+0x296/0x6e0 [ 7040.549932] ? rescuer_thread+0x570/0x570 [ 7040.551208] kthread+0x1d1/0x200 [ 7040.555776] ? set_kthread_struct+0x70/0x70 [ 7040.560771] ret_from_fork+0x1f/0x30 [ 7053.996968] rcu: INFO: rcu_sched self-detected stall on CPU [ 7054.014336] rcu: 0-...!: (2651462 ticks this GP) idle=98a/1/0x4000000000000002 softirq=125154/125154 fqs=959 [ 7054.024344] (t=5100156 jiffies g=253585 q=4452) [ 7054.032425] rcu: rcu_sched kthread starved for 5095116 jiffies! g253585 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [ 7054.045584] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 7054.066987] rcu: RCU grace-period kthread stack dump: [ 7054.072708] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 7054.097436] Call Trace: [ 7054.100511] __schedule+0x351/0xcb0 [ 7054.103333] schedule+0xc0/0x180 [ 7054.108818] schedule_timeout+0xb4/0x190 [ 7054.116668] ? do_raw_spin_unlock+0x75/0x190 [ 7054.119366] ? __next_timer_interrupt+0x160/0x160 [ 7054.124453] ? prepare_to_swait_event+0xf3/0x140 [ 7054.137202] rcu_gp_kthread+0x8dd/0x15b0 [ 7054.153241] ? force_qs_rnp+0x2c0/0x2c0 [ 7054.161241] kthread+0x1d1/0x200 [ 7054.169733] ? set_kthread_struct+0x70/0x70 [ 7054.182902] ret_from_fork+0x1f/0x30 [ 7054.192034] rcu: Stack dump where RCU GP kthread last ran: [ 7054.207761] Sending NMI from CPU 0 to CPUs 2: [ 7054.215092] NMI backtrace for cpu 2 [ 7054.215101] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7054.215103] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7054.215105] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7054.215109] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7054.215111] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7054.215113] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 [ 7054.215115] RAX: 0000000000002aed RBX: ffff96848811cae0 RCX: 0000000000000001 [ 7054.215117] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 7054.215119] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 7054.215120] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 7054.215122] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 7054.215124] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 7054.215125] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7054.215126] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 7054.215127] Call Trace: [ 7054.215128] [ 7054.215129] ? show_regs.cold.9+0x22/0x2f [ 7054.215130] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 7054.215131] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 7054.215132] ? nmi_handle+0xbd/0x210 [ 7054.215133] ? default_do_nmi+0x4b/0x1b0 [ 7054.215134] ? do_nmi+0x280/0x380 [ 7054.215135] ? end_repeat_nmi+0x16/0x1f [ 7054.215137] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7054.215138] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7054.215139] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7054.215140] [ 7054.215141] do_raw_spin_lock+0x177/0x190 [ 7054.215143] _raw_spin_lock+0x21/0x30 [ 7054.215144] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7054.215145] process_one_work+0x2c8/0x700 [ 7054.215147] worker_thread+0x296/0x6e0 [ 7054.215148] ? rescuer_thread+0x570/0x570 [ 7054.215149] kthread+0x1d1/0x200 [ 7054.215151] ? set_kthread_struct+0x70/0x70 [ 7054.215152] ret_from_fork+0x1f/0x30 [ 7054.227347] NMI backtrace for cpu 0 [ 7054.380602] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7054.386718] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7054.391069] Call Trace: [ 7054.392110] [ 7054.393016] ? dump_stack+0xbb/0x10e [ 7054.394309] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 7054.397504] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 7054.399799] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 7054.401521] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 7054.404222] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 7054.406411] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 7054.408632] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 7054.415855] ? update_process_times+0x6d/0xc0 [ 7054.418058] ? tick_sched_handle.isra.18+0x27/0x90 [ 7054.421071] ? tick_sched_timer+0x53/0xb0 [ 7054.422691] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7054.425499] ? tick_sched_handle.isra.18+0x90/0x90 [ 7054.427607] ? hrtimer_interrupt+0x128/0x2c0 [ 7054.429069] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7054.430919] ? apic_timer_interrupt+0xf/0x20 [ 7054.433166] [ 7054.434489] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7054.437286] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7054.439559] ? do_raw_spin_lock+0x177/0x190 [ 7054.441399] ? _raw_spin_lock+0x21/0x30 [ 7054.442922] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7054.445219] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 7054.447246] ? mdc_commit_open+0x91/0xb0 [mdc] [ 7054.449236] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7054.451549] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7054.453890] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 7054.455798] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7054.459623] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7054.461625] ? do_wait_intr+0xf0/0xf0 [ 7054.463162] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7054.466257] ? kthread+0x1d1/0x200 [ 7054.468289] ? set_kthread_struct+0x70/0x70 [ 7054.471283] ? ret_from_fork+0x1f/0x30 [ 7056.060813] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 7056.063471] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7056.065881] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 7056.094248] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7056.094260] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7056.094262] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7056.094274] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7056.094277] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7056.094281] RAX: 000000000000301a RBX: ffff96848811cae0 RCX: 0000000000000001 [ 7056.094283] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 7056.094284] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7056.094286] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 7056.094288] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 7056.094290] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 7056.094292] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7056.094293] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 7056.094299] Call Trace: [ 7056.095202] [ 7056.099695] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7056.099772] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7056.099779] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7056.099782] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7056.099796] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7056.099800] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7056.099805] RAX: 0000000000006d89 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 7056.099807] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 7056.099809] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7056.099811] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 7056.099814] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 7056.099817] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 7056.099820] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7056.099822] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 7056.099830] Call Trace: [ 7056.099835] [ 7056.099840] ? show_regs.cold.9+0x22/0x2f [ 7056.099849] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7056.099855] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7056.099859] ? watchdog+0x40/0x40 [ 7056.099864] ? hrtimer_interrupt+0x128/0x2c0 [ 7056.099868] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7056.099876] ? apic_timer_interrupt+0xf/0x20 [ 7056.100017] [ 7056.100021] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7056.100030] do_raw_spin_lock+0x177/0x190 [ 7056.100036] _raw_spin_lock+0x21/0x30 [ 7056.100041] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7056.100335] ? do_raw_spin_unlock+0x75/0x190 [ 7056.100342] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7056.100561] ? do_wait_intr+0xf0/0xf0 [ 7056.100569] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7056.111583] ? show_regs.cold.9+0x22/0x2f [ 7056.111681] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7056.111687] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7056.111692] ? watchdog+0x40/0x40 [ 7056.111697] ? hrtimer_interrupt+0x128/0x2c0 [ 7056.111701] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7056.111875] ? apic_timer_interrupt+0xf/0x20 [ 7056.111882] [ 7056.111888] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7056.111898] do_raw_spin_lock+0x177/0x190 [ 7056.111905] _raw_spin_lock+0x21/0x30 [ 7056.111910] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7056.112209] ? do_raw_spin_unlock+0x75/0x190 [ 7056.112215] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7056.112412] ? do_wait_intr+0xf0/0xf0 [ 7056.112421] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7056.114171] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7056.114429] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7056.114655] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7056.114786] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7056.114840] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7056.114869] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7056.114894] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7056.114979] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7056.115199] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7056.115225] lmv_intent_open+0x338/0x1150 [lmv] [ 7056.115254] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7056.115337] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7056.115419] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7056.115442] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7056.115524] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7056.115602] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7056.115681] ? do_raw_spin_unlock+0x75/0x190 [ 7056.115687] ? _raw_spin_unlock+0x12/0x30 [ 7056.115694] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7056.115723] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7056.115744] ll_atomic_open+0x36a/0xd70 [lustre] [ 7056.115828] lookup_open+0x356/0xb40 [ 7056.115838] do_last+0x313/0xfc0 [ 7056.115843] ? nd_jump_root+0xe5/0x160 [ 7056.115847] ? path_init+0x437/0x520 [ 7056.115853] path_openat+0xf7/0x500 [ 7056.115858] ? do_raw_spin_unlock+0x75/0x190 [ 7056.115863] do_filp_open+0x99/0x140 [ 7056.115870] ? getname_flags+0x6e/0x330 [ 7056.115874] ? __check_object_size+0xff/0x256 [ 7056.115881] ? do_raw_spin_unlock+0x75/0x190 [ 7056.115886] ? _raw_spin_unlock+0x12/0x30 [ 7056.115890] do_sys_openat2+0x2b4/0x410 [ 7056.115896] do_sys_open+0x73/0xa0 [ 7056.115900] __x64_sys_openat+0x24/0x30 [ 7056.115903] do_syscall_64+0xc1/0x440 [ 7056.115909] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7056.115915] RIP: 0033:0x7ff54af45332 [ 7056.115923] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7056.115928] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7056.115934] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 7056.115936] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 7056.115938] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 7056.115941] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 7056.115943] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 7056.593319] mdc_close+0x53b/0xd70 [mdc] [ 7056.597860] lmv_close+0x18d/0x490 [lmv] [ 7056.599658] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 7056.602378] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7056.604633] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7056.607740] ll_md_close+0x274/0x800 [lustre] [ 7056.615583] ll_file_release+0x128/0xc50 [lustre] [ 7056.617326] __fput+0x10d/0x440 [ 7056.625484] ____fput+0x12/0x20 [ 7056.633121] task_work_run+0xc6/0x110 [ 7056.634244] exit_to_usermode_loop+0x1dd/0x1f0 [ 7056.635545] do_syscall_64+0x3d6/0x440 [ 7056.642780] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7056.655220] RIP: 0033:0x7ff54af45cd8 [ 7056.658346] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7056.672452] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7056.676652] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 7056.683190] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 7056.687257] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 7056.697031] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 7056.699223] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7063.007034] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5104s! [ 7063.010792] Showing busy workqueues and worker pools: [ 7063.013286] workqueue events: flags=0x0 [ 7063.014846] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7063.018092] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 7063.023824] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7063.027061] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 7063.032867] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7063.036529] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 7063.039962] workqueue events_long: flags=0x0 [ 7063.041805] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7063.044901] pending: lu_objects_flush [obdclass] [ 7063.047761] workqueue events_unbound: flags=0x2 [ 7063.049496] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7063.051932] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7063.057055] workqueue events_power_efficient: flags=0x80 [ 7063.059212] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7063.061976] pending: neigh_periodic_work [ 7063.064810] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7063.067533] pending: do_cache_clean, check_lifetime [ 7063.069120] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7063.071091] pending: neigh_periodic_work [ 7063.072397] workqueue writeback: flags=0x4a [ 7063.074263] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7063.076781] pending: wb_workfn, wb_workfn [ 7063.078440] workqueue xprtiod: flags=0xa [ 7063.079933] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7063.082542] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 7063.085669] workqueue ipv6_addrconf: flags=0x40008 [ 7063.087808] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7063.091013] pending: addrconf_verify_work [ 7063.092570] workqueue knbd0-recv: flags=0x1a [ 7063.094357] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7063.096933] in-flight: 657:recv_work [ 7063.098576] workqueue ptlrpc_pinger: flags=0x2 [ 7063.100336] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7063.102922] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 7063.105990] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5104s workers=4 idle: 20831 12642 28859 [ 7063.110571] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7053s workers=2 idle: 659 [ 7080.059046] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 7080.064063] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7080.085034] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7080.110172] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7080.117332] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7080.128359] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7080.131357] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7080.160120] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7080.171739] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 7080.179339] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 7080.182867] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 7080.184837] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 7080.190864] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 7080.204488] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 7080.212336] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7080.214820] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 7080.218533] Call Trace: [ 7080.219762] [ 7080.220570] ? show_regs.cold.9+0x22/0x2f [ 7080.225959] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7080.229869] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7080.236811] ? watchdog+0x40/0x40 [ 7080.239336] ? hrtimer_interrupt+0x128/0x2c0 [ 7080.240860] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7080.242266] ? apic_timer_interrupt+0xf/0x20 [ 7080.247244] [ 7080.248478] ? native_safe_halt+0x2e/0x50 [ 7080.249893] kvm_wait+0xcd/0xe0 [ 7080.251274] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7080.253877] do_raw_spin_lock+0x177/0x190 [ 7080.255144] _raw_spin_lock+0x21/0x30 [ 7080.256713] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7080.259004] process_one_work+0x2c8/0x700 [ 7080.260867] worker_thread+0x296/0x6e0 [ 7080.262520] ? rescuer_thread+0x570/0x570 [ 7080.266232] kthread+0x1d1/0x200 [ 7080.271289] ? set_kthread_struct+0x70/0x70 [ 7080.274895] ret_from_fork+0x1f/0x30 [ 7084.058266] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 7084.061011] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 7084.067040] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7084.067126] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7084.067134] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7084.067137] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7084.067151] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7084.067156] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7084.067161] RAX: 0000000000002230 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 7084.067163] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 7084.067165] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7084.067168] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 7084.067170] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 7084.067172] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 7084.067175] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7084.067177] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 7084.067185] Call Trace: [ 7084.067191] [ 7084.067196] ? show_regs.cold.9+0x22/0x2f [ 7084.067206] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7084.067214] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7084.067218] ? watchdog+0x40/0x40 [ 7084.067223] ? hrtimer_interrupt+0x128/0x2c0 [ 7084.067228] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7084.067235] ? apic_timer_interrupt+0xf/0x20 [ 7084.067240] [ 7084.067243] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7084.067250] do_raw_spin_lock+0x177/0x190 [ 7084.067255] _raw_spin_lock+0x21/0x30 [ 7084.067260] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7084.074466] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7084.075004] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7084.075013] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7084.075016] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7084.075032] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7084.075036] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7084.075041] RAX: 0000000000001006 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 7084.075044] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 7084.075046] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7084.075048] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 7084.075051] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 7084.075054] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 7084.075056] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7084.075059] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 7084.075067] Call Trace: [ 7084.075073] [ 7084.075079] ? show_regs.cold.9+0x22/0x2f [ 7084.075089] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7084.075096] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7084.075101] ? watchdog+0x40/0x40 [ 7084.075105] ? hrtimer_interrupt+0x128/0x2c0 [ 7084.075108] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7084.075116] ? apic_timer_interrupt+0xf/0x20 [ 7084.075121] [ 7084.075124] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7084.075258] do_raw_spin_lock+0x177/0x190 [ 7084.075266] _raw_spin_lock+0x21/0x30 [ 7084.075271] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7084.077904] ? do_raw_spin_unlock+0x75/0x190 [ 7084.077914] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7084.120692] ? do_raw_spin_unlock+0x75/0x190 [ 7084.120725] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7084.124437] ? do_wait_intr+0xf0/0xf0 [ 7084.124448] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7084.145871] ? do_wait_intr+0xf0/0xf0 [ 7084.145897] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7084.146200] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7084.146420] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7084.146653] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7084.146800] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7084.146843] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7084.147418] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7084.147449] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7084.147649] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7084.149010] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7084.158009] mdc_close+0x53b/0xd70 [mdc] [ 7084.158068] lmv_close+0x18d/0x490 [lmv] [ 7084.158099] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 7084.158810] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7084.159628] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7084.159730] ll_md_close+0x274/0x800 [lustre] [ 7084.160369] ll_file_release+0x128/0xc50 [lustre] [ 7084.160457] __fput+0x10d/0x440 [ 7084.160471] ____fput+0x12/0x20 [ 7084.160476] task_work_run+0xc6/0x110 [ 7084.160484] exit_to_usermode_loop+0x1dd/0x1f0 [ 7084.160491] do_syscall_64+0x3d6/0x440 [ 7084.160496] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7084.160514] RIP: 0033:0x7ff54af45cd8 [ 7084.160523] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7084.160528] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7084.160533] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 7084.160657] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 7084.160660] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 7084.160662] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 7084.160664] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7084.458845] lmv_intent_open+0x338/0x1150 [lmv] [ 7084.460891] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7084.463404] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7084.465310] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7084.467420] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7084.470239] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7084.473407] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7084.476060] ? do_raw_spin_unlock+0x75/0x190 [ 7084.477830] ? _raw_spin_unlock+0x12/0x30 [ 7084.479147] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7084.481422] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7084.483857] ll_atomic_open+0x36a/0xd70 [lustre] [ 7084.486022] lookup_open+0x356/0xb40 [ 7084.487779] do_last+0x313/0xfc0 [ 7084.489314] ? nd_jump_root+0xe5/0x160 [ 7084.490801] ? path_init+0x437/0x520 [ 7084.491783] path_openat+0xf7/0x500 [ 7084.492802] ? do_raw_spin_unlock+0x75/0x190 [ 7084.494249] do_filp_open+0x99/0x140 [ 7084.495180] ? getname_flags+0x6e/0x330 [ 7084.496460] ? __check_object_size+0xff/0x256 [ 7084.498399] ? do_raw_spin_unlock+0x75/0x190 [ 7084.499751] ? _raw_spin_unlock+0x12/0x30 [ 7084.501042] do_sys_openat2+0x2b4/0x410 [ 7084.502427] do_sys_open+0x73/0xa0 [ 7084.504111] __x64_sys_openat+0x24/0x30 [ 7084.505860] do_syscall_64+0xc1/0x440 [ 7084.507622] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7084.509676] RIP: 0033:0x7ff54af45332 [ 7084.511069] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7084.517486] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7084.520357] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 7084.523314] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 7084.526400] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 7084.529357] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 7084.532334] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 7088.055052] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [ 7088.059009] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7088.073199] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7088.076440] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7088.079307] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7088.080889] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7088.086328] RSP: 0000:ffffaf9fc1b97bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7088.088771] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 7088.090640] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 7088.092464] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 7088.094464] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 7088.096443] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7088.099309] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 7088.102928] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7088.104902] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 7088.108393] Call Trace: [ 7088.109476] [ 7088.110211] ? show_regs.cold.9+0x22/0x2f [ 7088.111997] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7088.114135] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7088.115499] ? watchdog+0x40/0x40 [ 7088.116891] ? hrtimer_interrupt+0x128/0x2c0 [ 7088.118123] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7088.119774] ? apic_timer_interrupt+0xf/0x20 [ 7088.121247] [ 7088.121924] ? native_safe_halt+0x2e/0x50 [ 7088.123261] kvm_wait+0xcd/0xe0 [ 7088.124329] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7088.126259] do_raw_spin_lock+0x177/0x190 [ 7088.127472] _raw_spin_lock+0x21/0x30 [ 7088.128873] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7088.131049] obd_mod_free+0x46/0x140 [ptlrpc] [ 7088.132611] mdc_commit_open+0x91/0xb0 [mdc] [ 7088.133953] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7088.135564] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7088.138100] after_reply+0xe2c/0x1780 [ptlrpc] [ 7088.140254] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7088.142732] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7088.144552] ? do_wait_intr+0xf0/0xf0 [ 7088.146229] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7088.148325] kthread+0x1d1/0x200 [ 7088.149548] ? set_kthread_struct+0x70/0x70 [ 7088.150901] ret_from_fork+0x1f/0x30 [ 7093.216571] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5134s! [ 7093.221579] Showing busy workqueues and worker pools: [ 7093.224462] workqueue events: flags=0x0 [ 7093.226705] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7093.229965] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 7093.240551] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7093.243993] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 7093.249834] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7093.252907] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 7093.258630] workqueue events_long: flags=0x0 [ 7093.260562] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7093.263052] pending: lu_objects_flush [obdclass] [ 7093.264684] workqueue events_unbound: flags=0x2 [ 7093.266305] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7093.268059] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7093.271257] workqueue events_power_efficient: flags=0x80 [ 7093.273015] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7093.275984] pending: neigh_periodic_work [ 7093.277368] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7093.279589] pending: do_cache_clean, check_lifetime [ 7093.281217] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7093.283496] pending: neigh_periodic_work [ 7093.284955] workqueue writeback: flags=0x4a [ 7093.286266] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7093.288148] pending: wb_workfn, wb_workfn [ 7093.289504] workqueue xprtiod: flags=0xa [ 7093.290790] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7093.292934] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 7093.295668] workqueue ipv6_addrconf: flags=0x40008 [ 7093.297444] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7093.300295] pending: addrconf_verify_work [ 7093.301639] workqueue knbd0-recv: flags=0x1a [ 7093.303048] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7093.305409] in-flight: 657:recv_work [ 7093.307703] workqueue ptlrpc_pinger: flags=0x2 [ 7093.309162] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7093.311053] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 7093.313044] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5134s workers=4 idle: 20831 12642 28859 [ 7093.317583] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7083s workers=2 idle: 659 [ 7108.059348] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 7108.070277] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7108.133829] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7108.144239] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7108.155272] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7108.163281] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7108.171230] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7108.195156] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7108.204219] RAX: 0000000000007f86 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 7108.217615] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 7108.224144] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 7108.231614] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 7108.235151] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 7108.243742] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 7108.248208] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7108.251362] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 7108.256792] Call Trace: [ 7108.257807] [ 7108.258798] ? show_regs.cold.9+0x22/0x2f [ 7108.260957] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7108.263127] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7108.265225] ? watchdog+0x40/0x40 [ 7108.266955] ? hrtimer_interrupt+0x128/0x2c0 [ 7108.269230] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7108.278034] ? apic_timer_interrupt+0xf/0x20 [ 7108.284297] [ 7108.286143] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7108.291387] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7108.295417] do_raw_spin_lock+0x177/0x190 [ 7108.298187] _raw_spin_lock+0x21/0x30 [ 7108.300289] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7108.305853] process_one_work+0x2c8/0x700 [ 7108.308162] worker_thread+0x296/0x6e0 [ 7108.310404] ? rescuer_thread+0x570/0x570 [ 7108.312573] kthread+0x1d1/0x200 [ 7108.314200] ? set_kthread_struct+0x70/0x70 [ 7108.318004] ret_from_fork+0x1f/0x30 [ 7112.063119] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 7112.068606] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7112.069047] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 7112.126711] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7112.126726] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7112.126730] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7112.126743] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7112.126747] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7112.126753] RAX: 0000000000006e20 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 7112.126756] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 7112.126758] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7112.126760] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 7112.126763] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 7112.126766] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 7112.126769] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7112.126772] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 7112.126779] Call Trace: [ 7112.126784] [ 7112.126789] ? show_regs.cold.9+0x22/0x2f [ 7112.126797] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7112.126803] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7112.126807] ? watchdog+0x40/0x40 [ 7112.126811] ? hrtimer_interrupt+0x128/0x2c0 [ 7112.126815] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7112.126824] ? apic_timer_interrupt+0xf/0x20 [ 7112.126828] [ 7112.126831] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7112.126945] do_raw_spin_lock+0x177/0x190 [ 7112.126954] _raw_spin_lock+0x21/0x30 [ 7112.126959] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7112.127225] ? do_raw_spin_unlock+0x75/0x190 [ 7112.127231] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7112.132737] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7112.133036] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7112.133044] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7112.133048] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7112.133062] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7112.133067] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7112.133072] RAX: 0000000000002009 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 7112.133075] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 7112.133077] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7112.133079] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 7112.133082] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 7112.133085] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 7112.133088] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7112.133091] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 7112.133099] Call Trace: [ 7112.133104] [ 7112.133109] ? show_regs.cold.9+0x22/0x2f [ 7112.133118] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7112.133125] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7112.133129] ? watchdog+0x40/0x40 [ 7112.133133] ? hrtimer_interrupt+0x128/0x2c0 [ 7112.133138] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7112.133147] ? apic_timer_interrupt+0xf/0x20 [ 7112.133151] [ 7112.133154] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7112.133161] do_raw_spin_lock+0x177/0x190 [ 7112.133166] _raw_spin_lock+0x21/0x30 [ 7112.133170] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7112.133636] ? do_raw_spin_unlock+0x75/0x190 [ 7112.133641] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7112.134698] ? do_wait_intr+0xf0/0xf0 [ 7112.134706] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7112.136680] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7112.137451] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7112.137681] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7112.150912] ? do_wait_intr+0xf0/0xf0 [ 7112.150934] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7112.151218] mdc_close+0x53b/0xd70 [mdc] [ 7112.151255] lmv_close+0x18d/0x490 [lmv] [ 7112.152008] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 7112.152169] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7112.152828] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7112.152925] ll_md_close+0x274/0x800 [lustre] [ 7112.153007] ll_file_release+0x128/0xc50 [lustre] [ 7112.153088] __fput+0x10d/0x440 [ 7112.153098] ____fput+0x12/0x20 [ 7112.153102] task_work_run+0xc6/0x110 [ 7112.153110] exit_to_usermode_loop+0x1dd/0x1f0 [ 7112.153118] do_syscall_64+0x3d6/0x440 [ 7112.153122] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7112.153131] RIP: 0033:0x7ff54af45cd8 [ 7112.153157] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7112.153162] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7112.153167] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 7112.153170] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 7112.153172] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 7112.153174] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 7112.153176] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7112.683774] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7112.687057] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7112.692589] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7112.694468] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7112.701706] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7112.705492] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7112.712771] lmv_intent_open+0x338/0x1150 [lmv] [ 7112.717795] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7112.720687] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7112.728089] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7112.731632] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7112.739000] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7112.745591] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7112.748665] ? do_raw_spin_unlock+0x75/0x190 [ 7112.752350] ? _raw_spin_unlock+0x12/0x30 [ 7112.755268] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7112.764480] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7112.771251] ll_atomic_open+0x36a/0xd70 [lustre] [ 7112.778354] lookup_open+0x356/0xb40 [ 7112.781546] do_last+0x313/0xfc0 [ 7112.785983] ? nd_jump_root+0xe5/0x160 [ 7112.789796] ? path_init+0x437/0x520 [ 7112.791729] path_openat+0xf7/0x500 [ 7112.794502] ? do_raw_spin_unlock+0x75/0x190 [ 7112.796959] do_filp_open+0x99/0x140 [ 7112.803283] ? getname_flags+0x6e/0x330 [ 7112.808293] ? __check_object_size+0xff/0x256 [ 7112.813505] ? do_raw_spin_unlock+0x75/0x190 [ 7112.818532] ? _raw_spin_unlock+0x12/0x30 [ 7112.833502] do_sys_openat2+0x2b4/0x410 [ 7112.834896] do_sys_open+0x73/0xa0 [ 7112.837946] __x64_sys_openat+0x24/0x30 [ 7112.841208] do_syscall_64+0xc1/0x440 [ 7112.848063] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7112.853221] RIP: 0033:0x7ff54af45332 [ 7112.857162] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7112.873833] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7112.879823] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 7112.886180] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 7112.891135] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 7112.894334] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 7112.898966] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 7120.055433] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [ 7120.059178] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7120.098994] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7120.117402] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7120.122437] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7120.127760] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7120.141572] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7120.145794] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 7120.148332] RDX: 0000000000007033 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 7120.158376] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 7120.165525] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 7120.169068] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7120.173319] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 7120.176569] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7120.178927] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 7120.187004] Call Trace: [ 7120.188535] [ 7120.189293] ? show_regs.cold.9+0x22/0x2f [ 7120.190611] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7120.193371] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7120.195853] ? watchdog+0x40/0x40 [ 7120.197303] ? hrtimer_interrupt+0x128/0x2c0 [ 7120.198755] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7120.207463] ? apic_timer_interrupt+0xf/0x20 [ 7120.209735] [ 7120.210836] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7120.212559] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7120.217189] do_raw_spin_lock+0x177/0x190 [ 7120.220265] _raw_spin_lock+0x21/0x30 [ 7120.221672] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7120.224454] obd_mod_free+0x46/0x140 [ptlrpc] [ 7120.226917] mdc_commit_open+0x91/0xb0 [mdc] [ 7120.228973] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7120.231895] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7120.238705] after_reply+0xe2c/0x1780 [ptlrpc] [ 7120.244280] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7120.249741] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7120.252124] ? do_wait_intr+0xf0/0xf0 [ 7120.254093] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7120.257136] kthread+0x1d1/0x200 [ 7120.258575] ? set_kthread_struct+0x70/0x70 [ 7120.260617] ret_from_fork+0x1f/0x30 [ 7123.423120] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5164s! [ 7123.448341] Showing busy workqueues and worker pools: [ 7123.455438] workqueue events: flags=0x0 [ 7123.460896] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7123.466685] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 7123.474103] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7123.481008] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 7123.511313] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7123.517790] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 7123.523051] workqueue events_long: flags=0x0 [ 7123.527715] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7123.536225] pending: lu_objects_flush [obdclass] [ 7123.537933] workqueue events_unbound: flags=0x2 [ 7123.545022] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7123.553323] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7123.564079] workqueue events_power_efficient: flags=0x80 [ 7123.567696] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7123.580579] pending: neigh_periodic_work [ 7123.586535] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7123.588309] pending: do_cache_clean, check_lifetime [ 7123.591394] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7123.604344] pending: neigh_periodic_work [ 7123.610671] workqueue writeback: flags=0x4a [ 7123.614400] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7123.623901] pending: wb_workfn, wb_workfn [ 7123.627974] workqueue xprtiod: flags=0xa [ 7123.630933] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7123.641978] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 7123.649258] workqueue ipv6_addrconf: flags=0x40008 [ 7123.657424] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7123.660918] pending: addrconf_verify_work [ 7123.666829] workqueue knbd0-recv: flags=0x1a [ 7123.670825] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7123.674569] in-flight: 657:recv_work [ 7123.677864] workqueue ptlrpc_pinger: flags=0x2 [ 7123.681813] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7123.687933] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 7123.691834] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5164s workers=4 idle: 20831 12642 28859 [ 7123.695273] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7113s workers=2 idle: 659 [ 7136.059010] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 7136.064660] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7136.093329] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7136.102371] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7136.109723] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7136.113975] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7136.117396] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7136.129504] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7136.133627] RAX: 0000000000002198 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 7136.137761] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 7136.141591] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 7136.145937] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 7136.150518] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 7136.155765] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 7136.163333] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7136.170329] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 7136.177711] Call Trace: [ 7136.179705] [ 7136.180987] ? show_regs.cold.9+0x22/0x2f [ 7136.184395] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7136.191754] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7136.194707] ? watchdog+0x40/0x40 [ 7136.197363] ? hrtimer_interrupt+0x128/0x2c0 [ 7136.200335] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7136.203010] ? apic_timer_interrupt+0xf/0x20 [ 7136.208977] [ 7136.209790] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7136.213479] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7136.216727] do_raw_spin_lock+0x177/0x190 [ 7136.219026] _raw_spin_lock+0x21/0x30 [ 7136.221660] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7136.225617] process_one_work+0x2c8/0x700 [ 7136.229293] worker_thread+0x296/0x6e0 [ 7136.234545] ? rescuer_thread+0x570/0x570 [ 7136.236562] kthread+0x1d1/0x200 [ 7136.238679] ? set_kthread_struct+0x70/0x70 [ 7136.241932] ret_from_fork+0x1f/0x30 [ 7140.058189] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 7140.061095] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 7140.064745] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7140.064830] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7140.064838] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7140.064840] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7140.064854] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7140.064858] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7140.064863] RAX: 0000000000007283 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 7140.064866] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 7140.064868] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7140.064871] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 7140.064873] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 7140.064876] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 7140.064879] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7140.064881] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 7140.064946] Call Trace: [ 7140.064952] [ 7140.064958] ? show_regs.cold.9+0x22/0x2f [ 7140.065228] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7140.065236] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7140.065240] ? watchdog+0x40/0x40 [ 7140.065245] ? hrtimer_interrupt+0x128/0x2c0 [ 7140.065250] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7140.065258] ? apic_timer_interrupt+0xf/0x20 [ 7140.065263] [ 7140.065266] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7140.065274] do_raw_spin_lock+0x177/0x190 [ 7140.065279] _raw_spin_lock+0x21/0x30 [ 7140.065283] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7140.065558] ? do_raw_spin_unlock+0x75/0x190 [ 7140.065565] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7140.073517] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7140.074437] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7140.074445] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7140.074448] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7140.074461] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7140.074466] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7140.074471] RAX: 000000000000760f RBX: ffff96848811cae0 RCX: 0000000000000001 [ 7140.074473] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 7140.074475] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7140.074477] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 7140.074479] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 7140.074483] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 7140.074486] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7140.074488] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 7140.074496] Call Trace: [ 7140.074501] [ 7140.074506] ? show_regs.cold.9+0x22/0x2f [ 7140.074516] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7140.074523] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7140.074527] ? watchdog+0x40/0x40 [ 7140.074532] ? hrtimer_interrupt+0x128/0x2c0 [ 7140.074536] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7140.074544] ? apic_timer_interrupt+0xf/0x20 [ 7140.074549] [ 7140.074553] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7140.074560] do_raw_spin_lock+0x177/0x190 [ 7140.074565] _raw_spin_lock+0x21/0x30 [ 7140.074570] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7140.074842] ? do_raw_spin_unlock+0x75/0x190 [ 7140.074849] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7140.126879] ? do_wait_intr+0xf0/0xf0 [ 7140.126913] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7140.127499] mdc_close+0x53b/0xd70 [mdc] [ 7140.127541] lmv_close+0x18d/0x490 [lmv] [ 7140.127841] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 7140.128026] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7140.128981] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7140.129333] ll_md_close+0x274/0x800 [lustre] [ 7140.129415] ll_file_release+0x128/0xc50 [lustre] [ 7140.129498] __fput+0x10d/0x440 [ 7140.129508] ____fput+0x12/0x20 [ 7140.147561] ? do_wait_intr+0xf0/0xf0 [ 7140.147585] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7140.147862] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7140.149250] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7140.149503] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7140.150258] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7140.150297] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7140.150327] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7140.150354] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7140.153437] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7140.153733] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7140.153776] lmv_intent_open+0x338/0x1150 [lmv] [ 7140.153820] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7140.153931] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7140.154015] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7140.154039] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7140.155447] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7140.155589] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7140.155667] ? do_raw_spin_unlock+0x75/0x190 [ 7140.155675] ? _raw_spin_unlock+0x12/0x30 [ 7140.155682] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7140.155714] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7140.155732] ll_atomic_open+0x36a/0xd70 [lustre] [ 7140.155819] lookup_open+0x356/0xb40 [ 7140.155828] do_last+0x313/0xfc0 [ 7140.155833] ? nd_jump_root+0xe5/0x160 [ 7140.155838] ? path_init+0x437/0x520 [ 7140.155842] path_openat+0xf7/0x500 [ 7140.155847] ? do_raw_spin_unlock+0x75/0x190 [ 7140.155852] do_filp_open+0x99/0x140 [ 7140.155858] ? getname_flags+0x6e/0x330 [ 7140.155862] ? __check_object_size+0xff/0x256 [ 7140.155869] ? do_raw_spin_unlock+0x75/0x190 [ 7140.155874] ? _raw_spin_unlock+0x12/0x30 [ 7140.155878] do_sys_openat2+0x2b4/0x410 [ 7140.156053] do_sys_open+0x73/0xa0 [ 7140.166820] task_work_run+0xc6/0x110 [ 7140.166845] exit_to_usermode_loop+0x1dd/0x1f0 [ 7140.166853] do_syscall_64+0x3d6/0x440 [ 7140.166857] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7140.166866] RIP: 0033:0x7ff54af45cd8 [ 7140.166875] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7140.166879] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7140.166885] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 7140.166887] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 7140.166890] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 7140.166892] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 7140.166895] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7141.164840] __x64_sys_openat+0x24/0x30 [ 7141.172379] do_syscall_64+0xc1/0x440 [ 7141.182314] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7141.183888] RIP: 0033:0x7ff54af45332 [ 7141.189928] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7141.214946] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7141.222700] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 7141.231499] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 7141.243034] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 7141.254277] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 7141.255845] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 7148.055112] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [ 7148.059239] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7148.100996] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7148.112473] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7148.126533] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7148.129547] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7148.138907] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7148.144671] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 7148.150992] RDX: 00000000000009e4 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 7148.155461] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 7148.168930] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 7148.178384] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7148.185551] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 7148.200175] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7148.206293] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 7148.219937] Call Trace: [ 7148.222684] [ 7148.230759] ? show_regs.cold.9+0x22/0x2f [ 7148.235812] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7148.249469] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7148.260571] ? watchdog+0x40/0x40 [ 7148.263452] ? hrtimer_interrupt+0x128/0x2c0 [ 7148.266133] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7148.269802] ? apic_timer_interrupt+0xf/0x20 [ 7148.273873] [ 7148.275401] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7148.287845] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7148.290504] do_raw_spin_lock+0x177/0x190 [ 7148.302622] _raw_spin_lock+0x21/0x30 [ 7148.309697] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7148.322499] obd_mod_free+0x46/0x140 [ptlrpc] [ 7148.331554] mdc_commit_open+0x91/0xb0 [mdc] [ 7148.340549] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7148.347789] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7148.355479] after_reply+0xe2c/0x1780 [ptlrpc] [ 7148.360661] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7148.373883] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7148.377778] ? do_wait_intr+0xf0/0xf0 [ 7148.384465] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7148.388288] kthread+0x1d1/0x200 [ 7148.391067] ? set_kthread_struct+0x70/0x70 [ 7148.392431] ret_from_fork+0x1f/0x30 [ 7154.143063] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5195s! [ 7154.146286] Showing busy workqueues and worker pools: [ 7154.148497] workqueue events: flags=0x0 [ 7154.150156] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7154.156700] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 7154.161937] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7154.167202] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 7154.179623] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7154.185453] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 7154.187798] workqueue events_long: flags=0x0 [ 7154.189008] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7154.191067] pending: lu_objects_flush [obdclass] [ 7154.194204] workqueue events_unbound: flags=0x2 [ 7154.196211] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7154.199234] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7154.203940] workqueue events_power_efficient: flags=0x80 [ 7154.210416] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7154.213143] pending: neigh_periodic_work [ 7154.216738] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7154.218959] pending: do_cache_clean, check_lifetime [ 7154.222933] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7154.227543] pending: neigh_periodic_work [ 7154.230847] workqueue writeback: flags=0x4a [ 7154.233019] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7154.242419] pending: wb_workfn, wb_workfn [ 7154.244791] workqueue xprtiod: flags=0xa [ 7154.248219] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7154.254092] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 7154.257413] workqueue ipv6_addrconf: flags=0x40008 [ 7154.268175] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7154.274353] pending: addrconf_verify_work [ 7154.277710] workqueue knbd0-recv: flags=0x1a [ 7154.281248] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7154.285799] in-flight: 657:recv_work [ 7154.290489] workqueue ptlrpc_pinger: flags=0x2 [ 7154.291889] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7154.295574] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 7154.301485] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5195s workers=4 idle: 20831 12642 28859 [ 7154.306317] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7144s workers=2 idle: 659 [ 7164.059226] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 7164.065618] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7164.087915] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7164.092021] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7164.095717] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7164.099711] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7164.103428] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7164.120504] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7164.130897] RAX: 0000000000007b85 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 7164.144660] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 7164.153259] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 7164.155104] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 7164.160302] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 7164.167078] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 7164.174908] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7164.179466] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 7164.188708] Call Trace: [ 7164.190163] [ 7164.191122] ? show_regs.cold.9+0x22/0x2f [ 7164.192528] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7164.195499] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7164.198321] ? watchdog+0x40/0x40 [ 7164.201170] ? hrtimer_interrupt+0x128/0x2c0 [ 7164.203568] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7164.207514] ? apic_timer_interrupt+0xf/0x20 [ 7164.209791] [ 7164.210639] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7164.212737] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7164.214525] do_raw_spin_lock+0x177/0x190 [ 7164.220072] _raw_spin_lock+0x21/0x30 [ 7164.221357] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7164.226940] process_one_work+0x2c8/0x700 [ 7164.229412] worker_thread+0x296/0x6e0 [ 7164.231157] ? rescuer_thread+0x570/0x570 [ 7164.238567] kthread+0x1d1/0x200 [ 7164.244204] ? set_kthread_struct+0x70/0x70 [ 7164.246789] ret_from_fork+0x1f/0x30 [ 7168.060433] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 7168.063611] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 7168.067958] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7168.068391] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7168.068399] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7168.068402] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7168.068415] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7168.068419] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7168.068423] RAX: 000000000000140c RBX: ffff96848811cae0 RCX: 0000000000000001 [ 7168.068426] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 7168.068428] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7168.068430] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 7168.068433] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 7168.068436] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 7168.068439] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7168.068441] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 7168.068448] Call Trace: [ 7168.068453] [ 7168.068459] ? show_regs.cold.9+0x22/0x2f [ 7168.068467] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7168.068474] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7168.068479] ? watchdog+0x40/0x40 [ 7168.068483] ? hrtimer_interrupt+0x128/0x2c0 [ 7168.068487] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7168.068495] ? apic_timer_interrupt+0xf/0x20 [ 7168.068499] [ 7168.068502] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7168.068508] do_raw_spin_lock+0x177/0x190 [ 7168.068513] _raw_spin_lock+0x21/0x30 [ 7168.068518] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7168.083106] Modules linked in: [ 7168.105593] ? do_raw_spin_unlock+0x75/0x190 [ 7168.109754] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7168.109834] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7168.109841] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7168.109844] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7168.109860] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7168.109864] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7168.109869] RAX: 000000000000720d RBX: ffff96848811cae0 RCX: 0000000000000001 [ 7168.109871] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 7168.109874] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7168.109876] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 7168.109878] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 7168.109881] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 7168.109885] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7168.109888] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 7168.109908] Call Trace: [ 7168.109914] [ 7168.109920] ? show_regs.cold.9+0x22/0x2f [ 7168.109930] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7168.109937] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7168.109942] ? watchdog+0x40/0x40 [ 7168.109946] ? hrtimer_interrupt+0x128/0x2c0 [ 7168.109951] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7168.109960] ? apic_timer_interrupt+0xf/0x20 [ 7168.109965] [ 7168.109968] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7168.109974] do_raw_spin_lock+0x177/0x190 [ 7168.109979] _raw_spin_lock+0x21/0x30 [ 7168.109984] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7168.121640] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7168.132588] ? do_raw_spin_unlock+0x75/0x190 [ 7168.155544] ? do_wait_intr+0xf0/0xf0 [ 7168.163872] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7168.164184] ? do_wait_intr+0xf0/0xf0 [ 7168.164195] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7168.167960] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7168.168583] mdc_close+0x53b/0xd70 [mdc] [ 7168.168623] lmv_close+0x18d/0x490 [lmv] [ 7168.168654] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 7168.168961] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7168.169110] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7168.169262] ll_md_close+0x274/0x800 [lustre] [ 7168.169343] ll_file_release+0x128/0xc50 [lustre] [ 7168.169441] __fput+0x10d/0x440 [ 7168.169450] ____fput+0x12/0x20 [ 7168.169454] task_work_run+0xc6/0x110 [ 7168.169462] exit_to_usermode_loop+0x1dd/0x1f0 [ 7168.169469] do_syscall_64+0x3d6/0x440 [ 7168.169474] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7168.169483] RIP: 0033:0x7ff54af45cd8 [ 7168.169491] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7168.169495] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7168.169500] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 7168.169503] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 7168.169506] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 7168.169508] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 7168.169509] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7168.567305] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7168.571581] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7168.578315] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7168.582002] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7168.587475] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7168.589284] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7168.596607] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7168.601381] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7168.608545] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7168.613747] lmv_intent_open+0x338/0x1150 [lmv] [ 7168.619924] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7168.624506] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7168.628018] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7168.629863] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7168.632701] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7168.635635] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7168.638203] ? do_raw_spin_unlock+0x75/0x190 [ 7168.640389] ? _raw_spin_unlock+0x12/0x30 [ 7168.644552] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7168.649800] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7168.653479] ll_atomic_open+0x36a/0xd70 [lustre] [ 7168.658296] lookup_open+0x356/0xb40 [ 7168.664270] do_last+0x313/0xfc0 [ 7168.666490] ? nd_jump_root+0xe5/0x160 [ 7168.669161] ? path_init+0x437/0x520 [ 7168.672472] path_openat+0xf7/0x500 [ 7168.677322] ? do_raw_spin_unlock+0x75/0x190 [ 7168.682474] do_filp_open+0x99/0x140 [ 7168.688241] ? getname_flags+0x6e/0x330 [ 7168.691377] ? __check_object_size+0xff/0x256 [ 7168.694502] ? do_raw_spin_unlock+0x75/0x190 [ 7168.697513] ? _raw_spin_unlock+0x12/0x30 [ 7168.699689] do_sys_openat2+0x2b4/0x410 [ 7168.701916] do_sys_open+0x73/0xa0 [ 7168.703518] __x64_sys_openat+0x24/0x30 [ 7168.705547] do_syscall_64+0xc1/0x440 [ 7168.707713] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7168.710598] RIP: 0033:0x7ff54af45332 [ 7168.712514] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7168.721257] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7168.726502] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 7168.735426] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 7168.742471] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 7168.754370] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 7168.760181] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 7180.059871] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 7180.066168] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7180.095493] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7180.100787] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7180.105684] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7180.111520] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7180.121719] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7180.125084] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 7180.127891] RDX: 00000000000006d8 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 7180.131527] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 7180.135697] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 7180.144388] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7180.150412] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 7180.154664] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7180.158303] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 7180.162505] Call Trace: [ 7180.164424] [ 7180.165474] ? show_regs.cold.9+0x22/0x2f [ 7180.166769] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7180.171612] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7180.174618] ? watchdog+0x40/0x40 [ 7180.177560] ? hrtimer_interrupt+0x128/0x2c0 [ 7180.180795] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7180.184097] ? apic_timer_interrupt+0xf/0x20 [ 7180.187714] [ 7180.189406] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7180.192207] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7180.195416] do_raw_spin_lock+0x177/0x190 [ 7180.197560] _raw_spin_lock+0x21/0x30 [ 7180.199352] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7180.201830] obd_mod_free+0x46/0x140 [ptlrpc] [ 7180.204286] mdc_commit_open+0x91/0xb0 [mdc] [ 7180.206742] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7180.209402] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7180.212110] after_reply+0xe2c/0x1780 [ptlrpc] [ 7180.217453] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7180.221696] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7180.227622] ? do_wait_intr+0xf0/0xf0 [ 7180.229319] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7180.232437] kthread+0x1d1/0x200 [ 7180.234020] ? set_kthread_struct+0x70/0x70 [ 7180.235425] ret_from_fork+0x1f/0x30 [ 7184.356574] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5225s! [ 7184.372390] Showing busy workqueues and worker pools: [ 7184.375867] workqueue events: flags=0x0 [ 7184.381439] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7184.388440] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 7184.398437] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7184.410810] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 7184.432152] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7184.448409] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 7184.455258] workqueue events_long: flags=0x0 [ 7184.459970] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7184.465610] pending: lu_objects_flush [obdclass] [ 7184.473874] workqueue events_unbound: flags=0x2 [ 7184.477196] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7184.481380] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7184.491215] workqueue events_power_efficient: flags=0x80 [ 7184.495504] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7184.511072] pending: neigh_periodic_work [ 7184.512664] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7184.517827] pending: do_cache_clean, check_lifetime [ 7184.527135] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7184.529534] pending: neigh_periodic_work [ 7184.537466] workqueue writeback: flags=0x4a [ 7184.539554] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7184.544213] pending: wb_workfn, wb_workfn [ 7184.545898] workqueue xprtiod: flags=0xa [ 7184.553131] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7184.558520] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 7184.568840] workqueue ipv6_addrconf: flags=0x40008 [ 7184.571818] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7184.582133] pending: addrconf_verify_work [ 7184.584670] workqueue knbd0-recv: flags=0x1a [ 7184.586977] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7184.591550] in-flight: 657:recv_work [ 7184.594226] workqueue ptlrpc_pinger: flags=0x2 [ 7184.596012] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7184.599417] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 7184.602144] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5225s workers=4 idle: 20831 12642 28859 [ 7184.605377] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7174s workers=2 idle: 659 [ 7192.059015] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 7192.065390] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7192.101840] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7192.115443] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7192.121916] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7192.128493] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7192.132138] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7192.155164] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7192.162259] RAX: 000000000000564c RBX: ffff96848811cae0 RCX: 0000000000000001 [ 7192.167477] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 7192.172258] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 7192.177256] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 7192.183763] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 7192.193431] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 7192.201816] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7192.208225] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 7192.214947] Call Trace: [ 7192.217247] [ 7192.221271] ? show_regs.cold.9+0x22/0x2f [ 7192.227073] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7192.232977] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7192.240514] ? watchdog+0x40/0x40 [ 7192.242191] ? hrtimer_interrupt+0x128/0x2c0 [ 7192.245032] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7192.248793] ? apic_timer_interrupt+0xf/0x20 [ 7192.252744] [ 7192.254379] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7192.258926] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7192.263643] do_raw_spin_lock+0x177/0x190 [ 7192.265975] _raw_spin_lock+0x21/0x30 [ 7192.268119] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7192.272413] process_one_work+0x2c8/0x700 [ 7192.273612] worker_thread+0x296/0x6e0 [ 7192.277977] ? rescuer_thread+0x570/0x570 [ 7192.279972] kthread+0x1d1/0x200 [ 7192.284567] ? set_kthread_struct+0x70/0x70 [ 7192.296860] ret_from_fork+0x1f/0x30 [ 7196.061712] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 7196.065594] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 7196.068524] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7196.068608] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7196.068615] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7196.068618] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7196.068643] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7196.068647] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7196.068651] RAX: 0000000000004d1c RBX: ffff96848811cae0 RCX: 0000000000000001 [ 7196.068654] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 7196.068656] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7196.068658] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 7196.068660] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 7196.068664] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 7196.068666] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7196.068669] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 7196.068676] Call Trace: [ 7196.068682] [ 7196.068688] ? show_regs.cold.9+0x22/0x2f [ 7196.068697] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7196.068704] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7196.068707] ? watchdog+0x40/0x40 [ 7196.068711] ? hrtimer_interrupt+0x128/0x2c0 [ 7196.068716] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7196.068724] ? apic_timer_interrupt+0xf/0x20 [ 7196.068729] [ 7196.068732] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7196.068739] do_raw_spin_lock+0x177/0x190 [ 7196.068744] _raw_spin_lock+0x21/0x30 [ 7196.068748] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7196.069150] ? do_raw_spin_unlock+0x75/0x190 [ 7196.069156] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7196.069437] ? do_wait_intr+0xf0/0xf0 [ 7196.069446] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7196.069656] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7196.069869] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7196.076191] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7196.076276] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7196.076285] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7196.076288] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7196.076306] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7196.076311] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7196.076315] RAX: 000000000000598c RBX: ffff96848811cae0 RCX: 0000000000000001 [ 7196.076317] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 7196.076319] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7196.076321] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 7196.076324] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 7196.076327] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 7196.076330] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7196.076333] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 7196.076341] Call Trace: [ 7196.076347] [ 7196.076354] ? show_regs.cold.9+0x22/0x2f [ 7196.076364] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7196.076371] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7196.076375] ? watchdog+0x40/0x40 [ 7196.076380] ? hrtimer_interrupt+0x128/0x2c0 [ 7196.076384] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7196.076393] ? apic_timer_interrupt+0xf/0x20 [ 7196.076397] [ 7196.076400] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7196.076406] do_raw_spin_lock+0x177/0x190 [ 7196.076411] _raw_spin_lock+0x21/0x30 [ 7196.076415] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7196.077376] ? do_raw_spin_unlock+0x75/0x190 [ 7196.077396] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7196.077732] ? do_wait_intr+0xf0/0xf0 [ 7196.077741] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7196.078211] mdc_close+0x53b/0xd70 [mdc] [ 7196.078247] lmv_close+0x18d/0x490 [lmv] [ 7196.078277] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 7196.078663] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7196.078786] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7196.078945] ll_md_close+0x274/0x800 [lustre] [ 7196.079544] ll_file_release+0x128/0xc50 [lustre] [ 7196.079637] __fput+0x10d/0x440 [ 7196.079649] ____fput+0x12/0x20 [ 7196.079654] task_work_run+0xc6/0x110 [ 7196.079665] exit_to_usermode_loop+0x1dd/0x1f0 [ 7196.079674] do_syscall_64+0x3d6/0x440 [ 7196.079678] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7196.079688] RIP: 0033:0x7ff54af45cd8 [ 7196.079696] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7196.079700] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7196.079706] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 7196.079708] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 7196.079710] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 7196.079713] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 7196.079715] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7196.659332] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7196.665002] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7196.671676] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7196.673411] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7196.678630] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7196.685749] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7196.693556] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7196.703069] lmv_intent_open+0x338/0x1150 [lmv] [ 7196.704807] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7196.712710] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7196.721169] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7196.725713] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7196.727444] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7196.747276] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7196.756112] ? do_raw_spin_unlock+0x75/0x190 [ 7196.765870] ? _raw_spin_unlock+0x12/0x30 [ 7196.768913] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7196.777364] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7196.785405] ll_atomic_open+0x36a/0xd70 [lustre] [ 7196.792031] lookup_open+0x356/0xb40 [ 7196.795123] do_last+0x313/0xfc0 [ 7196.798074] ? nd_jump_root+0xe5/0x160 [ 7196.799276] ? path_init+0x437/0x520 [ 7196.802746] path_openat+0xf7/0x500 [ 7196.805974] ? do_raw_spin_unlock+0x75/0x190 [ 7196.810312] do_filp_open+0x99/0x140 [ 7196.811643] ? getname_flags+0x6e/0x330 [ 7196.814500] ? __check_object_size+0xff/0x256 [ 7196.819165] ? do_raw_spin_unlock+0x75/0x190 [ 7196.820647] ? _raw_spin_unlock+0x12/0x30 [ 7196.821768] do_sys_openat2+0x2b4/0x410 [ 7196.824858] do_sys_open+0x73/0xa0 [ 7196.825826] __x64_sys_openat+0x24/0x30 [ 7196.828163] do_syscall_64+0xc1/0x440 [ 7196.831379] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7196.833326] RIP: 0033:0x7ff54af45332 [ 7196.835423] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7196.846107] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7196.849129] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 7196.853832] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 7196.859561] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 7196.864775] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 7196.868844] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 7212.055147] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 7212.068658] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7212.107691] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7212.121969] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7212.128747] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7212.132651] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7212.144862] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7212.149134] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 7212.154803] RDX: 0000000000003b46 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 7212.161626] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 7212.172899] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 7212.180456] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7212.185687] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 7212.191637] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7212.195439] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 7212.205754] Call Trace: [ 7212.208211] [ 7212.210792] ? show_regs.cold.9+0x22/0x2f [ 7212.212966] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7212.219680] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7212.221327] ? watchdog+0x40/0x40 [ 7212.222456] ? hrtimer_interrupt+0x128/0x2c0 [ 7212.233520] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7212.238118] ? apic_timer_interrupt+0xf/0x20 [ 7212.240861] [ 7212.243235] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7212.249829] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7212.254001] do_raw_spin_lock+0x177/0x190 [ 7212.256892] _raw_spin_lock+0x21/0x30 [ 7212.261278] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7212.266925] obd_mod_free+0x46/0x140 [ptlrpc] [ 7212.269161] mdc_commit_open+0x91/0xb0 [mdc] [ 7212.279577] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7212.282559] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7212.287514] after_reply+0xe2c/0x1780 [ptlrpc] [ 7212.294612] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7212.300231] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7212.301995] ? do_wait_intr+0xf0/0xf0 [ 7212.303251] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7212.310106] kthread+0x1d1/0x200 [ 7212.312037] ? set_kthread_struct+0x70/0x70 [ 7212.315040] ret_from_fork+0x1f/0x30 [ 7215.071738] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5256s! [ 7215.082320] Showing busy workqueues and worker pools: [ 7215.084436] workqueue events: flags=0x0 [ 7215.086014] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7215.091418] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 7215.095722] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7215.099251] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 7215.114077] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7215.119874] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 7215.126219] workqueue events_long: flags=0x0 [ 7215.129033] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7215.133855] pending: lu_objects_flush [obdclass] [ 7215.137707] workqueue events_unbound: flags=0x2 [ 7215.140819] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7215.149012] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7215.154784] workqueue events_power_efficient: flags=0x80 [ 7215.159794] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7215.166092] pending: neigh_periodic_work [ 7215.170580] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7215.182036] pending: do_cache_clean, check_lifetime [ 7215.185547] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7215.191873] pending: neigh_periodic_work [ 7215.194885] workqueue writeback: flags=0x4a [ 7215.199199] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7215.202793] pending: wb_workfn, wb_workfn [ 7215.205854] workqueue xprtiod: flags=0xa [ 7215.207530] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7215.212845] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 7215.219117] workqueue ipv6_addrconf: flags=0x40008 [ 7215.223415] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7215.227127] pending: addrconf_verify_work [ 7215.232575] workqueue knbd0-recv: flags=0x1a [ 7215.236388] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7215.247094] in-flight: 657:recv_work [ 7215.249032] workqueue ptlrpc_pinger: flags=0x2 [ 7215.251577] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7215.254769] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 7215.259115] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5256s workers=4 idle: 20831 12642 28859 [ 7215.265869] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7205s workers=2 idle: 659 [ 7220.059191] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 7220.063724] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7220.109093] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7220.114772] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7220.125621] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7220.134972] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7220.136730] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7220.158745] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7220.162918] RAX: 0000000000006415 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 7220.166283] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 7220.169109] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 7220.178486] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 7220.181183] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 7220.184854] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 7220.189326] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7220.196840] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 7220.202358] Call Trace: [ 7220.203573] [ 7220.206185] ? show_regs.cold.9+0x22/0x2f [ 7220.208000] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7220.211156] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7220.214769] ? watchdog+0x40/0x40 [ 7220.216310] ? hrtimer_interrupt+0x128/0x2c0 [ 7220.227598] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7220.235684] ? apic_timer_interrupt+0xf/0x20 [ 7220.237017] [ 7220.237571] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7220.239665] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7220.241885] do_raw_spin_lock+0x177/0x190 [ 7220.243817] _raw_spin_lock+0x21/0x30 [ 7220.245448] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7220.248156] process_one_work+0x2c8/0x700 [ 7220.250288] worker_thread+0x296/0x6e0 [ 7220.258758] ? rescuer_thread+0x570/0x570 [ 7220.260133] kthread+0x1d1/0x200 [ 7220.261290] ? set_kthread_struct+0x70/0x70 [ 7220.272578] ret_from_fork+0x1f/0x30 [ 7224.058418] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 7224.063753] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7224.064789] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 7224.118917] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7224.118935] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7224.118938] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7224.118955] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7224.118959] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7224.118965] RAX: 000000000000479b RBX: ffff96848811cae0 RCX: 0000000000000001 [ 7224.118967] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 7224.118970] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7224.118972] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 7224.118974] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 7224.118978] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 7224.118981] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7224.118983] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 7224.118991] Call Trace: [ 7224.118998] [ 7224.119004] ? show_regs.cold.9+0x22/0x2f [ 7224.124301] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7224.124448] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7224.124457] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7224.124459] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7224.124476] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7224.124481] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7224.124486] RAX: 000000000000721a RBX: ffff96848811cae0 RCX: 0000000000000001 [ 7224.124489] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 7224.124491] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7224.124494] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 7224.124496] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 7224.124499] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 7224.124502] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7224.124505] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 7224.124513] Call Trace: [ 7224.124520] [ 7224.124526] ? show_regs.cold.9+0x22/0x2f [ 7224.124539] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7224.124546] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7224.124550] ? watchdog+0x40/0x40 [ 7224.124555] ? hrtimer_interrupt+0x128/0x2c0 [ 7224.124558] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7224.124567] ? apic_timer_interrupt+0xf/0x20 [ 7224.124572] [ 7224.124575] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7224.124582] do_raw_spin_lock+0x177/0x190 [ 7224.124587] _raw_spin_lock+0x21/0x30 [ 7224.124593] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7224.124913] ? do_raw_spin_unlock+0x75/0x190 [ 7224.124918] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7224.134935] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7224.134958] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7224.134963] ? watchdog+0x40/0x40 [ 7224.134968] ? hrtimer_interrupt+0x128/0x2c0 [ 7224.134972] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7224.134981] ? apic_timer_interrupt+0xf/0x20 [ 7224.134986] [ 7224.134990] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7224.135000] do_raw_spin_lock+0x177/0x190 [ 7224.135007] _raw_spin_lock+0x21/0x30 [ 7224.135013] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7224.135676] ? do_raw_spin_unlock+0x75/0x190 [ 7224.135685] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7224.136280] ? do_wait_intr+0xf0/0xf0 [ 7224.136290] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7224.136568] mdc_close+0x53b/0xd70 [mdc] [ 7224.136602] lmv_close+0x18d/0x490 [lmv] [ 7224.136622] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 7224.136773] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7224.136899] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7224.136981] ll_md_close+0x274/0x800 [lustre] [ 7224.137064] ll_file_release+0x128/0xc50 [lustre] [ 7224.137151] __fput+0x10d/0x440 [ 7224.137160] ____fput+0x12/0x20 [ 7224.137164] task_work_run+0xc6/0x110 [ 7224.137170] exit_to_usermode_loop+0x1dd/0x1f0 [ 7224.137177] do_syscall_64+0x3d6/0x440 [ 7224.137180] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7224.137187] RIP: 0033:0x7ff54af45cd8 [ 7224.137193] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7224.137197] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7224.137201] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 7224.137599] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 7224.137602] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 7224.137603] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 7224.137605] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7224.634117] ? do_wait_intr+0xf0/0xf0 [ 7224.636498] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7224.641374] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7224.646972] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7224.651425] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7224.653722] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7224.656961] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7224.659440] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7224.665033] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7224.671011] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7224.680783] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7224.687041] lmv_intent_open+0x338/0x1150 [lmv] [ 7224.691625] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7224.696448] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7224.702767] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7224.706645] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7224.713140] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7224.715170] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7224.724464] ? do_raw_spin_unlock+0x75/0x190 [ 7224.728018] ? _raw_spin_unlock+0x12/0x30 [ 7224.732262] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7224.735988] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7224.741807] ll_atomic_open+0x36a/0xd70 [lustre] [ 7224.745295] lookup_open+0x356/0xb40 [ 7224.747112] do_last+0x313/0xfc0 [ 7224.749898] ? nd_jump_root+0xe5/0x160 [ 7224.753469] ? path_init+0x437/0x520 [ 7224.756933] path_openat+0xf7/0x500 [ 7224.762129] ? do_raw_spin_unlock+0x75/0x190 [ 7224.766154] do_filp_open+0x99/0x140 [ 7224.769816] ? getname_flags+0x6e/0x330 [ 7224.771182] ? __check_object_size+0xff/0x256 [ 7224.785485] ? do_raw_spin_unlock+0x75/0x190 [ 7224.789285] ? _raw_spin_unlock+0x12/0x30 [ 7224.799880] do_sys_openat2+0x2b4/0x410 [ 7224.801291] do_sys_open+0x73/0xa0 [ 7224.809977] __x64_sys_openat+0x24/0x30 [ 7224.814134] do_syscall_64+0xc1/0x440 [ 7224.817946] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7224.820153] RIP: 0033:0x7ff54af45332 [ 7224.824491] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7224.842740] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7224.852789] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 7224.856920] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 7224.864852] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 7224.867875] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 7224.872113] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 7233.999961] rcu: INFO: rcu_sched self-detected stall on CPU [ 7234.006841] rcu: 0-...!: (2748320 ticks this GP) idle=98a/1/0x4000000000000002 softirq=125154/125154 fqs=959 [ 7234.012562] (t=5280159 jiffies g=253585 q=4504) [ 7234.014647] rcu: rcu_sched kthread starved for 5275119 jiffies! g253585 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [ 7234.019725] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 7234.023776] rcu: RCU grace-period kthread stack dump: [ 7234.027533] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 7234.033381] Call Trace: [ 7234.034619] __schedule+0x351/0xcb0 [ 7234.037379] schedule+0xc0/0x180 [ 7234.039286] schedule_timeout+0xb4/0x190 [ 7234.041871] ? do_raw_spin_unlock+0x75/0x190 [ 7234.043625] ? __next_timer_interrupt+0x160/0x160 [ 7234.046878] ? prepare_to_swait_event+0xf3/0x140 [ 7234.049646] rcu_gp_kthread+0x8dd/0x15b0 [ 7234.052318] ? force_qs_rnp+0x2c0/0x2c0 [ 7234.054651] kthread+0x1d1/0x200 [ 7234.057318] ? set_kthread_struct+0x70/0x70 [ 7234.059316] ret_from_fork+0x1f/0x30 [ 7234.060929] rcu: Stack dump where RCU GP kthread last ran: [ 7234.064294] Sending NMI from CPU 0 to CPUs 1: [ 7234.068610] NMI backtrace for cpu 1 skipped: idling at native_safe_halt+0x2e/0x50 [ 7234.069572] NMI backtrace for cpu 0 [ 7234.081088] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7234.086591] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7234.090537] Call Trace: [ 7234.092323] [ 7234.093503] ? dump_stack+0xbb/0x10e [ 7234.095536] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 7234.097909] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 7234.100884] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 7234.103730] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 7234.106854] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 7234.109888] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 7234.112529] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 7234.115420] ? update_process_times+0x6d/0xc0 [ 7234.117793] ? tick_sched_handle.isra.18+0x27/0x90 [ 7234.120627] ? tick_sched_timer+0x53/0xb0 [ 7234.123057] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7234.125660] ? tick_sched_handle.isra.18+0x90/0x90 [ 7234.128559] ? hrtimer_interrupt+0x128/0x2c0 [ 7234.135804] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7234.140716] ? apic_timer_interrupt+0xf/0x20 [ 7234.147859] [ 7234.150373] ? native_safe_halt+0x2e/0x50 [ 7234.152900] ? kvm_wait+0xcd/0xe0 [ 7234.162052] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7234.164157] ? do_raw_spin_lock+0x177/0x190 [ 7234.165697] ? _raw_spin_lock+0x21/0x30 [ 7234.167205] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7234.169448] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 7234.171529] ? mdc_commit_open+0x91/0xb0 [mdc] [ 7234.173355] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7234.175555] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7234.178197] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 7234.180366] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7234.182461] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7234.184365] ? do_wait_intr+0xf0/0xf0 [ 7234.185781] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7234.187899] ? kthread+0x1d1/0x200 [ 7234.189293] ? set_kthread_struct+0x70/0x70 [ 7234.190774] ? ret_from_fork+0x1f/0x30 [ 7245.279198] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5286s! [ 7245.284426] Showing busy workqueues and worker pools: [ 7245.286511] workqueue events: flags=0x0 [ 7245.288419] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7245.290924] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 7245.301610] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7245.304389] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 7245.316424] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7245.325990] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 7245.341629] workqueue events_long: flags=0x0 [ 7245.343031] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7245.352625] pending: lu_objects_flush [obdclass] [ 7245.365911] workqueue events_unbound: flags=0x2 [ 7245.373924] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7245.382300] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7245.404935] workqueue events_power_efficient: flags=0x80 [ 7245.411308] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7245.418646] pending: neigh_periodic_work [ 7245.422262] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7245.425769] pending: do_cache_clean, check_lifetime [ 7245.428685] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7245.443886] pending: neigh_periodic_work [ 7245.445265] workqueue writeback: flags=0x4a [ 7245.449170] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7245.457319] pending: wb_workfn, wb_workfn [ 7245.459153] workqueue xprtiod: flags=0xa [ 7245.460382] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7245.469182] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 7245.475343] workqueue ipv6_addrconf: flags=0x40008 [ 7245.479479] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7245.482416] pending: addrconf_verify_work [ 7245.487065] workqueue knbd0-recv: flags=0x1a [ 7245.490015] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7245.501078] in-flight: 657:recv_work [ 7245.509797] workqueue ptlrpc_pinger: flags=0x2 [ 7245.512892] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7245.514856] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 7245.522817] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5286s workers=4 idle: 20831 12642 28859 [ 7245.527856] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7235s workers=2 idle: 659 [ 7248.059061] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 7248.063046] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7248.086335] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7248.091898] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7248.096531] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7248.099823] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7248.101589] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7248.111602] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7248.114662] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 7248.119318] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 7248.122518] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 7248.126174] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 7248.129621] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 7248.133171] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 7248.137053] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7248.139845] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 7248.142754] Call Trace: [ 7248.144499] [ 7248.145600] ? show_regs.cold.9+0x22/0x2f [ 7248.147683] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7248.149928] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7248.151879] ? watchdog+0x40/0x40 [ 7248.153280] ? hrtimer_interrupt+0x128/0x2c0 [ 7248.155762] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7248.159234] ? apic_timer_interrupt+0xf/0x20 [ 7248.162324] [ 7248.163325] ? native_safe_halt+0x2e/0x50 [ 7248.166127] kvm_wait+0xcd/0xe0 [ 7248.167528] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7248.170226] do_raw_spin_lock+0x177/0x190 [ 7248.171867] _raw_spin_lock+0x21/0x30 [ 7248.175174] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7248.178724] process_one_work+0x2c8/0x700 [ 7248.180758] worker_thread+0x296/0x6e0 [ 7248.182909] ? rescuer_thread+0x570/0x570 [ 7248.184875] kthread+0x1d1/0x200 [ 7248.186667] ? set_kthread_struct+0x70/0x70 [ 7248.188379] ret_from_fork+0x1f/0x30 [ 7252.060050] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [ 7252.068371] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7252.137601] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7252.150444] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7252.164097] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7252.165722] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7252.207440] RSP: 0000:ffffaf9fc15234f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7252.213467] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 7252.224213] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 7252.232297] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7252.236236] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 7252.244561] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 7252.257149] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 7252.268966] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7252.283074] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 7252.294135] Call Trace: [ 7252.297794] [ 7252.299664] ? show_regs.cold.9+0x22/0x2f [ 7252.304067] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7252.310233] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7252.314388] ? watchdog+0x40/0x40 [ 7252.319418] ? hrtimer_interrupt+0x128/0x2c0 [ 7252.321020] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7252.329625] ? apic_timer_interrupt+0xf/0x20 [ 7252.336089] [ 7252.338341] ? native_safe_halt+0x2e/0x50 [ 7252.343649] kvm_wait+0xcd/0xe0 [ 7252.345195] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7252.353380] do_raw_spin_lock+0x177/0x190 [ 7252.355192] _raw_spin_lock+0x21/0x30 [ 7252.360393] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7252.367523] ? do_raw_spin_unlock+0x75/0x190 [ 7252.369386] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7252.378352] ? do_wait_intr+0xf0/0xf0 [ 7252.386176] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7252.394655] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7252.401509] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7252.405358] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7252.409304] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7252.413113] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7252.417290] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7252.420600] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7252.422823] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7252.428606] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7252.431683] lmv_intent_open+0x338/0x1150 [lmv] [ 7252.436268] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7252.439664] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7252.443365] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7252.446262] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7252.449539] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7252.453169] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7252.459059] ? do_raw_spin_unlock+0x75/0x190 [ 7252.460581] ? _raw_spin_unlock+0x12/0x30 [ 7252.461940] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7252.469389] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7252.474400] ll_atomic_open+0x36a/0xd70 [lustre] [ 7252.478231] lookup_open+0x356/0xb40 [ 7252.480431] do_last+0x313/0xfc0 [ 7252.482662] ? nd_jump_root+0xe5/0x160 [ 7252.485222] ? path_init+0x437/0x520 [ 7252.487652] path_openat+0xf7/0x500 [ 7252.489411] ? do_raw_spin_unlock+0x75/0x190 [ 7252.495370] do_filp_open+0x99/0x140 [ 7252.506972] ? getname_flags+0x6e/0x330 [ 7252.512222] ? __check_object_size+0xff/0x256 [ 7252.515131] ? do_raw_spin_unlock+0x75/0x190 [ 7252.516335] ? _raw_spin_unlock+0x12/0x30 [ 7252.521841] do_sys_openat2+0x2b4/0x410 [ 7252.524328] do_sys_open+0x73/0xa0 [ 7252.527582] __x64_sys_openat+0x24/0x30 [ 7252.529881] do_syscall_64+0xc1/0x440 [ 7252.530922] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7252.535417] RIP: 0033:0x7ff54af45332 [ 7252.537855] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7252.552246] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7252.563120] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 7252.571417] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 7252.580123] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 7252.587294] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 7252.598380] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 7260.058138] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 7260.061881] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7260.080190] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7260.084493] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7260.089040] RIP: 0010:__pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 7260.092209] Code: d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 05 18 d4 dc 02 01 e0 48 83 05 06 d4 dc 02 01 48 83 05 be d5 dc 02 01 8b 45 08 85 [ 7260.100053] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7260.102550] RAX: 00000000000025ce RBX: ffff96848811cae0 RCX: 0000000000000001 [ 7260.104623] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 7260.108324] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7260.110471] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 7260.114039] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 7260.117112] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 7260.119898] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7260.122364] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 7260.125725] Call Trace: [ 7260.126964] [ 7260.127935] ? show_regs.cold.9+0x22/0x2f [ 7260.130964] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7260.133100] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7260.135515] ? watchdog+0x40/0x40 [ 7260.137238] ? hrtimer_interrupt+0x128/0x2c0 [ 7260.139183] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7260.141522] ? apic_timer_interrupt+0xf/0x20 [ 7260.144060] [ 7260.145080] ? __pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 7260.147098] do_raw_spin_lock+0x177/0x190 [ 7260.148474] _raw_spin_lock+0x21/0x30 [ 7260.150393] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7260.153256] ? do_raw_spin_unlock+0x75/0x190 [ 7260.156768] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7260.160133] ? do_wait_intr+0xf0/0xf0 [ 7260.162900] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7260.166111] mdc_close+0x53b/0xd70 [mdc] [ 7260.168146] lmv_close+0x18d/0x490 [lmv] [ 7260.170550] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 7260.173989] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7260.177249] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7260.180267] ll_md_close+0x274/0x800 [lustre] [ 7260.183062] ll_file_release+0x128/0xc50 [lustre] [ 7260.185818] __fput+0x10d/0x440 [ 7260.187810] ____fput+0x12/0x20 [ 7260.189579] task_work_run+0xc6/0x110 [ 7260.191268] exit_to_usermode_loop+0x1dd/0x1f0 [ 7260.193688] do_syscall_64+0x3d6/0x440 [ 7260.195813] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7260.198519] RIP: 0033:0x7ff54af45cd8 [ 7260.200328] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7260.210611] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7260.214895] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 7260.218076] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 7260.221071] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 7260.224805] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 7260.228523] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7272.058931] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 7272.070649] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7272.131304] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7272.143666] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7272.147872] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7272.155001] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7272.172954] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7272.180821] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 7272.192870] RDX: 0000000000006637 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 7272.213142] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 7272.223303] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 7272.233326] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7272.243645] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 7272.259546] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7272.267816] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 7272.281171] Call Trace: [ 7272.286867] [ 7272.287615] ? show_regs.cold.9+0x22/0x2f [ 7272.300137] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7272.309457] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7272.324986] ? watchdog+0x40/0x40 [ 7272.329494] ? hrtimer_interrupt+0x128/0x2c0 [ 7272.334520] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7272.336940] ? apic_timer_interrupt+0xf/0x20 [ 7272.343068] [ 7272.345174] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7272.356339] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7272.364854] do_raw_spin_lock+0x177/0x190 [ 7272.375424] _raw_spin_lock+0x21/0x30 [ 7272.376567] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7272.380023] obd_mod_free+0x46/0x140 [ptlrpc] [ 7272.382936] mdc_commit_open+0x91/0xb0 [mdc] [ 7272.384699] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7272.387987] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7272.392989] after_reply+0xe2c/0x1780 [ptlrpc] [ 7272.397195] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7272.400000] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7272.401423] ? do_wait_intr+0xf0/0xf0 [ 7272.405350] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7272.412073] kthread+0x1d1/0x200 [ 7272.412912] ? set_kthread_struct+0x70/0x70 [ 7272.413736] ret_from_fork+0x1f/0x30 [ 7275.999084] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5317s! [ 7276.003579] Showing busy workqueues and worker pools: [ 7276.006280] workqueue events: flags=0x0 [ 7276.007772] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7276.009570] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 7276.012652] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7276.015045] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 7276.018200] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7276.021467] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 7276.023940] workqueue events_long: flags=0x0 [ 7276.027509] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7276.032039] pending: lu_objects_flush [obdclass] [ 7276.038387] workqueue events_unbound: flags=0x2 [ 7276.040115] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7276.043280] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7276.050426] workqueue events_power_efficient: flags=0x80 [ 7276.052635] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7276.057642] pending: neigh_periodic_work [ 7276.059476] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 7276.061541] pwq 4: [ 7276.067827] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7276.067921] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7276.067938] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7276.068127] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7276.068537] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7276.068545] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7276.068548] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7276.068552] RAX: 0000000000005f8a RBX: ffff96848811cae0 RCX: 0000000000000001 [ 7276.068553] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 7276.068555] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 7276.068557] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 7276.068559] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 7276.068561] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 7276.068563] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7276.068565] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 7276.068571] Call Trace: [ 7276.068577] [ 7276.068581] ? show_regs.cold.9+0x22/0x2f [ 7276.068589] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7276.068594] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7276.068597] ? watchdog+0x40/0x40 [ 7276.068600] ? hrtimer_interrupt+0x128/0x2c0 [ 7276.068602] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7276.068610] ? apic_timer_interrupt+0xf/0x20 [ 7276.068614] [ 7276.068616] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7276.068618] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7276.068622] do_raw_spin_lock+0x177/0x190 [ 7276.068625] _raw_spin_lock+0x21/0x30 [ 7276.068628] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7276.069176] process_one_work+0x2c8/0x700 [ 7276.069185] worker_thread+0x296/0x6e0 [ 7276.069190] ? rescuer_thread+0x570/0x570 [ 7276.069194] kthread+0x1d1/0x200 [ 7276.069199] ? set_kthread_struct+0x70/0x70 [ 7276.069203] ret_from_fork+0x1f/0x30 [ 7276.229629] cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7276.232228] pending: do_cache_clean, check_lifetime [ 7276.235224] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7276.239179] pending: neigh_periodic_work [ 7276.241340] workqueue writeback: flags=0x4a [ 7276.243722] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7276.246883] pending: wb_workfn, wb_workfn [ 7276.249612] workqueue xprtiod: flags=0xa [ 7276.254591] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7276.259387] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 7276.263405] workqueue ipv6_addrconf: flags=0x40008 [ 7276.265157] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7276.270212] pending: addrconf_verify_work [ 7276.272465] workqueue knbd0-recv: flags=0x1a [ 7276.275050] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7276.278916] in-flight: 657:recv_work [ 7276.281727] workqueue ptlrpc_pinger: flags=0x2 [ 7276.283021] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7276.287580] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 7276.290436] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5317s workers=4 idle: 20831 12642 28859 [ 7276.296748] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7266s workers=2 idle: 659 [ 7280.060404] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [ 7280.065980] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7280.104878] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7280.109378] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7280.116111] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7280.119645] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7280.127572] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7280.131310] RAX: 0000000000003c41 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 7280.134356] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 7280.137597] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7280.140802] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 7280.143505] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 7280.145541] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 7280.148178] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7280.150544] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 7280.154062] Call Trace: [ 7280.155002] [ 7280.155786] ? show_regs.cold.9+0x22/0x2f [ 7280.158328] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7280.159932] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7280.161371] ? watchdog+0x40/0x40 [ 7280.162489] ? hrtimer_interrupt+0x128/0x2c0 [ 7280.163993] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7280.165797] ? apic_timer_interrupt+0xf/0x20 [ 7280.168014] [ 7280.168831] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7280.170076] do_raw_spin_lock+0x177/0x190 [ 7280.171163] _raw_spin_lock+0x21/0x30 [ 7280.172859] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7280.174214] ? do_raw_spin_unlock+0x75/0x190 [ 7280.175593] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7280.177537] ? do_wait_intr+0xf0/0xf0 [ 7280.179426] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7280.181373] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7280.183144] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7280.185944] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7280.188155] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7280.190209] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7280.193417] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7280.195115] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7280.197317] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7280.200494] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7280.202328] lmv_intent_open+0x338/0x1150 [lmv] [ 7280.204230] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7280.206240] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7280.208729] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7280.210394] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7280.213011] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7280.216461] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7280.218542] ? do_raw_spin_unlock+0x75/0x190 [ 7280.221710] ? _raw_spin_unlock+0x12/0x30 [ 7280.223422] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7280.226671] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7280.229709] ll_atomic_open+0x36a/0xd70 [lustre] [ 7280.232236] lookup_open+0x356/0xb40 [ 7280.234307] do_last+0x313/0xfc0 [ 7280.236244] ? nd_jump_root+0xe5/0x160 [ 7280.237962] ? path_init+0x437/0x520 [ 7280.239296] path_openat+0xf7/0x500 [ 7280.240684] ? do_raw_spin_unlock+0x75/0x190 [ 7280.243022] do_filp_open+0x99/0x140 [ 7280.244132] ? getname_flags+0x6e/0x330 [ 7280.245377] ? __check_object_size+0xff/0x256 [ 7280.247135] ? do_raw_spin_unlock+0x75/0x190 [ 7280.248491] ? _raw_spin_unlock+0x12/0x30 [ 7280.249953] do_sys_openat2+0x2b4/0x410 [ 7280.251394] do_sys_open+0x73/0xa0 [ 7280.252771] __x64_sys_openat+0x24/0x30 [ 7280.254094] do_syscall_64+0xc1/0x440 [ 7280.255420] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7280.257335] RIP: 0033:0x7ff54af45332 [ 7280.258897] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7280.265118] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7280.267584] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 7280.270334] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 7280.273135] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 7280.276118] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 7280.278846] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 7288.058036] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 7288.066739] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7288.101626] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7288.109706] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7288.118172] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7288.122520] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7288.135616] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7288.140498] RAX: 0000000000003c82 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 7288.145333] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 7288.153317] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7288.159108] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 7288.163965] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 7288.168721] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 7288.175256] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7288.179243] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 7288.184941] Call Trace: [ 7288.186655] [ 7288.188175] ? show_regs.cold.9+0x22/0x2f [ 7288.190637] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7288.196858] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7288.200146] ? watchdog+0x40/0x40 [ 7288.202114] ? hrtimer_interrupt+0x128/0x2c0 [ 7288.204867] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7288.208336] ? apic_timer_interrupt+0xf/0x20 [ 7288.211914] [ 7288.213264] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7288.218839] do_raw_spin_lock+0x177/0x190 [ 7288.222014] _raw_spin_lock+0x21/0x30 [ 7288.224109] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7288.227382] ? do_raw_spin_unlock+0x75/0x190 [ 7288.231757] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7288.235628] ? do_wait_intr+0xf0/0xf0 [ 7288.237755] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7288.241610] mdc_close+0x53b/0xd70 [mdc] [ 7288.244422] lmv_close+0x18d/0x490 [lmv] [ 7288.247102] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 7288.253184] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7288.257766] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7288.261107] ll_md_close+0x274/0x800 [lustre] [ 7288.264352] ll_file_release+0x128/0xc50 [lustre] [ 7288.267393] __fput+0x10d/0x440 [ 7288.269466] ____fput+0x12/0x20 [ 7288.271397] task_work_run+0xc6/0x110 [ 7288.274373] exit_to_usermode_loop+0x1dd/0x1f0 [ 7288.277099] do_syscall_64+0x3d6/0x440 [ 7288.280162] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7288.283097] RIP: 0033:0x7ff54af45cd8 [ 7288.285193] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7288.296046] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7288.300515] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 7288.304302] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 7288.311272] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 7288.316102] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 7288.320434] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7304.055621] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 7304.059765] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 7304.061164] Modules linked in: lustre(O) [ 7304.065158] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7304.065249] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7304.065257] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7304.065265] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7304.098153] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7304.098237] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7304.098246] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7304.098248] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7304.098262] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7304.098267] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7304.098272] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 7304.098275] RDX: 0000000000005bd8 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 7304.098278] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 7304.098280] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 7304.098283] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7304.098286] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 7304.098290] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7304.098292] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 7304.098300] Call Trace: [ 7304.098306] [ 7304.098312] ? show_regs.cold.9+0x22/0x2f [ 7304.098322] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7304.098327] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7304.098332] ? watchdog+0x40/0x40 [ 7304.098335] ? hrtimer_interrupt+0x128/0x2c0 [ 7304.098339] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7304.098348] ? apic_timer_interrupt+0xf/0x20 [ 7304.098352] [ 7304.098355] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7304.098360] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7304.098364] do_raw_spin_lock+0x177/0x190 [ 7304.098368] _raw_spin_lock+0x21/0x30 [ 7304.098374] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7304.103343] [ 7304.103355] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7304.103374] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7304.103378] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7304.103383] RAX: 0000000000007991 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 7304.103385] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 7304.103387] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 7304.103389] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 7304.103392] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 7304.103395] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 7304.103398] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7304.103400] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 7304.103473] Call Trace: [ 7304.103478] [ 7304.103485] ? show_regs.cold.9+0x22/0x2f [ 7304.103497] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7304.103504] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7304.103508] ? watchdog+0x40/0x40 [ 7304.103513] ? hrtimer_interrupt+0x128/0x2c0 [ 7304.103517] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7304.103525] ? apic_timer_interrupt+0xf/0x20 [ 7304.103529] [ 7304.103532] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7304.103536] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7304.103543] do_raw_spin_lock+0x177/0x190 [ 7304.103548] _raw_spin_lock+0x21/0x30 [ 7304.103552] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7304.103851] process_one_work+0x2c8/0x700 [ 7304.103858] worker_thread+0x296/0x6e0 [ 7304.103863] ? rescuer_thread+0x570/0x570 [ 7304.103866] kthread+0x1d1/0x200 [ 7304.103871] ? set_kthread_struct+0x70/0x70 [ 7304.103875] ret_from_fork+0x1f/0x30 [ 7304.406721] obd_mod_free+0x46/0x140 [ptlrpc] [ 7304.411363] mdc_commit_open+0x91/0xb0 [mdc] [ 7304.415826] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7304.421718] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7304.427433] after_reply+0xe2c/0x1780 [ptlrpc] [ 7304.438168] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7304.446163] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7304.452651] ? do_wait_intr+0xf0/0xf0 [ 7304.454282] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7304.458136] kthread+0x1d1/0x200 [ 7304.460124] ? set_kthread_struct+0x70/0x70 [ 7304.462665] ret_from_fork+0x1f/0x30 [ 7306.719101] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5347s! [ 7306.722856] Showing busy workqueues and worker pools: [ 7306.725961] workqueue events: flags=0x0 [ 7306.727336] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7306.730136] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 7306.734138] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7306.736953] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 7306.741757] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7306.744280] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 7306.746815] workqueue events_long: flags=0x0 [ 7306.748726] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7306.751571] pending: lu_objects_flush [obdclass] [ 7306.753955] workqueue events_unbound: flags=0x2 [ 7306.756094] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7306.758995] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7306.763653] workqueue events_power_efficient: flags=0x80 [ 7306.766459] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7306.769861] pending: neigh_periodic_work [ 7306.772808] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7306.776529] pending: do_cache_clean, check_lifetime [ 7306.778714] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7306.782540] pending: neigh_periodic_work [ 7306.784991] workqueue writeback: flags=0x4a [ 7306.788229] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7306.792903] pending: wb_workfn, wb_workfn [ 7306.795706] workqueue xprtiod: flags=0xa [ 7306.799813] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7306.804875] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 7306.809330] workqueue ipv6_addrconf: flags=0x40008 [ 7306.812283] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7306.819600] pending: addrconf_verify_work [ 7306.825484] workqueue knbd0-recv: flags=0x1a [ 7306.827095] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7306.830075] in-flight: 657:recv_work [ 7306.831753] workqueue ptlrpc_pinger: flags=0x2 [ 7306.833568] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7306.835880] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 7306.838485] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5347s workers=4 idle: 20831 12642 28859 [ 7306.841888] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7297s workers=2 idle: 659 [ 7308.060071] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [ 7308.066713] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7308.102548] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7308.107613] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7308.111725] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7308.115325] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7308.123748] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7308.127714] RAX: 0000000000007534 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 7308.130707] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 7308.133513] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7308.138459] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 7308.146701] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 7308.149771] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 7308.153661] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7308.155977] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 7308.159468] Call Trace: [ 7308.160583] [ 7308.161623] ? show_regs.cold.9+0x22/0x2f [ 7308.163314] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7308.165251] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7308.166811] ? watchdog+0x40/0x40 [ 7308.168448] ? hrtimer_interrupt+0x128/0x2c0 [ 7308.170716] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7308.172777] ? apic_timer_interrupt+0xf/0x20 [ 7308.174032] [ 7308.175927] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7308.182473] do_raw_spin_lock+0x177/0x190 [ 7308.183412] _raw_spin_lock+0x21/0x30 [ 7308.184287] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7308.190788] ? do_raw_spin_unlock+0x75/0x190 [ 7308.196752] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7308.200647] ? do_wait_intr+0xf0/0xf0 [ 7308.201927] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7308.208804] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7308.210232] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7308.211723] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7308.214095] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7308.217629] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7308.220754] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7308.222281] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7308.232186] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7308.238092] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7308.242983] lmv_intent_open+0x338/0x1150 [lmv] [ 7308.244612] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7308.247725] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7308.250731] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7308.257669] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7308.260719] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7308.271020] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7308.274526] ? do_raw_spin_unlock+0x75/0x190 [ 7308.282744] ? _raw_spin_unlock+0x12/0x30 [ 7308.286357] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7308.288102] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7308.299330] ll_atomic_open+0x36a/0xd70 [lustre] [ 7308.304350] lookup_open+0x356/0xb40 [ 7308.306710] do_last+0x313/0xfc0 [ 7308.307617] ? nd_jump_root+0xe5/0x160 [ 7308.308866] ? path_init+0x437/0x520 [ 7308.312703] path_openat+0xf7/0x500 [ 7308.313513] ? do_raw_spin_unlock+0x75/0x190 [ 7308.315259] do_filp_open+0x99/0x140 [ 7308.316294] ? getname_flags+0x6e/0x330 [ 7308.317918] ? __check_object_size+0xff/0x256 [ 7308.319656] ? do_raw_spin_unlock+0x75/0x190 [ 7308.326225] ? _raw_spin_unlock+0x12/0x30 [ 7308.328206] do_sys_openat2+0x2b4/0x410 [ 7308.330148] do_sys_open+0x73/0xa0 [ 7308.330905] __x64_sys_openat+0x24/0x30 [ 7308.331753] do_syscall_64+0xc1/0x440 [ 7308.336544] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7308.344111] RIP: 0033:0x7ff54af45332 [ 7308.350458] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7308.365744] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7308.371933] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 7308.377600] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 7308.380619] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 7308.387736] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 7308.394605] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 7316.062398] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 7316.080041] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7316.146265] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7316.158220] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7316.170569] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7316.180390] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7316.204513] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7316.216015] RAX: 0000000000005558 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 7316.230849] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 7316.245073] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7316.253433] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 7316.265675] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 7316.278806] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 7316.300685] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7316.312166] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 7316.323321] Call Trace: [ 7316.324391] [ 7316.325256] ? show_regs.cold.9+0x22/0x2f [ 7316.340920] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7316.348483] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7316.349840] ? watchdog+0x40/0x40 [ 7316.356354] ? hrtimer_interrupt+0x128/0x2c0 [ 7316.358428] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7316.367658] ? apic_timer_interrupt+0xf/0x20 [ 7316.370874] [ 7316.372993] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7316.381869] do_raw_spin_lock+0x177/0x190 [ 7316.385110] _raw_spin_lock+0x21/0x30 [ 7316.388583] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7316.393850] ? do_raw_spin_unlock+0x75/0x190 [ 7316.401656] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7316.409816] ? do_wait_intr+0xf0/0xf0 [ 7316.417535] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7316.433589] mdc_close+0x53b/0xd70 [mdc] [ 7316.438999] lmv_close+0x18d/0x490 [lmv] [ 7316.444908] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 7316.449529] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7316.452939] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7316.456482] ll_md_close+0x274/0x800 [lustre] [ 7316.458971] ll_file_release+0x128/0xc50 [lustre] [ 7316.461104] __fput+0x10d/0x440 [ 7316.462842] ____fput+0x12/0x20 [ 7316.464265] task_work_run+0xc6/0x110 [ 7316.469947] exit_to_usermode_loop+0x1dd/0x1f0 [ 7316.481221] do_syscall_64+0x3d6/0x440 [ 7316.484988] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7316.492353] RIP: 0033:0x7ff54af45cd8 [ 7316.495481] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7316.516365] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7316.520799] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 7316.530609] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 7316.541620] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 7316.549026] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 7316.561812] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7332.059174] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 7332.062469] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 7332.063662] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7332.063838] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7332.063845] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7332.063851] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7332.064371] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7332.064382] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7332.064387] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7332.064391] RAX: 000000000000060f RBX: ffff96848811cae0 RCX: 0000000000000001 [ 7332.064393] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 7332.064396] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 7332.064398] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 7332.064400] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 7332.064403] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 7332.064406] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7332.064408] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 7332.064574] Call Trace: [ 7332.064581] [ 7332.064587] ? show_regs.cold.9+0x22/0x2f [ 7332.064597] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7332.064604] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7332.064609] ? watchdog+0x40/0x40 [ 7332.064614] ? hrtimer_interrupt+0x128/0x2c0 [ 7332.064618] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7332.064626] ? apic_timer_interrupt+0xf/0x20 [ 7332.064631] [ 7332.064635] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7332.064640] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7332.064645] do_raw_spin_lock+0x177/0x190 [ 7332.064650] _raw_spin_lock+0x21/0x30 [ 7332.064655] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7332.064895] process_one_work+0x2c8/0x700 [ 7332.064903] worker_thread+0x296/0x6e0 [ 7332.064908] ? rescuer_thread+0x570/0x570 [ 7332.064911] kthread+0x1d1/0x200 [ 7332.064915] ? set_kthread_struct+0x70/0x70 [ 7332.064919] ret_from_fork+0x1f/0x30 [ 7332.309013] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7332.342662] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7332.346991] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7332.350614] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7332.353882] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7332.364637] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7332.368818] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 7332.376848] RDX: 0000000000002022 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 7332.383040] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 7332.388831] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 7332.392167] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7332.399210] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 7332.413829] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7332.421693] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 7332.431037] Call Trace: [ 7332.434981] [ 7332.437006] ? show_regs.cold.9+0x22/0x2f [ 7332.443838] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7332.449783] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7332.457433] ? watchdog+0x40/0x40 [ 7332.461830] ? hrtimer_interrupt+0x128/0x2c0 [ 7332.471358] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7332.476986] ? apic_timer_interrupt+0xf/0x20 [ 7332.485640] [ 7332.490512] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7332.495456] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7332.503346] do_raw_spin_lock+0x177/0x190 [ 7332.511694] _raw_spin_lock+0x21/0x30 [ 7332.521000] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7332.525144] obd_mod_free+0x46/0x140 [ptlrpc] [ 7332.534238] mdc_commit_open+0x91/0xb0 [mdc] [ 7332.546900] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7332.563000] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7332.571092] after_reply+0xe2c/0x1780 [ptlrpc] [ 7332.579000] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7332.584775] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7332.591007] ? do_wait_intr+0xf0/0xf0 [ 7332.597025] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7332.603312] kthread+0x1d1/0x200 [ 7332.607097] ? set_kthread_struct+0x70/0x70 [ 7332.621853] ret_from_fork+0x1f/0x30 [ 7336.061044] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [ 7336.066760] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7336.114543] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7336.126597] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7336.136903] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7336.140418] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7336.151383] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7336.157031] RAX: 0000000000006109 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 7336.161839] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 7336.165998] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7336.171437] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 7336.175684] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 7336.180941] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 7336.186851] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7336.191748] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 7336.196556] Call Trace: [ 7336.198952] [ 7336.201404] ? show_regs.cold.9+0x22/0x2f [ 7336.205075] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7336.209589] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7336.213495] ? watchdog+0x40/0x40 [ 7336.214709] ? hrtimer_interrupt+0x128/0x2c0 [ 7336.217348] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7336.223288] ? apic_timer_interrupt+0xf/0x20 [ 7336.225968] [ 7336.226832] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7336.233908] do_raw_spin_lock+0x177/0x190 [ 7336.237236] _raw_spin_lock+0x21/0x30 [ 7336.240962] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7336.244053] ? do_raw_spin_unlock+0x75/0x190 [ 7336.248366] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7336.251945] ? do_wait_intr+0xf0/0xf0 [ 7336.253426] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7336.255799] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7336.259031] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7336.263327] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7336.267399] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7336.270827] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7336.272411] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7336.274003] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7336.280004] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7336.283540] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7336.294418] lmv_intent_open+0x338/0x1150 [lmv] [ 7336.297048] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7336.304322] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7336.308524] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7336.311995] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7336.314514] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7336.317991] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7336.321961] ? do_raw_spin_unlock+0x75/0x190 [ 7336.328481] ? _raw_spin_unlock+0x12/0x30 [ 7336.331319] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7336.337623] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7336.340023] ll_atomic_open+0x36a/0xd70 [lustre] [ 7336.342844] lookup_open+0x356/0xb40 [ 7336.346508] do_last+0x313/0xfc0 [ 7336.348328] ? nd_jump_root+0xe5/0x160 [ 7336.355735] ? path_init+0x437/0x520 [ 7336.358073] path_openat+0xf7/0x500 [ 7336.362657] ? do_raw_spin_unlock+0x75/0x190 [ 7336.364487] do_filp_open+0x99/0x140 [ 7336.366945] ? getname_flags+0x6e/0x330 [ 7336.372610] ? __check_object_size+0xff/0x256 [ 7336.379831] ? do_raw_spin_unlock+0x75/0x190 [ 7336.382318] ? _raw_spin_unlock+0x12/0x30 [ 7336.384527] do_sys_openat2+0x2b4/0x410 [ 7336.386657] do_sys_open+0x73/0xa0 [ 7336.388572] __x64_sys_openat+0x24/0x30 [ 7336.395415] do_syscall_64+0xc1/0x440 [ 7336.400210] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7336.403405] RIP: 0033:0x7ff54af45332 [ 7336.408699] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7336.416921] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7336.420477] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 7336.423589] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 7336.426863] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 7336.430290] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 7336.433666] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 7336.927171] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5378s! [ 7336.931152] Showing busy workqueues and worker pools: [ 7336.933884] workqueue events: flags=0x0 [ 7336.935419] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7336.941044] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 7336.945686] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7336.956939] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 7336.982334] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7336.992050] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 7337.001340] workqueue events_long: flags=0x0 [ 7337.006835] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7337.020016] pending: lu_objects_flush [obdclass] [ 7337.029979] workqueue events_unbound: flags=0x2 [ 7337.037203] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7337.039674] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7337.068735] workqueue events_power_efficient: flags=0x80 [ 7337.085067] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7337.091375] pending: neigh_periodic_work [ 7337.109580] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7337.111876] pending: do_cache_clean, check_lifetime [ 7337.121032] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7337.128557] pending: neigh_periodic_work [ 7337.130878] workqueue writeback: flags=0x4a [ 7337.137072] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7337.146041] pending: wb_workfn, wb_workfn [ 7337.149957] workqueue xprtiod: flags=0xa [ 7337.153665] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7337.156859] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 7337.163822] workqueue ipv6_addrconf: flags=0x40008 [ 7337.170766] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7337.174617] pending: addrconf_verify_work [ 7337.177651] workqueue knbd0-recv: flags=0x1a [ 7337.180850] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7337.187737] in-flight: 657:recv_work [ 7337.191169] workqueue ptlrpc_pinger: flags=0x2 [ 7337.195584] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7337.198639] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 7337.201508] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5378s workers=4 idle: 20831 12642 28859 [ 7337.205039] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7327s workers=2 idle: 659 [ 7344.065016] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 7344.070407] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7344.101652] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7344.115994] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7344.134953] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7344.139909] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7344.161953] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 7344.165053] RAX: 0000000000001087 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 7344.185444] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 7344.194324] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7344.200561] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 7344.220080] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 7344.228405] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 7344.239522] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7344.248578] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 7344.258291] Call Trace: [ 7344.259540] [ 7344.260586] ? show_regs.cold.9+0x22/0x2f [ 7344.266686] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7344.271787] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7344.279559] ? watchdog+0x40/0x40 [ 7344.284242] ? hrtimer_interrupt+0x128/0x2c0 [ 7344.290781] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7344.293443] ? apic_timer_interrupt+0xf/0x20 [ 7344.295790] [ 7344.300585] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7344.302412] do_raw_spin_lock+0x177/0x190 [ 7344.306691] _raw_spin_lock+0x21/0x30 [ 7344.308327] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7344.314463] ? do_raw_spin_unlock+0x75/0x190 [ 7344.321174] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7344.325539] ? do_wait_intr+0xf0/0xf0 [ 7344.328252] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7344.330144] mdc_close+0x53b/0xd70 [mdc] [ 7344.337411] lmv_close+0x18d/0x490 [lmv] [ 7344.341297] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 7344.348195] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7344.353762] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7344.356903] ll_md_close+0x274/0x800 [lustre] [ 7344.362665] ll_file_release+0x128/0xc50 [lustre] [ 7344.365584] __fput+0x10d/0x440 [ 7344.367336] ____fput+0x12/0x20 [ 7344.368649] task_work_run+0xc6/0x110 [ 7344.370340] exit_to_usermode_loop+0x1dd/0x1f0 [ 7344.371817] do_syscall_64+0x3d6/0x440 [ 7344.373159] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7344.374422] RIP: 0033:0x7ff54af45cd8 [ 7344.375313] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7344.380985] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7344.383634] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 7344.386109] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 7344.388563] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 7344.391091] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 7344.394494] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7360.059556] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 7360.068345] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7360.101751] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7360.110751] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7360.119985] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7360.125848] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7360.130002] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7360.143067] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7360.150369] RAX: 0000000000002089 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 7360.160071] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 7360.164040] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 7360.168428] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 7360.173095] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 7360.178072] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 7360.183995] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7360.189247] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 7360.194099] Call Trace: [ 7360.195277] [ 7360.196387] ? show_regs.cold.9+0x22/0x2f [ 7360.198454] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7360.203364] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7360.206437] ? watchdog+0x40/0x40 [ 7360.208957] ? hrtimer_interrupt+0x128/0x2c0 [ 7360.213814] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7360.222285] ? apic_timer_interrupt+0xf/0x20 [ 7360.226905] [ 7360.228163] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7360.234170] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7360.237663] do_raw_spin_lock+0x177/0x190 [ 7360.239608] _raw_spin_lock+0x21/0x30 [ 7360.243218] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7360.246936] process_one_work+0x2c8/0x700 [ 7360.249058] worker_thread+0x296/0x6e0 [ 7360.251350] ? rescuer_thread+0x570/0x570 [ 7360.254218] kthread+0x1d1/0x200 [ 7360.256015] ? set_kthread_struct+0x70/0x70 [ 7360.258430] ret_from_fork+0x1f/0x30 [ 7364.064593] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [ 7364.067791] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7364.070741] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [ 7364.112683] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7364.112709] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7364.112712] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7364.112732] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7364.112741] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7364.112748] RAX: 0000000000001305 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 7364.112751] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 7364.112753] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7364.112755] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 7364.112758] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 7364.112762] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 7364.112765] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7364.112768] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 7364.112778] Call Trace: [ 7364.112785] [ 7364.112797] ? show_regs.cold.9+0x22/0x2f [ 7364.112810] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7364.112923] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7364.112929] ? watchdog+0x40/0x40 [ 7364.112933] ? hrtimer_interrupt+0x128/0x2c0 [ 7364.112937] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7364.112954] ? apic_timer_interrupt+0xf/0x20 [ 7364.112959] [ 7364.112963] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7364.112971] do_raw_spin_lock+0x177/0x190 [ 7364.112977] _raw_spin_lock+0x21/0x30 [ 7364.112983] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7364.120179] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7364.120332] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7364.120340] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7364.120343] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7364.120358] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7364.120363] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7364.120368] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 7364.120370] RDX: 0000000000001046 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 7364.120372] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 7364.120374] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 7364.120377] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7364.120380] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 7364.120384] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7364.120387] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 7364.120459] Call Trace: [ 7364.120467] [ 7364.120473] ? show_regs.cold.9+0x22/0x2f [ 7364.120483] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7364.120489] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7364.120493] ? watchdog+0x40/0x40 [ 7364.120497] ? hrtimer_interrupt+0x128/0x2c0 [ 7364.120501] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7364.120509] ? apic_timer_interrupt+0xf/0x20 [ 7364.120515] [ 7364.120517] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7364.120522] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7364.120591] do_raw_spin_lock+0x177/0x190 [ 7364.120596] _raw_spin_lock+0x21/0x30 [ 7364.120602] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7364.121196] obd_mod_free+0x46/0x140 [ptlrpc] [ 7364.124883] mdc_commit_open+0x91/0xb0 [mdc] [ 7364.124925] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7364.125162] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7364.125569] after_reply+0xe2c/0x1780 [ptlrpc] [ 7364.125768] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7364.126083] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7364.127010] ? do_wait_intr+0xf0/0xf0 [ 7364.127643] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7364.128508] kthread+0x1d1/0x200 [ 7364.128517] ? set_kthread_struct+0x70/0x70 [ 7364.128521] ret_from_fork+0x1f/0x30 [ 7364.459388] ? do_raw_spin_unlock+0x75/0x190 [ 7364.461769] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7364.465885] ? do_wait_intr+0xf0/0xf0 [ 7364.467272] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7364.470766] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7364.474446] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7364.477388] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7364.479544] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7364.481199] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7364.482978] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7364.484600] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7364.487615] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7364.489716] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7364.491563] lmv_intent_open+0x338/0x1150 [lmv] [ 7364.493339] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7364.495281] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7364.498231] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7364.499780] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7364.501348] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7364.503061] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7364.504739] ? do_raw_spin_unlock+0x75/0x190 [ 7364.505881] ? _raw_spin_unlock+0x12/0x30 [ 7364.507192] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7364.508756] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7364.510321] ll_atomic_open+0x36a/0xd70 [lustre] [ 7364.512116] lookup_open+0x356/0xb40 [ 7364.513137] do_last+0x313/0xfc0 [ 7364.514074] ? nd_jump_root+0xe5/0x160 [ 7364.515106] ? path_init+0x437/0x520 [ 7364.516220] path_openat+0xf7/0x500 [ 7364.517164] ? do_raw_spin_unlock+0x75/0x190 [ 7364.518271] do_filp_open+0x99/0x140 [ 7364.519223] ? getname_flags+0x6e/0x330 [ 7364.520327] ? __check_object_size+0xff/0x256 [ 7364.521457] ? do_raw_spin_unlock+0x75/0x190 [ 7364.522696] ? _raw_spin_unlock+0x12/0x30 [ 7364.523722] do_sys_openat2+0x2b4/0x410 [ 7364.524667] do_sys_open+0x73/0xa0 [ 7364.525564] __x64_sys_openat+0x24/0x30 [ 7364.526779] do_syscall_64+0xc1/0x440 [ 7364.530553] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7364.533239] RIP: 0033:0x7ff54af45332 [ 7364.534607] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7364.542324] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7364.544745] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 7364.547210] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 7364.550168] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 7364.553420] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 7364.555642] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 7367.649460] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5408s! [ 7367.659382] Showing busy workqueues and worker pools: [ 7367.666013] workqueue events: flags=0x0 [ 7367.671557] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7367.679948] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 7367.705306] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7367.719649] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 7367.737013] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7367.742682] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 7367.755052] workqueue events_long: flags=0x0 [ 7367.764279] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7367.774270] pending: lu_objects_flush [obdclass] [ 7367.779123] workqueue events_unbound: flags=0x2 [ 7367.783924] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7367.790948] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7367.797499] workqueue events_power_efficient: flags=0x80 [ 7367.800280] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7367.805927] pending: neigh_periodic_work [ 7367.807075] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7367.816462] pending: do_cache_clean, check_lifetime [ 7367.823904] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7367.834481] pending: neigh_periodic_work [ 7367.841325] workqueue writeback: flags=0x4a [ 7367.843502] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7367.860541] pending: wb_workfn, wb_workfn [ 7367.872195] workqueue xprtiod: flags=0xa [ 7367.884672] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7367.890704] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 7367.901169] workqueue ipv6_addrconf: flags=0x40008 [ 7367.904170] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7367.910161] pending: addrconf_verify_work [ 7367.914648] workqueue knbd0-recv: flags=0x1a [ 7367.922319] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7367.926868] in-flight: 657:recv_work [ 7367.928090] workqueue ptlrpc_pinger: flags=0x2 [ 7367.930210] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7367.937029] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 7367.940976] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5409s workers=4 idle: 20831 12642 28859 [ 7367.951321] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7358s workers=2 idle: 659 [ 7372.059558] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 7372.069508] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7372.132043] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7372.143458] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7372.160492] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7372.171501] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7372.208493] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7372.220537] RAX: 0000000000007085 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 7372.224096] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 7372.244419] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7372.265215] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 7372.267724] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 7372.291947] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 7372.298415] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7372.303854] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 7372.311172] Call Trace: [ 7372.314141] [ 7372.319068] ? show_regs.cold.9+0x22/0x2f [ 7372.329266] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7372.344323] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7372.351449] ? watchdog+0x40/0x40 [ 7372.356628] ? hrtimer_interrupt+0x128/0x2c0 [ 7372.363212] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7372.404982] ? apic_timer_interrupt+0xf/0x20 [ 7372.409150] [ 7372.410618] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7372.417205] do_raw_spin_lock+0x177/0x190 [ 7372.429256] _raw_spin_lock+0x21/0x30 [ 7372.439135] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7372.445581] ? do_raw_spin_unlock+0x75/0x190 [ 7372.461154] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7372.501542] ? do_wait_intr+0xf0/0xf0 [ 7372.507684] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7372.518607] mdc_close+0x53b/0xd70 [mdc] [ 7372.524334] lmv_close+0x18d/0x490 [lmv] [ 7372.541716] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 7372.543815] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7372.563043] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7372.578496] ll_md_close+0x274/0x800 [lustre] [ 7372.585082] ll_file_release+0x128/0xc50 [lustre] [ 7372.602969] __fput+0x10d/0x440 [ 7372.616871] ____fput+0x12/0x20 [ 7372.620418] task_work_run+0xc6/0x110 [ 7372.628354] exit_to_usermode_loop+0x1dd/0x1f0 [ 7372.634086] do_syscall_64+0x3d6/0x440 [ 7372.639899] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7372.647139] RIP: 0033:0x7ff54af45cd8 [ 7372.652384] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7372.682962] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7372.689174] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 7372.693875] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 7372.701624] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 7372.705828] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 7372.718082] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7388.060853] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 7388.096293] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7388.174500] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7388.196886] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7388.236077] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7388.238705] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7388.248037] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7388.287192] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 7388.307941] RAX: 0000000000007c8b RBX: ffff96848811cae0 RCX: 0000000000000001 [ 7388.326686] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 7388.345035] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 7388.360328] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 7388.385811] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 7388.397283] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 7388.416794] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7388.418741] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 7388.437690] Call Trace: [ 7388.448685] [ 7388.449715] ? show_regs.cold.9+0x22/0x2f [ 7388.455153] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7388.469100] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7388.470688] ? watchdog+0x40/0x40 [ 7388.486127] ? hrtimer_interrupt+0x128/0x2c0 [ 7388.501921] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7388.503593] ? apic_timer_interrupt+0xf/0x20 [ 7388.527910] [ 7388.528697] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7388.538458] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7388.543512] do_raw_spin_lock+0x177/0x190 [ 7388.552827] _raw_spin_lock+0x21/0x30 [ 7388.561259] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7388.570087] process_one_work+0x2c8/0x700 [ 7388.572894] worker_thread+0x296/0x6e0 [ 7388.587069] ? rescuer_thread+0x570/0x570 [ 7388.601646] kthread+0x1d1/0x200 [ 7388.607512] ? set_kthread_struct+0x70/0x70 [ 7388.625038] ret_from_fork+0x1f/0x30 [ 7392.055007] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 7392.060023] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 7392.062830] Modules linked in: [ 7392.066080] Modules linked in: [ 7392.066083] lustre(O) osp(O) ofd(O) [ 7392.067394] lustre(O) [ 7392.072289] lod(O) [ 7392.073878] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7392.074019] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7392.074028] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7392.074032] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7392.074049] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7392.074054] RSP: 0000:ffffaf9fc15234f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7392.074058] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 7392.074060] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 7392.074064] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7392.074067] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 7392.074069] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 7392.074074] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 7392.074077] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7392.074080] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 7392.074088] Call Trace: [ 7392.074094] [ 7392.074101] ? show_regs.cold.9+0x22/0x2f [ 7392.074113] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7392.074121] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7392.074125] ? watchdog+0x40/0x40 [ 7392.074129] ? hrtimer_interrupt+0x128/0x2c0 [ 7392.074133] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7392.074140] ? apic_timer_interrupt+0xf/0x20 [ 7392.074145] [ 7392.074148] ? native_safe_halt+0x2e/0x50 [ 7392.074152] kvm_wait+0xcd/0xe0 [ 7392.074160] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7392.074169] do_raw_spin_lock+0x177/0x190 [ 7392.074174] _raw_spin_lock+0x21/0x30 [ 7392.074178] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7392.074467] ? do_raw_spin_unlock+0x75/0x190 [ 7392.074473] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7392.074666] ? do_wait_intr+0xf0/0xf0 [ 7392.074675] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7392.077316] mdt(O) [ 7392.078250] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7392.078577] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7392.078867] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7392.079068] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7392.079114] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7392.079147] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7392.079178] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7392.079268] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7392.079520] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7392.079553] lmv_intent_open+0x338/0x1150 [lmv] [ 7392.079589] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7392.079685] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7392.079854] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7392.079876] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7392.080031] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7392.080116] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7392.080197] ? do_raw_spin_unlock+0x75/0x190 [ 7392.080206] ? _raw_spin_unlock+0x12/0x30 [ 7392.080213] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7392.080242] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7392.080263] ll_atomic_open+0x36a/0xd70 [lustre] [ 7392.080348] lookup_open+0x356/0xb40 [ 7392.080357] do_last+0x313/0xfc0 [ 7392.080362] ? nd_jump_root+0xe5/0x160 [ 7392.080366] ? path_init+0x437/0x520 [ 7392.080371] path_openat+0xf7/0x500 [ 7392.080375] ? do_raw_spin_unlock+0x75/0x190 [ 7392.080381] do_filp_open+0x99/0x140 [ 7392.080387] ? getname_flags+0x6e/0x330 [ 7392.080391] ? __check_object_size+0xff/0x256 [ 7392.080399] ? do_raw_spin_unlock+0x75/0x190 [ 7392.080403] ? _raw_spin_unlock+0x12/0x30 [ 7392.080408] do_sys_openat2+0x2b4/0x410 [ 7392.080412] do_sys_open+0x73/0xa0 [ 7392.080416] __x64_sys_openat+0x24/0x30 [ 7392.080420] do_syscall_64+0xc1/0x440 [ 7392.080427] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7392.080433] RIP: 0033:0x7ff54af45332 [ 7392.080441] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7392.080445] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7392.080450] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 7392.080453] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 7392.080455] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 7392.080457] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 7392.080459] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 7392.719966] mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7392.750207] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7392.762569] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7392.771331] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7392.776263] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7392.785253] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7392.789021] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 7392.795330] RDX: 00000000000078ab RSI: 0000000000000003 RDI: ffff96848811cae0 [ 7392.805483] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 7392.811399] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 7392.813634] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7392.819212] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 7392.834050] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7392.838750] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 7392.842342] Call Trace: [ 7392.846368] [ 7392.847365] ? show_regs.cold.9+0x22/0x2f [ 7392.852827] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7392.854547] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7392.863862] ? watchdog+0x40/0x40 [ 7392.867388] ? hrtimer_interrupt+0x128/0x2c0 [ 7392.872452] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7392.876295] ? apic_timer_interrupt+0xf/0x20 [ 7392.878504] [ 7392.879572] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7392.883180] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7392.889119] do_raw_spin_lock+0x177/0x190 [ 7392.891424] _raw_spin_lock+0x21/0x30 [ 7392.898431] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7392.911746] obd_mod_free+0x46/0x140 [ptlrpc] [ 7392.920188] mdc_commit_open+0x91/0xb0 [mdc] [ 7392.926912] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7392.932269] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7392.942090] after_reply+0xe2c/0x1780 [ptlrpc] [ 7392.947186] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7392.956393] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7392.967565] ? do_wait_intr+0xf0/0xf0 [ 7392.971866] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7392.973870] kthread+0x1d1/0x200 [ 7392.984417] ? set_kthread_struct+0x70/0x70 [ 7392.991529] ret_from_fork+0x1f/0x30 [ 7398.367191] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5439s! [ 7398.370910] Showing busy workqueues and worker pools: [ 7398.373015] workqueue events: flags=0x0 [ 7398.374527] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7398.378436] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 7398.388558] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7398.395025] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 7398.403439] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7398.405935] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 7398.407919] workqueue events_long: flags=0x0 [ 7398.409080] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7398.414896] pending: lu_objects_flush [obdclass] [ 7398.417457] workqueue events_unbound: flags=0x2 [ 7398.419645] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7398.422345] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7398.426338] workqueue events_power_efficient: flags=0x80 [ 7398.428351] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7398.431082] pending: neigh_periodic_work [ 7398.432615] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7398.435900] pending: do_cache_clean, check_lifetime [ 7398.438400] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7398.441686] pending: neigh_periodic_work [ 7398.444570] workqueue writeback: flags=0x4a [ 7398.446072] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7398.450347] pending: wb_workfn, wb_workfn [ 7398.452323] workqueue xprtiod: flags=0xa [ 7398.454855] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7398.457266] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 7398.461176] workqueue ipv6_addrconf: flags=0x40008 [ 7398.465323] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7398.476919] pending: addrconf_verify_work [ 7398.482701] workqueue knbd0-recv: flags=0x1a [ 7398.484196] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7398.492370] in-flight: 657:recv_work [ 7398.500308] workqueue ptlrpc_pinger: flags=0x2 [ 7398.503256] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7398.516935] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 7398.527843] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5439s workers=4 idle: 20831 12642 28859 [ 7398.533959] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7388s workers=2 idle: 659 [ 7400.058861] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 7400.063821] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7400.107144] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7400.118445] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7400.129662] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7400.144064] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7400.178674] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7400.187907] RAX: 0000000000007c47 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 7400.194317] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 7400.198239] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7400.203235] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 7400.210116] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 7400.212670] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 7400.216272] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7400.218341] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 7400.221621] Call Trace: [ 7400.223251] [ 7400.224083] ? show_regs.cold.9+0x22/0x2f [ 7400.225438] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7400.227719] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7400.229346] ? watchdog+0x40/0x40 [ 7400.230776] ? hrtimer_interrupt+0x128/0x2c0 [ 7400.232254] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7400.235356] ? apic_timer_interrupt+0xf/0x20 [ 7400.237115] [ 7400.237889] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7400.240112] do_raw_spin_lock+0x177/0x190 [ 7400.241694] _raw_spin_lock+0x21/0x30 [ 7400.242984] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7400.244957] ? do_raw_spin_unlock+0x75/0x190 [ 7400.247437] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7400.249638] ? do_wait_intr+0xf0/0xf0 [ 7400.251052] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7400.253057] mdc_close+0x53b/0xd70 [mdc] [ 7400.254715] lmv_close+0x18d/0x490 [lmv] [ 7400.256126] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 7400.258559] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7400.261229] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7400.264410] ll_md_close+0x274/0x800 [lustre] [ 7400.270878] ll_file_release+0x128/0xc50 [lustre] [ 7400.274341] __fput+0x10d/0x440 [ 7400.276168] ____fput+0x12/0x20 [ 7400.278322] task_work_run+0xc6/0x110 [ 7400.279831] exit_to_usermode_loop+0x1dd/0x1f0 [ 7400.282886] do_syscall_64+0x3d6/0x440 [ 7400.287754] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7400.290466] RIP: 0033:0x7ff54af45cd8 [ 7400.294672] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7400.313282] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7400.318769] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 7400.324445] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 7400.330294] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 7400.336097] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 7400.341009] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7414.002033] rcu: INFO: rcu_sched self-detected stall on CPU [ 7414.006194] rcu: 0-...!: (2836790 ticks this GP) idle=98a/1/0x4000000000000002 softirq=125154/125154 fqs=959 [ 7414.014246] (t=5460162 jiffies g=253585 q=4576) [ 7414.017161] rcu: rcu_sched kthread starved for 5455122 jiffies! g253585 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [ 7414.024606] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 7414.031045] rcu: RCU grace-period kthread stack dump: [ 7414.034913] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 7414.041060] Call Trace: [ 7414.044018] __schedule+0x351/0xcb0 [ 7414.046911] schedule+0xc0/0x180 [ 7414.053575] schedule_timeout+0xb4/0x190 [ 7414.057897] ? do_raw_spin_unlock+0x75/0x190 [ 7414.062923] ? __next_timer_interrupt+0x160/0x160 [ 7414.068263] ? prepare_to_swait_event+0xf3/0x140 [ 7414.072455] rcu_gp_kthread+0x8dd/0x15b0 [ 7414.073939] ? force_qs_rnp+0x2c0/0x2c0 [ 7414.079689] kthread+0x1d1/0x200 [ 7414.082033] ? set_kthread_struct+0x70/0x70 [ 7414.086274] ret_from_fork+0x1f/0x30 [ 7414.091063] rcu: Stack dump where RCU GP kthread last ran: [ 7414.096892] Sending NMI from CPU 0 to CPUs 1: [ 7414.098563] NMI backtrace for cpu 1 skipped: idling at native_safe_halt+0x2e/0x50 [ 7414.109981] NMI backtrace for cpu 0 [ 7414.115809] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7414.130855] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7414.143192] Call Trace: [ 7414.144814] [ 7414.146306] ? dump_stack+0xbb/0x10e [ 7414.148775] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 7414.159079] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 7414.166601] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 7414.170516] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 7414.174151] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 7414.177945] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 7414.180648] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 7414.183167] ? update_process_times+0x6d/0xc0 [ 7414.185813] ? tick_sched_handle.isra.18+0x27/0x90 [ 7414.188533] ? tick_sched_timer+0x53/0xb0 [ 7414.190657] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7414.193100] ? tick_sched_handle.isra.18+0x90/0x90 [ 7414.195769] ? hrtimer_interrupt+0x128/0x2c0 [ 7414.198532] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7414.207591] ? apic_timer_interrupt+0xf/0x20 [ 7414.209248] [ 7414.210053] ? native_safe_halt+0x2e/0x50 [ 7414.217552] ? kvm_wait+0xcd/0xe0 [ 7414.223631] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7414.232672] ? do_raw_spin_lock+0x177/0x190 [ 7414.238794] ? _raw_spin_lock+0x21/0x30 [ 7414.244498] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7414.253489] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 7414.260209] ? mdc_commit_open+0x91/0xb0 [mdc] [ 7414.267142] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7414.272085] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7414.275615] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 7414.279746] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7414.283352] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7414.285648] ? do_wait_intr+0xf0/0xf0 [ 7414.290653] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7414.295583] ? kthread+0x1d1/0x200 [ 7414.301325] ? set_kthread_struct+0x70/0x70 [ 7414.304115] ? ret_from_fork+0x1f/0x30 [ 7416.059060] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 7416.063459] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7416.131699] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7416.139976] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7416.150675] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7416.157280] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7416.160553] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7416.171711] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7416.181543] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 7416.188559] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 7416.190313] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 7416.196518] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 7416.200075] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 7416.202583] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 7416.206629] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7416.210090] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 7416.213578] Call Trace: [ 7416.224485] [ 7416.225189] ? show_regs.cold.9+0x22/0x2f [ 7416.227505] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7416.236479] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7416.238471] ? watchdog+0x40/0x40 [ 7416.239767] ? hrtimer_interrupt+0x128/0x2c0 [ 7416.241218] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7416.245840] ? apic_timer_interrupt+0xf/0x20 [ 7416.247937] [ 7416.249206] ? native_safe_halt+0x2e/0x50 [ 7416.254921] kvm_wait+0xcd/0xe0 [ 7416.257972] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7416.263908] do_raw_spin_lock+0x177/0x190 [ 7416.265123] _raw_spin_lock+0x21/0x30 [ 7416.268181] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7416.273208] process_one_work+0x2c8/0x700 [ 7416.280173] worker_thread+0x296/0x6e0 [ 7416.282785] ? rescuer_thread+0x570/0x570 [ 7416.290411] kthread+0x1d1/0x200 [ 7416.294783] ? set_kthread_struct+0x70/0x70 [ 7416.297509] ret_from_fork+0x1f/0x30 [ 7420.060134] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 7420.067203] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7420.111097] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7420.122466] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7420.137016] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7420.142053] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7420.159735] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7420.165163] RAX: 0000000000007e89 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 7420.168866] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 7420.171310] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7420.177823] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 7420.188219] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 7420.193147] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 7420.203222] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7420.208601] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 7420.217515] Call Trace: [ 7420.218288] [ 7420.219501] ? show_regs.cold.9+0x22/0x2f [ 7420.222178] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7420.228548] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7420.231036] ? watchdog+0x40/0x40 [ 7420.232146] ? hrtimer_interrupt+0x128/0x2c0 [ 7420.235633] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7420.238855] ? apic_timer_interrupt+0xf/0x20 [ 7420.240844] [ 7420.241656] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7420.248077] do_raw_spin_lock+0x177/0x190 [ 7420.249852] _raw_spin_lock+0x21/0x30 [ 7420.252273] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7420.262126] ? do_raw_spin_unlock+0x75/0x190 [ 7420.266762] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7420.274604] ? do_wait_intr+0xf0/0xf0 [ 7420.278597] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7420.283149] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7420.286338] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7420.293670] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7420.296804] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7420.299796] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7420.302669] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7420.308248] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7420.312252] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7420.316421] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7420.324895] lmv_intent_open+0x338/0x1150 [lmv] [ 7420.329327] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7420.332460] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7420.335739] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7420.337802] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7420.341515] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7420.345465] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7420.348081] ? do_raw_spin_unlock+0x75/0x190 [ 7420.350652] ? _raw_spin_unlock+0x12/0x30 [ 7420.355997] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7420.359161] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7420.366024] ll_atomic_open+0x36a/0xd70 [lustre] [ 7420.370060] lookup_open+0x356/0xb40 [ 7420.371276] do_last+0x313/0xfc0 [ 7420.377232] ? nd_jump_root+0xe5/0x160 [ 7420.378562] ? path_init+0x437/0x520 [ 7420.380644] path_openat+0xf7/0x500 [ 7420.382046] ? do_raw_spin_unlock+0x75/0x190 [ 7420.385509] do_filp_open+0x99/0x140 [ 7420.389419] ? getname_flags+0x6e/0x330 [ 7420.391722] ? __check_object_size+0xff/0x256 [ 7420.395940] ? do_raw_spin_unlock+0x75/0x190 [ 7420.402380] ? _raw_spin_unlock+0x12/0x30 [ 7420.412237] do_sys_openat2+0x2b4/0x410 [ 7420.415298] do_sys_open+0x73/0xa0 [ 7420.418098] __x64_sys_openat+0x24/0x30 [ 7420.420504] do_syscall_64+0xc1/0x440 [ 7420.422236] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7420.426276] RIP: 0033:0x7ff54af45332 [ 7420.432357] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7420.449885] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7420.456085] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 7420.470213] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 7420.481995] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 7420.484408] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 7420.500868] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 7428.578173] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5469s! [ 7428.583090] Showing busy workqueues and worker pools: [ 7428.592520] workqueue events: flags=0x0 [ 7428.598425] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7428.606312] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 7428.619346] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7428.625304] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 7428.639173] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7428.647149] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 7428.654036] workqueue events_long: flags=0x0 [ 7428.657537] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7428.664817] pending: lu_objects_flush [obdclass] [ 7428.670762] workqueue events_unbound: flags=0x2 [ 7428.675557] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7428.680483] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7428.690805] workqueue events_power_efficient: flags=0x80 [ 7428.699425] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7428.703999] pending: neigh_periodic_work [ 7428.706902] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7428.714086] pending: do_cache_clean, check_lifetime [ 7428.719617] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7428.727810] pending: neigh_periodic_work [ 7428.732925] workqueue writeback: flags=0x4a [ 7428.735437] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7428.741148] pending: wb_workfn, wb_workfn [ 7428.743361] workqueue xprtiod: flags=0xa [ 7428.744263] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7428.746309] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 7428.753712] workqueue ipv6_addrconf: flags=0x40008 [ 7428.760523] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7428.771315] pending: addrconf_verify_work [ 7428.774405] workqueue knbd0-recv: flags=0x1a [ 7428.776674] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7428.780584] in-flight: 657:recv_work [ 7428.782116] workqueue ptlrpc_pinger: flags=0x2 [ 7428.786389] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7428.792340] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 7428.798459] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5469s workers=4 idle: 20831 12642 28859 [ 7428.803720] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7419s workers=2 idle: 659 [ 7440.058018] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [ 7440.068994] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7440.105967] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7440.116985] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7440.127393] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7440.134414] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7440.154766] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7440.167290] RAX: 0000000000003126 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 7440.174476] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 7440.179385] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7440.184348] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 7440.191428] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 7440.205683] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 7440.216193] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7440.227367] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 7440.240025] Call Trace: [ 7440.240934] [ 7440.241661] ? show_regs.cold.9+0x22/0x2f [ 7440.253580] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7440.259640] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7440.261292] ? watchdog+0x40/0x40 [ 7440.263637] ? hrtimer_interrupt+0x128/0x2c0 [ 7440.275637] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7440.287925] ? apic_timer_interrupt+0xf/0x20 [ 7440.289055] [ 7440.289681] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7440.308788] do_raw_spin_lock+0x177/0x190 [ 7440.310164] _raw_spin_lock+0x21/0x30 [ 7440.320020] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7440.326620] ? do_raw_spin_unlock+0x75/0x190 [ 7440.331956] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7440.338598] ? do_wait_intr+0xf0/0xf0 [ 7440.341784] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7440.349404] mdc_close+0x53b/0xd70 [mdc] [ 7440.354484] lmv_close+0x18d/0x490 [lmv] [ 7440.359779] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 7440.363448] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7440.366436] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7440.368921] ll_md_close+0x274/0x800 [lustre] [ 7440.373783] ll_file_release+0x128/0xc50 [lustre] [ 7440.377229] __fput+0x10d/0x440 [ 7440.381316] ____fput+0x12/0x20 [ 7440.385168] task_work_run+0xc6/0x110 [ 7440.387376] exit_to_usermode_loop+0x1dd/0x1f0 [ 7440.389561] do_syscall_64+0x3d6/0x440 [ 7440.392683] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7440.396652] RIP: 0033:0x7ff54af45cd8 [ 7440.399954] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7440.409176] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7440.411521] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 7440.418645] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 7440.421969] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 7440.425367] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 7440.428925] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7444.063590] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 7444.071963] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7444.109170] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7444.126299] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7444.138265] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7444.142608] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7444.148575] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7444.173256] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7444.183420] RAX: 0000000000003788 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 7444.194559] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 7444.202343] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 7444.205419] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 7444.211391] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 7444.220711] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 7444.232894] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7444.239291] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 7444.250746] Call Trace: [ 7444.254000] [ 7444.255393] ? show_regs.cold.9+0x22/0x2f [ 7444.258950] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7444.264148] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7444.265782] ? watchdog+0x40/0x40 [ 7444.267484] ? hrtimer_interrupt+0x128/0x2c0 [ 7444.269015] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7444.274363] ? apic_timer_interrupt+0xf/0x20 [ 7444.277394] [ 7444.278488] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7444.285802] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7444.289371] do_raw_spin_lock+0x177/0x190 [ 7444.291601] _raw_spin_lock+0x21/0x30 [ 7444.293650] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7444.297987] process_one_work+0x2c8/0x700 [ 7444.300856] worker_thread+0x296/0x6e0 [ 7444.302569] ? rescuer_thread+0x570/0x570 [ 7444.305650] kthread+0x1d1/0x200 [ 7444.307515] ? set_kthread_struct+0x70/0x70 [ 7444.310883] ret_from_fork+0x1f/0x30 [ 7448.060254] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 7448.064048] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7448.115951] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7448.127583] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7448.132856] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7448.136117] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7448.151563] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7448.165446] RAX: 0000000000007d13 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 7448.170805] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 7448.177978] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7448.187633] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 7448.195188] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 7448.199851] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 7448.204963] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7448.209393] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 7448.213996] Call Trace: [ 7448.215228] [ 7448.215840] ? show_regs.cold.9+0x22/0x2f [ 7448.218342] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7448.223092] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7448.225738] ? watchdog+0x40/0x40 [ 7448.227212] ? hrtimer_interrupt+0x128/0x2c0 [ 7448.229442] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7448.231784] ? apic_timer_interrupt+0xf/0x20 [ 7448.234333] [ 7448.235076] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7448.240559] do_raw_spin_lock+0x177/0x190 [ 7448.241964] _raw_spin_lock+0x21/0x30 [ 7448.244802] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7448.247373] ? do_raw_spin_unlock+0x75/0x190 [ 7448.249277] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7448.251620] ? do_wait_intr+0xf0/0xf0 [ 7448.253242] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7448.255558] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7448.259409] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7448.261932] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7448.263740] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7448.266107] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7448.267773] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7448.269602] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7448.271730] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7448.274157] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7448.276395] lmv_intent_open+0x338/0x1150 [lmv] [ 7448.278455] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7448.280698] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7448.283244] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7448.285165] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7448.287159] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7448.289395] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7448.291358] ? do_raw_spin_unlock+0x75/0x190 [ 7448.292667] ? _raw_spin_unlock+0x12/0x30 [ 7448.294394] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7448.296425] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7448.298265] ll_atomic_open+0x36a/0xd70 [lustre] [ 7448.302350] lookup_open+0x356/0xb40 [ 7448.303458] do_last+0x313/0xfc0 [ 7448.309512] ? nd_jump_root+0xe5/0x160 [ 7448.310927] ? path_init+0x437/0x520 [ 7448.312145] path_openat+0xf7/0x500 [ 7448.317959] ? do_raw_spin_unlock+0x75/0x190 [ 7448.321053] do_filp_open+0x99/0x140 [ 7448.322785] ? getname_flags+0x6e/0x330 [ 7448.324512] ? __check_object_size+0xff/0x256 [ 7448.328704] ? do_raw_spin_unlock+0x75/0x190 [ 7448.330025] ? _raw_spin_unlock+0x12/0x30 [ 7448.335553] do_sys_openat2+0x2b4/0x410 [ 7448.340254] do_sys_open+0x73/0xa0 [ 7448.343787] __x64_sys_openat+0x24/0x30 [ 7448.346841] do_syscall_64+0xc1/0x440 [ 7448.350643] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7448.363806] RIP: 0033:0x7ff54af45332 [ 7448.365597] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7448.387982] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7448.399007] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 7448.403391] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 7448.410277] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 7448.413464] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 7448.417790] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 7456.055043] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 7456.058438] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7456.077206] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7456.082535] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7456.086026] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7456.088396] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7456.097687] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7456.101895] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 7456.105156] RDX: 0000000000005f01 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 7456.108452] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 7456.111966] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 7456.115916] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7456.119328] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 7456.122043] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7456.124752] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 7456.128397] Call Trace: [ 7456.129437] [ 7456.130430] ? show_regs.cold.9+0x22/0x2f [ 7456.132272] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7456.134685] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7456.136742] ? watchdog+0x40/0x40 [ 7456.137918] ? hrtimer_interrupt+0x128/0x2c0 [ 7456.139870] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7456.141371] ? apic_timer_interrupt+0xf/0x20 [ 7456.143481] [ 7456.145338] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7456.147538] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7456.149095] do_raw_spin_lock+0x177/0x190 [ 7456.150854] _raw_spin_lock+0x21/0x30 [ 7456.154890] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7456.159591] obd_mod_free+0x46/0x140 [ptlrpc] [ 7456.164631] mdc_commit_open+0x91/0xb0 [mdc] [ 7456.166397] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7456.168483] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7456.170541] after_reply+0xe2c/0x1780 [ptlrpc] [ 7456.177095] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7456.178949] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7456.188525] ? do_wait_intr+0xf0/0xf0 [ 7456.191763] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7456.194920] kthread+0x1d1/0x200 [ 7456.199433] ? set_kthread_struct+0x70/0x70 [ 7456.201608] ret_from_fork+0x1f/0x30 [ 7459.299634] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5500s! [ 7459.310989] Showing busy workqueues and worker pools: [ 7459.319704] workqueue events: flags=0x0 [ 7459.324276] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7459.332573] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 7459.339366] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7459.342364] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 7459.353414] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7459.356610] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 7459.367816] workqueue events_long: flags=0x0 [ 7459.374157] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7459.380676] pending: lu_objects_flush [obdclass] [ 7459.383791] workqueue events_unbound: flags=0x2 [ 7459.387368] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7459.394174] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7459.405540] workqueue events_power_efficient: flags=0x80 [ 7459.406918] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7459.411547] pending: neigh_periodic_work [ 7459.413507] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7459.416515] pending: do_cache_clean, check_lifetime [ 7459.418598] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7459.421612] pending: neigh_periodic_work [ 7459.423655] workqueue writeback: flags=0x4a [ 7459.426776] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7459.440755] pending: wb_workfn, wb_workfn [ 7459.444636] workqueue xprtiod: flags=0xa [ 7459.445948] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7459.466224] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 7459.485813] workqueue ipv6_addrconf: flags=0x40008 [ 7459.489026] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7459.496096] pending: addrconf_verify_work [ 7459.500188] workqueue knbd0-recv: flags=0x1a [ 7459.505348] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7459.511730] in-flight: 657:recv_work [ 7459.513706] workqueue ptlrpc_pinger: flags=0x2 [ 7459.525641] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7459.533760] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 7459.548143] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5500s workers=4 idle: 20831 12642 28859 [ 7459.564638] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7449s workers=2 idle: 659 [ 7468.058957] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [ 7468.062692] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7468.109572] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7468.114762] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7468.119913] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7468.124053] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7468.155927] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7468.162185] RAX: 0000000000007be3 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 7468.165895] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 7468.169153] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7468.174644] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 7468.197396] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 7468.207746] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 7468.210699] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7468.219825] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 7468.229303] Call Trace: [ 7468.230800] [ 7468.234396] ? show_regs.cold.9+0x22/0x2f [ 7468.236446] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7468.239581] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7468.241576] ? watchdog+0x40/0x40 [ 7468.243232] ? hrtimer_interrupt+0x128/0x2c0 [ 7468.248583] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7468.257126] ? apic_timer_interrupt+0xf/0x20 [ 7468.258586] [ 7468.259347] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7468.286236] do_raw_spin_lock+0x177/0x190 [ 7468.290783] _raw_spin_lock+0x21/0x30 [ 7468.294634] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7468.302474] ? do_raw_spin_unlock+0x75/0x190 [ 7468.309801] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7468.321551] ? do_wait_intr+0xf0/0xf0 [ 7468.329270] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7468.330948] mdc_close+0x53b/0xd70 [mdc] [ 7468.342940] lmv_close+0x18d/0x490 [lmv] [ 7468.352950] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 7468.358722] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7468.364046] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7468.374240] ll_md_close+0x274/0x800 [lustre] [ 7468.376366] ll_file_release+0x128/0xc50 [lustre] [ 7468.379682] __fput+0x10d/0x440 [ 7468.381209] ____fput+0x12/0x20 [ 7468.383365] task_work_run+0xc6/0x110 [ 7468.387072] exit_to_usermode_loop+0x1dd/0x1f0 [ 7468.390523] do_syscall_64+0x3d6/0x440 [ 7468.393633] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7468.400521] RIP: 0033:0x7ff54af45cd8 [ 7468.403132] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7468.428686] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7468.437507] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 7468.442433] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 7468.446265] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 7468.452618] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 7468.456692] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7472.062131] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 7472.070410] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7472.096434] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7472.104407] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7472.119931] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7472.127535] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7472.132450] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7472.145498] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 7472.149114] RAX: 0000000000005d8e RBX: ffff96848811cae0 RCX: 0000000000000001 [ 7472.153853] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 7472.157093] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 7472.163801] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 7472.166599] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 7472.175671] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 7472.186586] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7472.193205] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 7472.196493] Call Trace: [ 7472.197455] [ 7472.198362] ? show_regs.cold.9+0x22/0x2f [ 7472.200136] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7472.201752] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7472.204050] ? watchdog+0x40/0x40 [ 7472.205607] ? hrtimer_interrupt+0x128/0x2c0 [ 7472.207728] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7472.217360] ? apic_timer_interrupt+0xf/0x20 [ 7472.220405] [ 7472.221648] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7472.224160] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7472.226655] do_raw_spin_lock+0x177/0x190 [ 7472.228285] _raw_spin_lock+0x21/0x30 [ 7472.229367] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7472.232223] process_one_work+0x2c8/0x700 [ 7472.233675] worker_thread+0x296/0x6e0 [ 7472.235612] ? rescuer_thread+0x570/0x570 [ 7472.237617] kthread+0x1d1/0x200 [ 7472.239217] ? set_kthread_struct+0x70/0x70 [ 7472.241646] ret_from_fork+0x1f/0x30 [ 7476.062211] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 7476.068313] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7476.135471] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7476.145284] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7476.150306] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7476.152946] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7476.161300] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7476.163876] RAX: 0000000000007b0c RBX: ffff96848811cae0 RCX: 0000000000000001 [ 7476.166715] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 7476.169663] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7476.172476] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 7476.175369] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 7476.178387] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 7476.181565] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7476.183679] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 7476.187065] Call Trace: [ 7476.188066] [ 7476.188870] ? show_regs.cold.9+0x22/0x2f [ 7476.190397] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7476.192265] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7476.194060] ? watchdog+0x40/0x40 [ 7476.195252] ? hrtimer_interrupt+0x128/0x2c0 [ 7476.196980] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7476.198900] ? apic_timer_interrupt+0xf/0x20 [ 7476.200458] [ 7476.201306] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7476.203232] do_raw_spin_lock+0x177/0x190 [ 7476.204614] _raw_spin_lock+0x21/0x30 [ 7476.205946] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7476.209433] ? do_raw_spin_unlock+0x75/0x190 [ 7476.210930] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7476.213075] ? do_wait_intr+0xf0/0xf0 [ 7476.214539] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7476.217004] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7476.219104] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7476.221158] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7476.223206] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7476.225251] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7476.227156] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7476.228682] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7476.230096] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7476.232046] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7476.234245] lmv_intent_open+0x338/0x1150 [lmv] [ 7476.236501] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7476.238177] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7476.240203] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7476.242056] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7476.244022] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7476.246201] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7476.248184] ? do_raw_spin_unlock+0x75/0x190 [ 7476.250005] ? _raw_spin_unlock+0x12/0x30 [ 7476.251560] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7476.253656] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7476.255644] ll_atomic_open+0x36a/0xd70 [lustre] [ 7476.257378] lookup_open+0x356/0xb40 [ 7476.258725] do_last+0x313/0xfc0 [ 7476.260115] ? nd_jump_root+0xe5/0x160 [ 7476.261549] ? path_init+0x437/0x520 [ 7476.263114] path_openat+0xf7/0x500 [ 7476.264305] ? do_raw_spin_unlock+0x75/0x190 [ 7476.266090] do_filp_open+0x99/0x140 [ 7476.267625] ? getname_flags+0x6e/0x330 [ 7476.269137] ? __check_object_size+0xff/0x256 [ 7476.270867] ? do_raw_spin_unlock+0x75/0x190 [ 7476.272352] ? _raw_spin_unlock+0x12/0x30 [ 7476.273964] do_sys_openat2+0x2b4/0x410 [ 7476.275425] do_sys_open+0x73/0xa0 [ 7476.276878] __x64_sys_openat+0x24/0x30 [ 7476.278347] do_syscall_64+0xc1/0x440 [ 7476.279541] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7476.281378] RIP: 0033:0x7ff54af45332 [ 7476.282959] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7476.290304] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7476.293134] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 7476.295616] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 7476.298424] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 7476.301231] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 7476.303914] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 7484.055729] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [ 7484.075637] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7484.105974] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7484.110038] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7484.113682] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7484.116098] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7484.121773] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7484.123956] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 7484.126780] RDX: 00000000000058e4 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 7484.129519] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 7484.132067] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 7484.162691] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7484.170616] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 7484.177942] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7484.187726] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 7484.194655] Call Trace: [ 7484.196886] [ 7484.198222] ? show_regs.cold.9+0x22/0x2f [ 7484.201926] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7484.204750] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7484.207127] ? watchdog+0x40/0x40 [ 7484.208347] ? hrtimer_interrupt+0x128/0x2c0 [ 7484.217922] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7484.224706] ? apic_timer_interrupt+0xf/0x20 [ 7484.227483] [ 7484.228415] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7484.235398] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7484.238531] do_raw_spin_lock+0x177/0x190 [ 7484.240622] _raw_spin_lock+0x21/0x30 [ 7484.245127] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7484.253715] obd_mod_free+0x46/0x140 [ptlrpc] [ 7484.258811] mdc_commit_open+0x91/0xb0 [mdc] [ 7484.261004] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7484.266756] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7484.270385] after_reply+0xe2c/0x1780 [ptlrpc] [ 7484.273357] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7484.276670] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7484.278254] ? do_wait_intr+0xf0/0xf0 [ 7484.279320] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7484.282226] kthread+0x1d1/0x200 [ 7484.283810] ? set_kthread_struct+0x70/0x70 [ 7484.285907] ret_from_fork+0x1f/0x30 [ 7490.023038] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5531s! [ 7490.041920] Showing busy workqueues and worker pools: [ 7490.048353] workqueue events: flags=0x0 [ 7490.050951] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7490.056877] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 7490.067706] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7490.077729] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 7490.091505] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7490.095516] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 7490.100390] workqueue events_long: flags=0x0 [ 7490.105071] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7490.112997] pending: lu_objects_flush [obdclass] [ 7490.116327] workqueue events_unbound: flags=0x2 [ 7490.119169] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7490.128665] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7490.137488] workqueue events_power_efficient: flags=0x80 [ 7490.138717] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7490.143757] pending: neigh_periodic_work [ 7490.146682] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7490.152737] pending: do_cache_clean, check_lifetime [ 7490.155788] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7490.176035] pending: neigh_periodic_work [ 7490.197321] workqueue writeback: flags=0x4a [ 7490.209734] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7490.217165] pending: wb_workfn, wb_workfn [ 7490.223505] workqueue xprtiod: flags=0xa [ 7490.225420] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7490.228192] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 7490.231856] workqueue ipv6_addrconf: flags=0x40008 [ 7490.233901] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7490.236698] pending: addrconf_verify_work [ 7490.238619] workqueue knbd0-recv: flags=0x1a [ 7490.240567] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7490.243224] in-flight: 657:recv_work [ 7490.244941] workqueue ptlrpc_pinger: flags=0x2 [ 7490.246766] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7490.249190] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 7490.251775] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5531s workers=4 idle: 20831 12642 28859 [ 7490.255038] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7480s workers=2 idle: 659 [ 7496.065769] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [ 7496.073201] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7496.151026] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7496.158434] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7496.168295] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7496.173310] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7496.199527] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7496.210830] RAX: 000000000000795e RBX: ffff96848811cae0 RCX: 0000000000000001 [ 7496.214021] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 7496.223201] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7496.232336] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 7496.243319] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 7496.250455] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 7496.257027] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7496.258609] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 7496.266189] Call Trace: [ 7496.269641] [ 7496.278658] ? show_regs.cold.9+0x22/0x2f [ 7496.280138] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7496.289495] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7496.298409] ? watchdog+0x40/0x40 [ 7496.303395] ? hrtimer_interrupt+0x128/0x2c0 [ 7496.305321] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7496.318270] ? apic_timer_interrupt+0xf/0x20 [ 7496.327884] [ 7496.328433] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7496.340051] do_raw_spin_lock+0x177/0x190 [ 7496.341546] _raw_spin_lock+0x21/0x30 [ 7496.358749] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7496.361412] ? do_raw_spin_unlock+0x75/0x190 [ 7496.368241] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7496.376217] ? do_wait_intr+0xf0/0xf0 [ 7496.381918] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7496.396517] mdc_close+0x53b/0xd70 [mdc] [ 7496.404679] lmv_close+0x18d/0x490 [lmv] [ 7496.407496] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 7496.412165] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7496.417180] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7496.423008] ll_md_close+0x274/0x800 [lustre] [ 7496.424704] ll_file_release+0x128/0xc50 [lustre] [ 7496.427871] __fput+0x10d/0x440 [ 7496.429060] ____fput+0x12/0x20 [ 7496.434909] task_work_run+0xc6/0x110 [ 7496.435931] exit_to_usermode_loop+0x1dd/0x1f0 [ 7496.439247] do_syscall_64+0x3d6/0x440 [ 7496.440930] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7496.442362] RIP: 0033:0x7ff54af45cd8 [ 7496.449235] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7496.465127] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7496.469315] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 7496.472443] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 7496.482491] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 7496.492198] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 7496.498202] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7500.067667] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 7500.080270] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7500.163249] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7500.168050] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7500.173832] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7500.179681] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7500.183095] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7500.223072] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7500.224981] RAX: 0000000000007b85 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 7500.240445] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 7500.246303] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 7500.266171] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 7500.281027] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 7500.290853] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 7500.312163] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7500.323602] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 7500.333366] Call Trace: [ 7500.342705] [ 7500.352562] ? show_regs.cold.9+0x22/0x2f [ 7500.356189] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7500.367097] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7500.369330] ? watchdog+0x40/0x40 [ 7500.374868] ? hrtimer_interrupt+0x128/0x2c0 [ 7500.377503] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7500.385230] ? apic_timer_interrupt+0xf/0x20 [ 7500.392295] [ 7500.395760] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7500.402292] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7500.408407] do_raw_spin_lock+0x177/0x190 [ 7500.411454] _raw_spin_lock+0x21/0x30 [ 7500.416303] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7500.422476] process_one_work+0x2c8/0x700 [ 7500.426796] worker_thread+0x296/0x6e0 [ 7500.429865] ? rescuer_thread+0x570/0x570 [ 7500.433941] kthread+0x1d1/0x200 [ 7500.435853] ? set_kthread_struct+0x70/0x70 [ 7500.438338] ret_from_fork+0x1f/0x30 [ 7504.062052] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 7504.070003] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7504.101655] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7504.114646] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7504.118418] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7504.122878] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7504.137009] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 7504.143997] RAX: 0000000000003963 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 7504.150356] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 7504.157045] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7504.163969] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 7504.174693] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 7504.177073] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 7504.191881] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7504.197483] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 7504.204422] Call Trace: [ 7504.208039] [ 7504.209787] ? show_regs.cold.9+0x22/0x2f [ 7504.212931] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7504.217178] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7504.219848] ? watchdog+0x40/0x40 [ 7504.221544] ? hrtimer_interrupt+0x128/0x2c0 [ 7504.223586] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7504.225655] ? apic_timer_interrupt+0xf/0x20 [ 7504.228123] [ 7504.229151] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7504.233606] do_raw_spin_lock+0x177/0x190 [ 7504.235004] _raw_spin_lock+0x21/0x30 [ 7504.236272] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7504.245983] ? do_raw_spin_unlock+0x75/0x190 [ 7504.249602] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7504.251448] ? do_wait_intr+0xf0/0xf0 [ 7504.255997] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7504.259939] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7504.264825] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7504.268517] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7504.271279] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7504.273463] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7504.277003] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7504.279368] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7504.281912] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7504.284290] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7504.286614] lmv_intent_open+0x338/0x1150 [lmv] [ 7504.288449] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7504.291303] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7504.293712] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7504.295615] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7504.297603] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7504.300396] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7504.305713] ? do_raw_spin_unlock+0x75/0x190 [ 7504.307869] ? _raw_spin_unlock+0x12/0x30 [ 7504.311854] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7504.316157] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7504.320916] ll_atomic_open+0x36a/0xd70 [lustre] [ 7504.325550] lookup_open+0x356/0xb40 [ 7504.331616] do_last+0x313/0xfc0 [ 7504.335099] ? nd_jump_root+0xe5/0x160 [ 7504.337920] ? path_init+0x437/0x520 [ 7504.339812] path_openat+0xf7/0x500 [ 7504.341884] ? do_raw_spin_unlock+0x75/0x190 [ 7504.347203] do_filp_open+0x99/0x140 [ 7504.353596] ? getname_flags+0x6e/0x330 [ 7504.357439] ? __check_object_size+0xff/0x256 [ 7504.361612] ? do_raw_spin_unlock+0x75/0x190 [ 7504.364216] ? _raw_spin_unlock+0x12/0x30 [ 7504.366445] do_sys_openat2+0x2b4/0x410 [ 7504.368767] do_sys_open+0x73/0xa0 [ 7504.370771] __x64_sys_openat+0x24/0x30 [ 7504.372901] do_syscall_64+0xc1/0x440 [ 7504.374616] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7504.377389] RIP: 0033:0x7ff54af45332 [ 7504.382795] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7504.398858] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7504.403731] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 7504.407362] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 7504.413303] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 7504.416370] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 7504.420388] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 7516.055025] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 7516.058549] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7516.087736] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7516.096250] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7516.106772] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7516.109852] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7516.123096] RSP: 0000:ffffaf9fc1b97bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7516.130682] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 7516.134816] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 7516.141983] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 7516.148589] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 7516.153316] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7516.158094] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 7516.170135] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7516.178990] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 7516.187637] Call Trace: [ 7516.192903] [ 7516.193933] ? show_regs.cold.9+0x22/0x2f [ 7516.202002] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7516.204689] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7516.208601] ? watchdog+0x40/0x40 [ 7516.210504] ? hrtimer_interrupt+0x128/0x2c0 [ 7516.215302] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7516.218792] ? apic_timer_interrupt+0xf/0x20 [ 7516.221283] [ 7516.222748] ? native_safe_halt+0x2e/0x50 [ 7516.224113] kvm_wait+0xcd/0xe0 [ 7516.225226] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7516.228290] do_raw_spin_lock+0x177/0x190 [ 7516.230325] _raw_spin_lock+0x21/0x30 [ 7516.235077] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7516.239125] obd_mod_free+0x46/0x140 [ptlrpc] [ 7516.241784] mdc_commit_open+0x91/0xb0 [mdc] [ 7516.245511] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7516.250974] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7516.257033] after_reply+0xe2c/0x1780 [ptlrpc] [ 7516.260017] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7516.262575] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7516.264115] ? do_wait_intr+0xf0/0xf0 [ 7516.265311] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7516.269166] kthread+0x1d1/0x200 [ 7516.270973] ? set_kthread_struct+0x70/0x70 [ 7516.277939] ret_from_fork+0x1f/0x30 [ 7520.735046] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5561s! [ 7520.737793] Showing busy workqueues and worker pools: [ 7520.739459] workqueue events: flags=0x0 [ 7520.740627] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7520.742766] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 7520.745614] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7520.747794] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 7520.753084] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7520.755597] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 7520.758707] workqueue events_long: flags=0x0 [ 7520.760047] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7520.764271] pending: lu_objects_flush [obdclass] [ 7520.772294] workqueue events_unbound: flags=0x2 [ 7520.777602] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7520.783874] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7520.798915] workqueue events_power_efficient: flags=0x80 [ 7520.803467] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7520.815217] pending: neigh_periodic_work [ 7520.818591] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7520.828039] pending: do_cache_clean, check_lifetime [ 7520.840583] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7520.851550] pending: neigh_periodic_work [ 7520.864080] workqueue writeback: flags=0x4a [ 7520.875097] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7520.884659] pending: wb_workfn, wb_workfn [ 7520.890957] workqueue xprtiod: flags=0xa [ 7520.896978] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7520.905330] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 7520.920585] workqueue ipv6_addrconf: flags=0x40008 [ 7520.927684] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7520.933549] pending: addrconf_verify_work [ 7520.936350] workqueue knbd0-recv: flags=0x1a [ 7520.941182] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7520.948196] in-flight: 657:recv_work [ 7520.952595] workqueue ptlrpc_pinger: flags=0x2 [ 7520.957820] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7520.961070] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 7520.968201] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5562s workers=4 idle: 20831 12642 28859 [ 7520.971678] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7511s workers=2 idle: 659 [ 7524.058274] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [ 7524.065557] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7524.110015] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7524.115638] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7524.120674] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7524.123064] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7524.134660] RSP: 0018:ffffaf9fc1543ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7524.138251] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 7524.141841] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 7524.146495] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7524.150696] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 7524.154179] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 7524.157996] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 7524.161985] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7524.165510] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 7524.169430] Call Trace: [ 7524.175300] [ 7524.176279] ? show_regs.cold.9+0x22/0x2f [ 7524.178218] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7524.185104] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7524.186669] ? watchdog+0x40/0x40 [ 7524.191421] ? hrtimer_interrupt+0x128/0x2c0 [ 7524.194655] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7524.200923] ? apic_timer_interrupt+0xf/0x20 [ 7524.204265] [ 7524.207230] ? native_safe_halt+0x2e/0x50 [ 7524.211787] kvm_wait+0xcd/0xe0 [ 7524.215007] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7524.217270] do_raw_spin_lock+0x177/0x190 [ 7524.219925] _raw_spin_lock+0x21/0x30 [ 7524.221288] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7524.225030] ? do_raw_spin_unlock+0x75/0x190 [ 7524.227658] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7524.231105] ? do_wait_intr+0xf0/0xf0 [ 7524.234526] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7524.237510] mdc_close+0x53b/0xd70 [mdc] [ 7524.239784] lmv_close+0x18d/0x490 [lmv] [ 7524.241150] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 7524.244099] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7524.248184] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7524.255587] ll_md_close+0x274/0x800 [lustre] [ 7524.259330] ll_file_release+0x128/0xc50 [lustre] [ 7524.261772] __fput+0x10d/0x440 [ 7524.263144] ____fput+0x12/0x20 [ 7524.264790] task_work_run+0xc6/0x110 [ 7524.266653] exit_to_usermode_loop+0x1dd/0x1f0 [ 7524.268837] do_syscall_64+0x3d6/0x440 [ 7524.272629] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7524.285148] RIP: 0033:0x7ff54af45cd8 [ 7524.288904] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7524.321486] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7524.325593] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 7524.332101] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 7524.334620] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 7524.345765] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 7524.358246] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7528.059882] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 7528.066161] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7528.085167] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7528.088505] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7528.091062] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7528.095260] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7528.098399] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7528.104774] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7528.107346] RAX: 0000000000004449 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 7528.110765] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 7528.112826] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 7528.115780] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 7528.119126] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 7528.121367] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 7528.123327] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7528.125525] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 7528.128476] Call Trace: [ 7528.129397] [ 7528.129807] ? show_regs.cold.9+0x22/0x2f [ 7528.130785] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7528.132496] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7528.134338] ? watchdog+0x40/0x40 [ 7528.135251] ? hrtimer_interrupt+0x128/0x2c0 [ 7528.136148] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7528.137189] ? apic_timer_interrupt+0xf/0x20 [ 7528.138164] [ 7528.138737] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7528.140072] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7528.142058] do_raw_spin_lock+0x177/0x190 [ 7528.143711] _raw_spin_lock+0x21/0x30 [ 7528.145361] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7528.148386] process_one_work+0x2c8/0x700 [ 7528.153362] worker_thread+0x296/0x6e0 [ 7528.157167] ? rescuer_thread+0x570/0x570 [ 7528.166524] kthread+0x1d1/0x200 [ 7528.170801] ? set_kthread_struct+0x70/0x70 [ 7528.177089] ret_from_fork+0x1f/0x30 [ 7532.063352] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 7532.066154] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7532.092158] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7532.134919] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7532.141103] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7532.144946] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7532.150630] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7532.152850] RAX: 0000000000003787 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 7532.157164] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 7532.159585] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7532.163030] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 7532.170454] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 7532.178097] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 7532.182084] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7532.184916] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 7532.188019] Call Trace: [ 7532.189062] [ 7532.191176] ? show_regs.cold.9+0x22/0x2f [ 7532.195447] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7532.199846] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7532.201709] ? watchdog+0x40/0x40 [ 7532.205282] ? hrtimer_interrupt+0x128/0x2c0 [ 7532.207802] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7532.209905] ? apic_timer_interrupt+0xf/0x20 [ 7532.211024] [ 7532.211521] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7532.217137] do_raw_spin_lock+0x177/0x190 [ 7532.218219] _raw_spin_lock+0x21/0x30 [ 7532.219105] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7532.220721] ? do_raw_spin_unlock+0x75/0x190 [ 7532.221841] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7532.223379] ? do_wait_intr+0xf0/0xf0 [ 7532.226264] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7532.230336] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7532.231919] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7532.233436] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7532.234840] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7532.236139] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7532.237371] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7532.238584] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7532.240147] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7532.241756] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7532.243282] lmv_intent_open+0x338/0x1150 [lmv] [ 7532.245562] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7532.248247] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7532.250645] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7532.252069] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7532.254109] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7532.256399] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7532.258417] ? do_raw_spin_unlock+0x75/0x190 [ 7532.260418] ? _raw_spin_unlock+0x12/0x30 [ 7532.262806] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7532.264749] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7532.266945] ll_atomic_open+0x36a/0xd70 [lustre] [ 7532.268747] lookup_open+0x356/0xb40 [ 7532.270382] do_last+0x313/0xfc0 [ 7532.271374] ? nd_jump_root+0xe5/0x160 [ 7532.272281] ? path_init+0x437/0x520 [ 7532.273442] path_openat+0xf7/0x500 [ 7532.274390] ? do_raw_spin_unlock+0x75/0x190 [ 7532.275389] do_filp_open+0x99/0x140 [ 7532.276322] ? getname_flags+0x6e/0x330 [ 7532.277401] ? __check_object_size+0xff/0x256 [ 7532.278508] ? do_raw_spin_unlock+0x75/0x190 [ 7532.279577] ? _raw_spin_unlock+0x12/0x30 [ 7532.281569] do_sys_openat2+0x2b4/0x410 [ 7532.283337] do_sys_open+0x73/0xa0 [ 7532.284857] __x64_sys_openat+0x24/0x30 [ 7532.286627] do_syscall_64+0xc1/0x440 [ 7532.288341] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7532.290307] RIP: 0033:0x7ff54af45332 [ 7532.291566] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7532.298499] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7532.300678] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 7532.302939] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 7532.305697] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 7532.308918] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 7532.311339] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 7548.056461] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 7548.063262] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7548.102930] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7548.112330] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7548.123832] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7548.131830] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7548.156688] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7548.164713] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 7548.176404] RDX: 000000000000560b RSI: 0000000000000003 RDI: ffff96848811cae0 [ 7548.188845] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 7548.194865] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 7548.200829] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7548.214842] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 7548.223010] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7548.229505] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 7548.239821] Call Trace: [ 7548.244051] [ 7548.245715] ? show_regs.cold.9+0x22/0x2f [ 7548.253494] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7548.258509] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7548.262713] ? watchdog+0x40/0x40 [ 7548.264491] ? hrtimer_interrupt+0x128/0x2c0 [ 7548.272105] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7548.275868] ? apic_timer_interrupt+0xf/0x20 [ 7548.284098] [ 7548.284901] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7548.295159] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7548.299602] do_raw_spin_lock+0x177/0x190 [ 7548.301660] _raw_spin_lock+0x21/0x30 [ 7548.306714] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7548.313408] obd_mod_free+0x46/0x140 [ptlrpc] [ 7548.326521] mdc_commit_open+0x91/0xb0 [mdc] [ 7548.333958] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7548.339846] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7548.341952] after_reply+0xe2c/0x1780 [ptlrpc] [ 7548.354534] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7548.360619] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7548.363704] ? do_wait_intr+0xf0/0xf0 [ 7548.367740] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7548.369785] kthread+0x1d1/0x200 [ 7548.373931] ? set_kthread_struct+0x70/0x70 [ 7548.377241] ret_from_fork+0x1f/0x30 [ 7551.455146] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5592s! [ 7551.459202] Showing busy workqueues and worker pools: [ 7551.461467] workqueue events: flags=0x0 [ 7551.463346] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7551.466419] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 7551.471306] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7551.474710] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 7551.480537] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7551.483647] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 7551.487210] workqueue events_long: flags=0x0 [ 7551.489293] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7551.492913] pending: lu_objects_flush [obdclass] [ 7551.495620] workqueue events_unbound: flags=0x2 [ 7551.498386] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7551.502227] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7551.507313] workqueue events_power_efficient: flags=0x80 [ 7551.509495] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7551.512778] pending: neigh_periodic_work [ 7551.514491] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7551.517714] pending: do_cache_clean, check_lifetime [ 7551.520229] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7551.523586] pending: neigh_periodic_work [ 7551.532366] workqueue writeback: flags=0x4a [ 7551.533984] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7551.552373] pending: wb_workfn, wb_workfn [ 7551.558304] workqueue xprtiod: flags=0xa [ 7551.563711] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7551.576962] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 7551.583421] workqueue ipv6_addrconf: flags=0x40008 [ 7551.595038] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7551.599860] pending: addrconf_verify_work [ 7551.607668] workqueue knbd0-recv: flags=0x1a [ 7551.614061] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7551.615747] in-flight: 657:recv_work [ 7551.633763] workqueue ptlrpc_pinger: flags=0x2 [ 7551.635113] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7551.651882] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 7551.661544] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5592s workers=4 idle: 20831 12642 28859 [ 7551.668738] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7541s workers=2 idle: 659 [ 7552.058026] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 7552.060842] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7552.082479] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7552.091870] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7552.095834] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7552.099429] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7552.123527] RSP: 0018:ffffaf9fc1543ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7552.125153] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 7552.127313] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 7552.128942] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7552.130504] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 7552.132080] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 7552.135010] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 7552.137951] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7552.139822] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 7552.142839] Call Trace: [ 7552.143909] [ 7552.144916] ? show_regs.cold.9+0x22/0x2f [ 7552.146351] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7552.149390] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7552.150697] ? watchdog+0x40/0x40 [ 7552.151685] ? hrtimer_interrupt+0x128/0x2c0 [ 7552.152956] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7552.157967] ? apic_timer_interrupt+0xf/0x20 [ 7552.162356] [ 7552.162944] ? native_safe_halt+0x2e/0x50 [ 7552.164865] kvm_wait+0xcd/0xe0 [ 7552.166257] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7552.167865] do_raw_spin_lock+0x177/0x190 [ 7552.169383] _raw_spin_lock+0x21/0x30 [ 7552.170468] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7552.176864] ? do_raw_spin_unlock+0x75/0x190 [ 7552.179911] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7552.182696] ? do_wait_intr+0xf0/0xf0 [ 7552.187198] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7552.190341] mdc_close+0x53b/0xd70 [mdc] [ 7552.192261] lmv_close+0x18d/0x490 [lmv] [ 7552.193835] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 7552.197032] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7552.199163] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7552.201044] ll_md_close+0x274/0x800 [lustre] [ 7552.202784] ll_file_release+0x128/0xc50 [lustre] [ 7552.204563] __fput+0x10d/0x440 [ 7552.205618] ____fput+0x12/0x20 [ 7552.206514] task_work_run+0xc6/0x110 [ 7552.207639] exit_to_usermode_loop+0x1dd/0x1f0 [ 7552.209132] do_syscall_64+0x3d6/0x440 [ 7552.210440] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7552.213650] RIP: 0033:0x7ff54af45cd8 [ 7552.215570] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7552.226063] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7552.231263] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 7552.236765] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 7552.241836] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 7552.248761] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 7552.252809] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7556.059011] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 7556.081210] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7556.169202] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7556.175015] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7556.179707] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7556.188248] RIP: 0010:__pv_queued_spin_lock_slowpath+0x315/0x6d0 [ 7556.194517] Code: 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f 84 3b 02 00 00 <8b> 55 08 85 d2 75 0e 84 c0 74 d5 48 83 05 18 d4 dc 02 01 eb e0 48 [ 7556.204471] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7556.212974] RAX: 0000000000004ed0 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 7556.228437] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 7556.237909] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 7556.246465] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 7556.250829] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 7556.255117] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 7556.258785] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7556.261712] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 7556.264665] Call Trace: [ 7556.265587] [ 7556.266496] ? show_regs.cold.9+0x22/0x2f [ 7556.268325] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7556.270454] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7556.282482] ? watchdog+0x40/0x40 [ 7556.283691] ? hrtimer_interrupt+0x128/0x2c0 [ 7556.288368] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7556.291318] ? apic_timer_interrupt+0xf/0x20 [ 7556.293670] [ 7556.294613] ? __pv_queued_spin_lock_slowpath+0x315/0x6d0 [ 7556.300858] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7556.308568] do_raw_spin_lock+0x177/0x190 [ 7556.313765] _raw_spin_lock+0x21/0x30 [ 7556.318467] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7556.323313] process_one_work+0x2c8/0x700 [ 7556.327217] worker_thread+0x296/0x6e0 [ 7556.332668] ? rescuer_thread+0x570/0x570 [ 7556.334047] kthread+0x1d1/0x200 [ 7556.337699] ? set_kthread_struct+0x70/0x70 [ 7556.344271] ret_from_fork+0x1f/0x30 [ 7560.068540] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 7560.090701] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7560.170878] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7560.192058] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7560.215561] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7560.217540] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7560.250692] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7560.258468] RAX: 0000000000004380 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 7560.268697] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 7560.277635] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7560.284439] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 7560.289077] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 7560.292693] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 7560.298167] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7560.300656] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 7560.304122] Call Trace: [ 7560.304788] [ 7560.305365] ? show_regs.cold.9+0x22/0x2f [ 7560.307879] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7560.310539] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7560.312183] ? watchdog+0x40/0x40 [ 7560.313225] ? hrtimer_interrupt+0x128/0x2c0 [ 7560.314760] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7560.316191] ? apic_timer_interrupt+0xf/0x20 [ 7560.317356] [ 7560.318406] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7560.320701] do_raw_spin_lock+0x177/0x190 [ 7560.322760] _raw_spin_lock+0x21/0x30 [ 7560.327634] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7560.334748] ? do_raw_spin_unlock+0x75/0x190 [ 7560.339611] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7560.345144] ? do_wait_intr+0xf0/0xf0 [ 7560.349047] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7560.351295] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7560.355074] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7560.362077] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7560.366757] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7560.374470] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7560.378783] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7560.383684] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7560.389083] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7560.399291] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7560.407679] lmv_intent_open+0x338/0x1150 [lmv] [ 7560.413433] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7560.422906] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7560.427709] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7560.441431] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7560.445838] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7560.458146] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7560.465652] ? do_raw_spin_unlock+0x75/0x190 [ 7560.467417] ? _raw_spin_unlock+0x12/0x30 [ 7560.479599] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7560.487834] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7560.494378] ll_atomic_open+0x36a/0xd70 [lustre] [ 7560.505169] lookup_open+0x356/0xb40 [ 7560.508541] do_last+0x313/0xfc0 [ 7560.516734] ? nd_jump_root+0xe5/0x160 [ 7560.518742] ? path_init+0x437/0x520 [ 7560.530380] path_openat+0xf7/0x500 [ 7560.539715] ? do_raw_spin_unlock+0x75/0x190 [ 7560.547902] do_filp_open+0x99/0x140 [ 7560.556265] ? getname_flags+0x6e/0x330 [ 7560.561711] ? __check_object_size+0xff/0x256 [ 7560.573152] ? do_raw_spin_unlock+0x75/0x190 [ 7560.577351] ? _raw_spin_unlock+0x12/0x30 [ 7560.581881] do_sys_openat2+0x2b4/0x410 [ 7560.598849] do_sys_open+0x73/0xa0 [ 7560.605714] __x64_sys_openat+0x24/0x30 [ 7560.610463] do_syscall_64+0xc1/0x440 [ 7560.620923] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7560.626307] RIP: 0033:0x7ff54af45332 [ 7560.633524] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7560.683428] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7560.705496] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 7560.720777] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 7560.734843] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 7560.749149] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 7560.768212] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 7576.055398] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 7576.064273] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7576.124311] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7576.150181] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7576.166670] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7576.173668] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7576.196571] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7576.208294] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 7576.214135] RDX: 00000000000039e7 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 7576.221276] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 7576.224888] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 7576.232656] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7576.236965] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 7576.241203] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7576.244541] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 7576.253270] Call Trace: [ 7576.256071] [ 7576.257035] ? show_regs.cold.9+0x22/0x2f [ 7576.259461] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7576.264452] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7576.266679] ? watchdog+0x40/0x40 [ 7576.268598] ? hrtimer_interrupt+0x128/0x2c0 [ 7576.271644] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7576.274513] ? apic_timer_interrupt+0xf/0x20 [ 7576.279354] [ 7576.280433] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7576.284305] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7576.288497] do_raw_spin_lock+0x177/0x190 [ 7576.290327] _raw_spin_lock+0x21/0x30 [ 7576.292445] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7576.299614] obd_mod_free+0x46/0x140 [ptlrpc] [ 7576.304474] mdc_commit_open+0x91/0xb0 [mdc] [ 7576.307993] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7576.311666] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7576.315566] after_reply+0xe2c/0x1780 [ptlrpc] [ 7576.318190] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7576.321540] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7576.324650] ? do_wait_intr+0xf0/0xf0 [ 7576.329545] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7576.337144] kthread+0x1d1/0x200 [ 7576.342467] ? set_kthread_struct+0x70/0x70 [ 7576.347669] ret_from_fork+0x1f/0x30 [ 7580.058024] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 7580.060428] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7580.071570] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7580.074361] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7580.076664] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7580.077867] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7580.082774] RSP: 0018:ffffaf9fc1543ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7580.084712] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 7580.086523] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 7580.088614] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7580.090308] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 7580.092065] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 7580.101501] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 7580.105547] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7580.113661] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 7580.121281] Call Trace: [ 7580.122171] [ 7580.128557] ? show_regs.cold.9+0x22/0x2f [ 7580.134643] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7580.140696] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7580.149747] ? watchdog+0x40/0x40 [ 7580.153509] ? hrtimer_interrupt+0x128/0x2c0 [ 7580.156522] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7580.163608] ? apic_timer_interrupt+0xf/0x20 [ 7580.167085] [ 7580.167707] ? native_safe_halt+0x2e/0x50 [ 7580.173841] kvm_wait+0xcd/0xe0 [ 7580.175513] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7580.177213] do_raw_spin_lock+0x177/0x190 [ 7580.179599] _raw_spin_lock+0x21/0x30 [ 7580.182683] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7580.187215] ? do_raw_spin_unlock+0x75/0x190 [ 7580.189690] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7580.196767] ? do_wait_intr+0xf0/0xf0 [ 7580.199233] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7580.201442] mdc_close+0x53b/0xd70 [mdc] [ 7580.205441] lmv_close+0x18d/0x490 [lmv] [ 7580.207731] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 7580.216270] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7580.221238] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7580.223437] ll_md_close+0x274/0x800 [lustre] [ 7580.227419] ll_file_release+0x128/0xc50 [lustre] [ 7580.231189] __fput+0x10d/0x440 [ 7580.232205] ____fput+0x12/0x20 [ 7580.234347] task_work_run+0xc6/0x110 [ 7580.238244] exit_to_usermode_loop+0x1dd/0x1f0 [ 7580.242603] do_syscall_64+0x3d6/0x440 [ 7580.246384] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7580.252813] RIP: 0033:0x7ff54af45cd8 [ 7580.254182] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7580.276212] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7580.283741] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 7580.290796] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 7580.298786] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 7580.307247] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 7580.312230] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7582.175121] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5623s! [ 7582.179105] Showing busy workqueues and worker pools: [ 7582.181104] workqueue events: flags=0x0 [ 7582.182763] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7582.190564] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 7582.196520] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7582.201677] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 7582.214582] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7582.217839] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 7582.219779] workqueue events_long: flags=0x0 [ 7582.222494] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7582.224906] pending: lu_objects_flush [obdclass] [ 7582.227973] workqueue events_unbound: flags=0x2 [ 7582.230353] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7582.233035] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7582.238017] workqueue events_power_efficient: flags=0x80 [ 7582.240265] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7582.243689] pending: neigh_periodic_work [ 7582.246432] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7582.249846] pending: do_cache_clean, check_lifetime [ 7582.255107] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7582.260186] pending: neigh_periodic_work [ 7582.264260] workqueue writeback: flags=0x4a [ 7582.267016] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7582.270662] pending: wb_workfn, wb_workfn [ 7582.273775] workqueue xprtiod: flags=0xa [ 7582.277510] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7582.280404] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 7582.288961] workqueue ipv6_addrconf: flags=0x40008 [ 7582.290742] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7582.308582] pending: addrconf_verify_work [ 7582.311057] workqueue knbd0-recv: flags=0x1a [ 7582.313141] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7582.315947] in-flight: 657:recv_work [ 7582.317415] workqueue ptlrpc_pinger: flags=0x2 [ 7582.318957] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7582.321631] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 7582.326093] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5623s workers=4 idle: 20831 12642 28859 [ 7582.330039] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7572s workers=2 idle: 659 [ 7584.059085] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 7584.065213] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7584.134318] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7584.147544] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7584.155364] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7584.176068] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7584.193573] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7584.218320] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7584.227718] RAX: 0000000000003351 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 7584.238274] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 7584.246205] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 7584.253844] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 7584.256546] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 7584.261760] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 7584.271279] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7584.281167] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 7584.285228] Call Trace: [ 7584.290210] [ 7584.292207] ? show_regs.cold.9+0x22/0x2f [ 7584.300214] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7584.303108] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7584.312579] ? watchdog+0x40/0x40 [ 7584.315511] ? hrtimer_interrupt+0x128/0x2c0 [ 7584.320614] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7584.323841] ? apic_timer_interrupt+0xf/0x20 [ 7584.327418] [ 7584.329451] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7584.332308] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7584.334154] do_raw_spin_lock+0x177/0x190 [ 7584.336462] _raw_spin_lock+0x21/0x30 [ 7584.338200] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7584.342612] process_one_work+0x2c8/0x700 [ 7584.347194] worker_thread+0x296/0x6e0 [ 7584.352678] ? rescuer_thread+0x570/0x570 [ 7584.362592] kthread+0x1d1/0x200 [ 7584.363703] ? set_kthread_struct+0x70/0x70 [ 7584.369181] ret_from_fork+0x1f/0x30 [ 7588.060173] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 7588.062860] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7588.091582] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7588.098624] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7588.104123] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7588.108029] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7588.121385] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7588.126520] RAX: 000000000000238b RBX: ffff96848811cae0 RCX: 0000000000000001 [ 7588.133732] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 7588.144065] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7588.155402] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 7588.168054] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 7588.181080] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 7588.188467] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7588.198712] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 7588.210460] Call Trace: [ 7588.211510] [ 7588.216502] ? show_regs.cold.9+0x22/0x2f [ 7588.221420] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7588.224167] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7588.228502] ? watchdog+0x40/0x40 [ 7588.229804] ? hrtimer_interrupt+0x128/0x2c0 [ 7588.232521] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7588.235012] ? apic_timer_interrupt+0xf/0x20 [ 7588.237299] [ 7588.238065] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7588.241009] do_raw_spin_lock+0x177/0x190 [ 7588.242544] _raw_spin_lock+0x21/0x30 [ 7588.244613] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7588.247423] ? do_raw_spin_unlock+0x75/0x190 [ 7588.249312] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7588.252077] ? do_wait_intr+0xf0/0xf0 [ 7588.253442] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7588.256145] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7588.257982] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7588.261328] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7588.273385] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7588.276450] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7588.286031] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7588.297496] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7588.303667] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7588.313765] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7588.315575] lmv_intent_open+0x338/0x1150 [lmv] [ 7588.332440] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7588.344139] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7588.350463] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7588.354420] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7588.359456] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7588.376179] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7588.380581] ? do_raw_spin_unlock+0x75/0x190 [ 7588.388275] ? _raw_spin_unlock+0x12/0x30 [ 7588.389602] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7588.394460] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7588.402178] ll_atomic_open+0x36a/0xd70 [lustre] [ 7588.410116] lookup_open+0x356/0xb40 [ 7588.414144] do_last+0x313/0xfc0 [ 7588.419364] ? nd_jump_root+0xe5/0x160 [ 7588.420588] ? path_init+0x437/0x520 [ 7588.432010] path_openat+0xf7/0x500 [ 7588.433521] ? do_raw_spin_unlock+0x75/0x190 [ 7588.436413] do_filp_open+0x99/0x140 [ 7588.441601] ? getname_flags+0x6e/0x330 [ 7588.443465] ? __check_object_size+0xff/0x256 [ 7588.447669] ? do_raw_spin_unlock+0x75/0x190 [ 7588.450631] ? _raw_spin_unlock+0x12/0x30 [ 7588.458226] do_sys_openat2+0x2b4/0x410 [ 7588.460452] do_sys_open+0x73/0xa0 [ 7588.463482] __x64_sys_openat+0x24/0x30 [ 7588.465622] do_syscall_64+0xc1/0x440 [ 7588.473302] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7588.481921] RIP: 0033:0x7ff54af45332 [ 7588.482906] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7588.491288] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7588.508187] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 7588.516466] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 7588.526341] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 7588.532472] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 7588.540234] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 7594.005243] rcu: INFO: rcu_sched self-detected stall on CPU [ 7594.012015] rcu: 0-...!: (2927565 ticks this GP) idle=98a/1/0x4000000000000002 softirq=125154/125154 fqs=959 [ 7594.027511] (t=5640165 jiffies g=253585 q=4639) [ 7594.030454] rcu: rcu_sched kthread starved for 5635125 jiffies! g253585 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [ 7594.046911] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 7594.052415] rcu: RCU grace-period kthread stack dump: [ 7594.055241] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 7594.068500] Call Trace: [ 7594.070902] __schedule+0x351/0xcb0 [ 7594.074965] schedule+0xc0/0x180 [ 7594.079569] schedule_timeout+0xb4/0x190 [ 7594.085659] ? do_raw_spin_unlock+0x75/0x190 [ 7594.089771] ? __next_timer_interrupt+0x160/0x160 [ 7594.092345] ? prepare_to_swait_event+0xf3/0x140 [ 7594.097578] rcu_gp_kthread+0x8dd/0x15b0 [ 7594.100263] ? force_qs_rnp+0x2c0/0x2c0 [ 7594.104320] kthread+0x1d1/0x200 [ 7594.107366] ? set_kthread_struct+0x70/0x70 [ 7594.111645] ret_from_fork+0x1f/0x30 [ 7594.114592] rcu: Stack dump where RCU GP kthread last ran: [ 7594.116546] Sending NMI from CPU 0 to CPUs 2: [ 7594.135747] NMI backtrace for cpu 2 [ 7594.135755] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7594.135757] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7594.135758] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7594.135770] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7594.135772] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7594.135774] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000212 [ 7594.135776] RAX: 0000000000007c26 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 7594.135777] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 7594.135779] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 7594.135780] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 7594.135782] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 7594.135783] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 7594.135784] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7594.135786] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 7594.135787] Call Trace: [ 7594.135787] [ 7594.135788] ? show_regs.cold.9+0x22/0x2f [ 7594.135789] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 7594.135790] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 7594.135791] ? nmi_handle+0xbd/0x210 [ 7594.135792] ? default_do_nmi+0x4b/0x1b0 [ 7594.135793] ? do_nmi+0x280/0x380 [ 7594.135794] ? end_repeat_nmi+0x16/0x1f [ 7594.135795] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7594.135797] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7594.135798] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7594.135799] [ 7594.135799] do_raw_spin_lock+0x177/0x190 [ 7594.135800] _raw_spin_lock+0x21/0x30 [ 7594.135801] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7594.135802] process_one_work+0x2c8/0x700 [ 7594.135803] worker_thread+0x296/0x6e0 [ 7594.135804] ? rescuer_thread+0x570/0x570 [ 7594.135806] kthread+0x1d1/0x200 [ 7594.135807] ? set_kthread_struct+0x70/0x70 [ 7594.135807] ret_from_fork+0x1f/0x30 [ 7594.135853] NMI backtrace for cpu 0 [ 7594.402127] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7594.412705] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7594.421109] Call Trace: [ 7594.426411] [ 7594.428882] ? dump_stack+0xbb/0x10e [ 7594.435367] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 7594.446383] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 7594.450404] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 7594.452552] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 7594.458393] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 7594.467127] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 7594.469363] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 7594.481211] ? update_process_times+0x6d/0xc0 [ 7594.484679] ? tick_sched_handle.isra.18+0x27/0x90 [ 7594.495167] ? tick_sched_timer+0x53/0xb0 [ 7594.502324] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7594.506579] ? tick_sched_handle.isra.18+0x90/0x90 [ 7594.515276] ? hrtimer_interrupt+0x128/0x2c0 [ 7594.519462] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7594.528094] ? apic_timer_interrupt+0xf/0x20 [ 7594.529668] [ 7594.530462] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7594.541211] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7594.551340] ? do_raw_spin_lock+0x177/0x190 [ 7594.558629] ? _raw_spin_lock+0x21/0x30 [ 7594.564068] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7594.573066] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 7594.575054] ? mdc_commit_open+0x91/0xb0 [mdc] [ 7594.580771] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7594.596878] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7594.599004] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 7594.608383] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7594.617734] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7594.621886] ? do_wait_intr+0xf0/0xf0 [ 7594.625088] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7594.627279] ? kthread+0x1d1/0x200 [ 7594.628285] ? set_kthread_struct+0x70/0x70 [ 7594.630900] ? ret_from_fork+0x1f/0x30 [ 7608.060081] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 7608.064925] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7608.134918] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7608.158890] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7608.167177] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7608.181047] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7608.200848] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7608.215954] RAX: 0000000000007211 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 7608.223127] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 7608.230017] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7608.236885] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 7608.242872] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 7608.250665] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 7608.265429] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7608.273657] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 7608.289378] Call Trace: [ 7608.292057] [ 7608.295031] ? show_regs.cold.9+0x22/0x2f [ 7608.300248] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7608.301783] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7608.310022] ? watchdog+0x40/0x40 [ 7608.311896] ? hrtimer_interrupt+0x128/0x2c0 [ 7608.316796] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7608.320378] ? apic_timer_interrupt+0xf/0x20 [ 7608.322782] [ 7608.327466] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7608.330294] do_raw_spin_lock+0x177/0x190 [ 7608.335434] _raw_spin_lock+0x21/0x30 [ 7608.337617] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7608.340292] ? do_raw_spin_unlock+0x75/0x190 [ 7608.343060] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7608.347681] ? do_wait_intr+0xf0/0xf0 [ 7608.352237] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7608.357961] mdc_close+0x53b/0xd70 [mdc] [ 7608.360070] lmv_close+0x18d/0x490 [lmv] [ 7608.365164] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 7608.371038] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7608.377159] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7608.383719] ll_md_close+0x274/0x800 [lustre] [ 7608.385931] ll_file_release+0x128/0xc50 [lustre] [ 7608.399129] __fput+0x10d/0x440 [ 7608.400795] ____fput+0x12/0x20 [ 7608.421607] task_work_run+0xc6/0x110 [ 7608.423445] exit_to_usermode_loop+0x1dd/0x1f0 [ 7608.428141] do_syscall_64+0x3d6/0x440 [ 7608.431239] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7608.451435] RIP: 0033:0x7ff54af45cd8 [ 7608.453503] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7608.488655] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7608.494551] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 7608.501779] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 7608.505391] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 7608.510184] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 7608.514120] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7612.383155] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5653s! [ 7612.387234] Showing busy workqueues and worker pools: [ 7612.389722] workqueue events: flags=0x0 [ 7612.391121] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7612.404752] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 7612.420165] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7612.430118] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 7612.451624] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7612.458540] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 7612.463654] workqueue events_long: flags=0x0 [ 7612.471759] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7612.483171] pending: lu_objects_flush [obdclass] [ 7612.492970] workqueue events_unbound: flags=0x2 [ 7612.500341] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7612.507386] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7612.526347] workqueue events_power_efficient: flags=0x80 [ 7612.540526] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7612.547128] pending: neigh_periodic_work [ 7612.553728] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7612.561531] pending: do_cache_clean, check_lifetime [ 7612.567216] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7612.574772] pending: neigh_periodic_work [ 7612.579393] workqueue writeback: flags=0x4a [ 7612.588794] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7612.598390] pending: wb_workfn, wb_workfn [ 7612.604648] workqueue xprtiod: flags=0xa [ 7612.609726] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7612.615042] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 7612.625601] workqueue ipv6_addrconf: flags=0x40008 [ 7612.632118] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7612.640853] pending: addrconf_verify_work [ 7612.642412] workqueue knbd0-recv: flags=0x1a [ 7612.648018] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7612.657802] in-flight: 657:recv_work [ 7612.664303] workqueue ptlrpc_pinger: flags=0x2 [ 7612.671086] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7612.681123] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 7612.689110] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5653s workers=4 idle: 20831 12642 28859 [ 7612.699928] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7602s workers=2 idle: 659 [ 7616.063872] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [ 7616.069105] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7616.158870] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7616.172862] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7616.177346] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7616.180962] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7616.191813] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7616.197089] RAX: 0000000000007f07 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 7616.199929] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 7616.209559] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7616.212518] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 7616.219796] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 7616.227111] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 7616.233758] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7616.237392] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 7616.241388] Call Trace: [ 7616.242205] [ 7616.242824] ? show_regs.cold.9+0x22/0x2f [ 7616.244230] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7616.249186] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7616.252964] ? watchdog+0x40/0x40 [ 7616.255902] ? hrtimer_interrupt+0x128/0x2c0 [ 7616.260951] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7616.264911] ? apic_timer_interrupt+0xf/0x20 [ 7616.272055] [ 7616.273092] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7616.277798] do_raw_spin_lock+0x177/0x190 [ 7616.280891] _raw_spin_lock+0x21/0x30 [ 7616.284739] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7616.290654] ? do_raw_spin_unlock+0x75/0x190 [ 7616.297891] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7616.300972] ? do_wait_intr+0xf0/0xf0 [ 7616.303778] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7616.307891] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7616.309888] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7616.318107] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7616.322164] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7616.324896] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7616.327077] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7616.333134] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7616.343208] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7616.346931] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7616.354166] lmv_intent_open+0x338/0x1150 [lmv] [ 7616.360508] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7616.368655] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7616.374111] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7616.380801] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7616.385210] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7616.390814] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7616.392254] ? do_raw_spin_unlock+0x75/0x190 [ 7616.394034] ? _raw_spin_unlock+0x12/0x30 [ 7616.395917] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7616.398508] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7616.400650] ll_atomic_open+0x36a/0xd70 [lustre] [ 7616.402706] lookup_open+0x356/0xb40 [ 7616.404114] do_last+0x313/0xfc0 [ 7616.405597] ? nd_jump_root+0xe5/0x160 [ 7616.407070] ? path_init+0x437/0x520 [ 7616.409938] path_openat+0xf7/0x500 [ 7616.411093] ? do_raw_spin_unlock+0x75/0x190 [ 7616.412540] do_filp_open+0x99/0x140 [ 7616.421598] ? getname_flags+0x6e/0x330 [ 7616.423213] ? __check_object_size+0xff/0x256 [ 7616.427864] ? do_raw_spin_unlock+0x75/0x190 [ 7616.429758] ? _raw_spin_unlock+0x12/0x30 [ 7616.432090] do_sys_openat2+0x2b4/0x410 [ 7616.434212] do_sys_open+0x73/0xa0 [ 7616.436167] __x64_sys_openat+0x24/0x30 [ 7616.438005] do_syscall_64+0xc1/0x440 [ 7616.447044] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7616.454724] RIP: 0033:0x7ff54af45332 [ 7616.456008] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7616.496158] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7616.502133] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 7616.511772] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 7616.517859] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 7616.527095] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 7616.533799] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 7620.059024] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 7620.068229] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7620.098380] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7620.103585] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7620.107429] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7620.110945] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7620.119999] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7620.139084] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7620.147188] RAX: 0000000000005809 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 7620.152756] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 7620.159902] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 7620.163828] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 7620.181348] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 7620.185881] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 7620.197942] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7620.202646] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 7620.214467] Call Trace: [ 7620.215687] [ 7620.217802] ? show_regs.cold.9+0x22/0x2f [ 7620.221180] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7620.228306] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7620.231866] ? watchdog+0x40/0x40 [ 7620.234426] ? hrtimer_interrupt+0x128/0x2c0 [ 7620.238764] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7620.240372] ? apic_timer_interrupt+0xf/0x20 [ 7620.243695] [ 7620.244686] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7620.252681] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7620.259244] do_raw_spin_lock+0x177/0x190 [ 7620.261822] _raw_spin_lock+0x21/0x30 [ 7620.266387] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7620.269798] process_one_work+0x2c8/0x700 [ 7620.273908] worker_thread+0x296/0x6e0 [ 7620.280556] ? rescuer_thread+0x570/0x570 [ 7620.282940] kthread+0x1d1/0x200 [ 7620.289234] ? set_kthread_struct+0x70/0x70 [ 7620.292166] ret_from_fork+0x1f/0x30 [ 7636.060586] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 7636.067517] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7636.100835] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7636.107706] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7636.111755] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7636.116793] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7636.138393] RSP: 0018:ffffaf9fc1543ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7636.148793] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 7636.152611] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 7636.157375] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7636.165286] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 7636.171794] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 7636.174761] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 7636.178350] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7636.180178] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 7636.192122] Call Trace: [ 7636.195996] [ 7636.197093] ? show_regs.cold.9+0x22/0x2f [ 7636.198940] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7636.205203] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7636.212568] ? watchdog+0x40/0x40 [ 7636.213949] ? hrtimer_interrupt+0x128/0x2c0 [ 7636.217387] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7636.224177] ? apic_timer_interrupt+0xf/0x20 [ 7636.229347] [ 7636.231472] ? native_safe_halt+0x2e/0x50 [ 7636.238024] kvm_wait+0xcd/0xe0 [ 7636.238987] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7636.243524] do_raw_spin_lock+0x177/0x190 [ 7636.249484] _raw_spin_lock+0x21/0x30 [ 7636.250667] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7636.258330] ? do_raw_spin_unlock+0x75/0x190 [ 7636.271943] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7636.281572] ? do_wait_intr+0xf0/0xf0 [ 7636.282946] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7636.292497] mdc_close+0x53b/0xd70 [mdc] [ 7636.295722] lmv_close+0x18d/0x490 [lmv] [ 7636.297804] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 7636.302238] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7636.304310] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7636.312553] ll_md_close+0x274/0x800 [lustre] [ 7636.315831] ll_file_release+0x128/0xc50 [lustre] [ 7636.318516] __fput+0x10d/0x440 [ 7636.323423] ____fput+0x12/0x20 [ 7636.326037] task_work_run+0xc6/0x110 [ 7636.329580] exit_to_usermode_loop+0x1dd/0x1f0 [ 7636.333347] do_syscall_64+0x3d6/0x440 [ 7636.338886] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7636.340931] RIP: 0033:0x7ff54af45cd8 [ 7636.343481] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7636.359364] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7636.363806] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 7636.367976] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 7636.371151] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 7636.378839] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 7636.386474] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7640.055051] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 7640.061076] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7640.100064] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7640.105918] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7640.111879] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7640.115768] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7640.126951] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7640.132712] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 7640.139182] RDX: 0000000000006589 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 7640.144891] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 7640.148995] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 7640.153233] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7640.157336] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 7640.161986] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7640.165291] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 7640.169409] Call Trace: [ 7640.170855] [ 7640.172015] ? show_regs.cold.9+0x22/0x2f [ 7640.174330] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7640.177107] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7640.179864] ? watchdog+0x40/0x40 [ 7640.182151] ? hrtimer_interrupt+0x128/0x2c0 [ 7640.184882] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7640.187953] ? apic_timer_interrupt+0xf/0x20 [ 7640.190523] [ 7640.191755] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7640.194858] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7640.198049] do_raw_spin_lock+0x177/0x190 [ 7640.200515] _raw_spin_lock+0x21/0x30 [ 7640.202999] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7640.206080] obd_mod_free+0x46/0x140 [ptlrpc] [ 7640.209046] mdc_commit_open+0x91/0xb0 [mdc] [ 7640.211742] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7640.214949] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7640.218591] after_reply+0xe2c/0x1780 [ptlrpc] [ 7640.221608] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7640.224952] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7640.227839] ? do_wait_intr+0xf0/0xf0 [ 7640.229990] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7640.233522] kthread+0x1d1/0x200 [ 7640.235464] ? set_kthread_struct+0x70/0x70 [ 7640.238039] ret_from_fork+0x1f/0x30 [ 7643.103045] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5684s! [ 7643.107157] Showing busy workqueues and worker pools: [ 7643.109971] workqueue events: flags=0x0 [ 7643.112399] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7643.115735] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 7643.130381] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7643.138445] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 7643.145494] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7643.149514] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 7643.156724] workqueue events_long: flags=0x0 [ 7643.159905] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7643.166531] pending: lu_objects_flush [obdclass] [ 7643.170134] workqueue events_unbound: flags=0x2 [ 7643.175778] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7643.178941] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7643.196525] workqueue events_power_efficient: flags=0x80 [ 7643.199938] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7643.213482] pending: neigh_periodic_work [ 7643.216713] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7643.231970] pending: do_cache_clean, check_lifetime [ 7643.234756] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7643.242953] pending: neigh_periodic_work [ 7643.248489] workqueue writeback: flags=0x4a [ 7643.255346] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7643.261930] pending: wb_workfn, wb_workfn [ 7643.265715] workqueue xprtiod: flags=0xa [ 7643.270503] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7643.275441] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 7643.282178] workqueue ipv6_addrconf: flags=0x40008 [ 7643.289963] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7643.296839] pending: addrconf_verify_work [ 7643.302308] workqueue knbd0-recv: flags=0x1a [ 7643.309771] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7643.316666] in-flight: 657:recv_work [ 7643.319355] workqueue ptlrpc_pinger: flags=0x2 [ 7643.322957] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7643.329881] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 7643.332950] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5684s workers=4 idle: 20831 12642 28859 [ 7643.341338] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7633s workers=2 idle: 659 [ 7644.060037] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [ 7644.066529] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7644.085480] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7644.090027] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7644.094329] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7644.096348] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7644.104611] RSP: 0000:ffffaf9fc15234f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7644.108255] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 7644.111501] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 7644.114849] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7644.118088] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 7644.121253] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 7644.124653] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 7644.128953] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7644.131308] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 7644.134723] Call Trace: [ 7644.136051] [ 7644.136903] ? show_regs.cold.9+0x22/0x2f [ 7644.138520] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7644.139975] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7644.141914] ? watchdog+0x40/0x40 [ 7644.143372] ? hrtimer_interrupt+0x128/0x2c0 [ 7644.145009] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7644.146819] ? apic_timer_interrupt+0xf/0x20 [ 7644.148520] [ 7644.149410] ? native_safe_halt+0x2e/0x50 [ 7644.151471] kvm_wait+0xcd/0xe0 [ 7644.152898] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7644.155417] do_raw_spin_lock+0x177/0x190 [ 7644.156961] _raw_spin_lock+0x21/0x30 [ 7644.158764] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7644.161130] ? do_raw_spin_unlock+0x75/0x190 [ 7644.163031] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7644.165789] ? do_wait_intr+0xf0/0xf0 [ 7644.167229] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7644.169883] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7644.171964] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7644.174945] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7644.177332] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7644.179676] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7644.181298] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7644.183210] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7644.185399] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7644.187238] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7644.189050] lmv_intent_open+0x338/0x1150 [lmv] [ 7644.193215] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7644.198358] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7644.202469] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7644.210414] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7644.211841] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7644.225009] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7644.226476] ? do_raw_spin_unlock+0x75/0x190 [ 7644.236205] ? _raw_spin_unlock+0x12/0x30 [ 7644.240399] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7644.249442] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7644.256717] ll_atomic_open+0x36a/0xd70 [lustre] [ 7644.260026] lookup_open+0x356/0xb40 [ 7644.266393] do_last+0x313/0xfc0 [ 7644.270395] ? nd_jump_root+0xe5/0x160 [ 7644.273328] ? path_init+0x437/0x520 [ 7644.276240] path_openat+0xf7/0x500 [ 7644.278295] ? do_raw_spin_unlock+0x75/0x190 [ 7644.279937] do_filp_open+0x99/0x140 [ 7644.281759] ? getname_flags+0x6e/0x330 [ 7644.285886] ? __check_object_size+0xff/0x256 [ 7644.289398] ? do_raw_spin_unlock+0x75/0x190 [ 7644.292535] ? _raw_spin_unlock+0x12/0x30 [ 7644.294546] do_sys_openat2+0x2b4/0x410 [ 7644.296413] do_sys_open+0x73/0xa0 [ 7644.297841] __x64_sys_openat+0x24/0x30 [ 7644.299527] do_syscall_64+0xc1/0x440 [ 7644.301813] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7644.304967] RIP: 0033:0x7ff54af45332 [ 7644.306296] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7644.324057] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7644.328300] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 7644.331526] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 7644.336394] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 7644.339574] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 7644.342480] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 7648.059030] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 7648.066694] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7648.095168] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7648.104425] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7648.113887] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7648.117973] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7648.121986] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7648.137782] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7648.145707] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 7648.148968] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 7648.151942] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 7648.157800] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 7648.162247] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 7648.167681] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 7648.171755] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7648.173925] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 7648.179339] Call Trace: [ 7648.180217] [ 7648.182844] ? show_regs.cold.9+0x22/0x2f [ 7648.185300] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7648.188529] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7648.191584] ? watchdog+0x40/0x40 [ 7648.193675] ? hrtimer_interrupt+0x128/0x2c0 [ 7648.195980] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7648.198943] ? apic_timer_interrupt+0xf/0x20 [ 7648.203457] [ 7648.204461] ? native_safe_halt+0x2e/0x50 [ 7648.207834] kvm_wait+0xcd/0xe0 [ 7648.210500] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7648.212996] do_raw_spin_lock+0x177/0x190 [ 7648.214157] _raw_spin_lock+0x21/0x30 [ 7648.216304] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7648.219947] process_one_work+0x2c8/0x700 [ 7648.222871] worker_thread+0x296/0x6e0 [ 7648.224681] ? rescuer_thread+0x570/0x570 [ 7648.225860] kthread+0x1d1/0x200 [ 7648.228557] ? set_kthread_struct+0x70/0x70 [ 7648.230379] ret_from_fork+0x1f/0x30 [ 7664.058094] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 7664.061320] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7664.085367] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7664.090129] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7664.094263] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7664.095683] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7664.113504] RSP: 0018:ffffaf9fc1543ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7664.126528] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 7664.137676] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 7664.142554] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7664.148583] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 7664.158518] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 7664.165173] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 7664.169651] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7664.172194] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 7664.177617] Call Trace: [ 7664.179377] [ 7664.180432] ? show_regs.cold.9+0x22/0x2f [ 7664.182643] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7664.186118] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7664.187490] ? watchdog+0x40/0x40 [ 7664.199580] ? hrtimer_interrupt+0x128/0x2c0 [ 7664.204274] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7664.207479] ? apic_timer_interrupt+0xf/0x20 [ 7664.211705] [ 7664.212631] ? native_safe_halt+0x2e/0x50 [ 7664.214044] kvm_wait+0xcd/0xe0 [ 7664.214927] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7664.220344] do_raw_spin_lock+0x177/0x190 [ 7664.221509] _raw_spin_lock+0x21/0x30 [ 7664.222686] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7664.224347] ? do_raw_spin_unlock+0x75/0x190 [ 7664.225384] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7664.227182] ? do_wait_intr+0xf0/0xf0 [ 7664.228092] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7664.229589] mdc_close+0x53b/0xd70 [mdc] [ 7664.230695] lmv_close+0x18d/0x490 [lmv] [ 7664.231670] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 7664.233383] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7664.235321] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7664.236617] ll_md_close+0x274/0x800 [lustre] [ 7664.237720] ll_file_release+0x128/0xc50 [lustre] [ 7664.239212] __fput+0x10d/0x440 [ 7664.240106] ____fput+0x12/0x20 [ 7664.240747] task_work_run+0xc6/0x110 [ 7664.241585] exit_to_usermode_loop+0x1dd/0x1f0 [ 7664.242540] do_syscall_64+0x3d6/0x440 [ 7664.243421] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7664.244764] RIP: 0033:0x7ff54af45cd8 [ 7664.245613] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7664.251518] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7664.253321] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 7664.254896] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 7664.256513] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 7664.259033] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 7664.261054] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7668.055010] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 7668.057598] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7668.112938] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7668.127752] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7668.144480] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7668.149271] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7668.166280] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7668.177818] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 7668.186718] RDX: 0000000000005042 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 7668.191581] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 7668.201414] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 7668.210266] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7668.213998] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 7668.223712] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7668.232016] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 7668.245288] Call Trace: [ 7668.249451] [ 7668.250340] ? show_regs.cold.9+0x22/0x2f [ 7668.256245] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7668.260613] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7668.266706] ? watchdog+0x40/0x40 [ 7668.269493] ? hrtimer_interrupt+0x128/0x2c0 [ 7668.272575] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7668.277246] ? apic_timer_interrupt+0xf/0x20 [ 7668.279858] [ 7668.285499] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7668.302756] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7668.307479] do_raw_spin_lock+0x177/0x190 [ 7668.312667] _raw_spin_lock+0x21/0x30 [ 7668.316111] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7668.329824] obd_mod_free+0x46/0x140 [ptlrpc] [ 7668.337239] mdc_commit_open+0x91/0xb0 [mdc] [ 7668.343313] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7668.355071] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7668.368380] after_reply+0xe2c/0x1780 [ptlrpc] [ 7668.372313] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7668.380838] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7668.390080] ? do_wait_intr+0xf0/0xf0 [ 7668.394981] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7668.399763] kthread+0x1d1/0x200 [ 7668.407010] ? set_kthread_struct+0x70/0x70 [ 7668.415137] ret_from_fork+0x1f/0x30 [ 7672.064446] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [ 7672.092128] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7672.163531] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7672.178720] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7672.188855] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7672.197690] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7672.215832] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7672.220475] RAX: 000000000000791d RBX: ffff96848811cae0 RCX: 0000000000000001 [ 7672.236701] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 7672.250130] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7672.258817] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 7672.272150] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 7672.274513] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 7672.302434] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7672.308260] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 7672.322217] Call Trace: [ 7672.324823] [ 7672.328303] ? show_regs.cold.9+0x22/0x2f [ 7672.331822] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7672.341348] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7672.348355] ? watchdog+0x40/0x40 [ 7672.354403] ? hrtimer_interrupt+0x128/0x2c0 [ 7672.361513] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7672.370021] ? apic_timer_interrupt+0xf/0x20 [ 7672.374431] [ 7672.379034] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7672.387375] do_raw_spin_lock+0x177/0x190 [ 7672.392127] _raw_spin_lock+0x21/0x30 [ 7672.396327] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7672.403123] ? do_raw_spin_unlock+0x75/0x190 [ 7672.410237] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7672.419267] ? do_wait_intr+0xf0/0xf0 [ 7672.437955] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7672.447205] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7672.464387] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7672.473817] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7672.485670] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7672.500253] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7672.502573] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7672.504792] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7672.519659] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7672.527394] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7672.530543] lmv_intent_open+0x338/0x1150 [lmv] [ 7672.537466] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7672.540966] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7672.542482] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7672.543595] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7672.544980] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7672.546802] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7672.548612] ? do_raw_spin_unlock+0x75/0x190 [ 7672.550302] ? _raw_spin_unlock+0x12/0x30 [ 7672.553492] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7672.559633] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7672.562161] ll_atomic_open+0x36a/0xd70 [lustre] [ 7672.564727] lookup_open+0x356/0xb40 [ 7672.569736] do_last+0x313/0xfc0 [ 7672.570911] ? nd_jump_root+0xe5/0x160 [ 7672.573037] ? path_init+0x437/0x520 [ 7672.574650] path_openat+0xf7/0x500 [ 7672.576508] ? do_raw_spin_unlock+0x75/0x190 [ 7672.584189] do_filp_open+0x99/0x140 [ 7672.588147] ? getname_flags+0x6e/0x330 [ 7672.592746] ? __check_object_size+0xff/0x256 [ 7672.600967] ? do_raw_spin_unlock+0x75/0x190 [ 7672.603362] ? _raw_spin_unlock+0x12/0x30 [ 7672.606074] do_sys_openat2+0x2b4/0x410 [ 7672.609068] do_sys_open+0x73/0xa0 [ 7672.612219] __x64_sys_openat+0x24/0x30 [ 7672.620162] do_syscall_64+0xc1/0x440 [ 7672.626643] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7672.629365] RIP: 0033:0x7ff54af45332 [ 7672.630983] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7672.653161] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7672.662385] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 7672.670388] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 7672.679206] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 7672.682131] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 7672.687358] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 7673.825512] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5714s! [ 7673.833163] Showing busy workqueues and worker pools: [ 7673.843080] workqueue events: flags=0x0 [ 7673.848034] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7673.863059] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 7673.874967] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7673.878913] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 7673.885788] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7673.887677] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 7673.896012] workqueue events_long: flags=0x0 [ 7673.899077] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7673.902534] pending: lu_objects_flush [obdclass] [ 7673.905753] workqueue events_unbound: flags=0x2 [ 7673.908324] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7673.912263] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7673.921971] workqueue events_power_efficient: flags=0x80 [ 7673.927703] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7673.933112] pending: neigh_periodic_work [ 7673.935230] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7673.943851] pending: do_cache_clean, check_lifetime [ 7673.954971] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7673.966945] pending: neigh_periodic_work [ 7673.975433] workqueue writeback: flags=0x4a [ 7673.979202] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7673.991179] pending: wb_workfn, wb_workfn [ 7673.992834] workqueue xprtiod: flags=0xa [ 7673.996818] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7674.003655] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 7674.012167] workqueue ipv6_addrconf: flags=0x40008 [ 7674.032892] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7674.049082] pending: addrconf_verify_work [ 7674.050401] workqueue knbd0-recv: flags=0x1a [ 7674.051548] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7674.072311] in-flight: 657:recv_work [ 7674.076878] workqueue ptlrpc_pinger: flags=0x2 [ 7674.079754] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7674.086979] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 7674.096400] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5715s workers=4 idle: 20831 12642 28859 [ 7674.101897] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7664s workers=2 idle: 659 [ 7676.060502] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 7676.078160] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7676.146031] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7676.166832] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7676.193731] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7676.208551] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7676.219373] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7676.272453] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7676.274931] RAX: 0000000000007f84 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 7676.289759] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 7676.296689] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 7676.307414] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 7676.320877] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 7676.333978] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 7676.338744] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7676.350656] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 7676.357849] Call Trace: [ 7676.362428] [ 7676.365112] ? show_regs.cold.9+0x22/0x2f [ 7676.378324] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7676.384156] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7676.390352] ? watchdog+0x40/0x40 [ 7676.394649] ? hrtimer_interrupt+0x128/0x2c0 [ 7676.410555] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7676.423699] ? apic_timer_interrupt+0xf/0x20 [ 7676.429059] [ 7676.432630] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7676.446164] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7676.457537] do_raw_spin_lock+0x177/0x190 [ 7676.458683] _raw_spin_lock+0x21/0x30 [ 7676.459683] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7676.483528] process_one_work+0x2c8/0x700 [ 7676.492071] worker_thread+0x296/0x6e0 [ 7676.499017] ? rescuer_thread+0x570/0x570 [ 7676.506723] kthread+0x1d1/0x200 [ 7676.510488] ? set_kthread_struct+0x70/0x70 [ 7676.517739] ret_from_fork+0x1f/0x30 [ 7692.062317] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 7692.070422] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7692.124147] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7692.131851] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7692.155037] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7692.163293] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7692.189970] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7692.199395] RAX: 0000000000006a4e RBX: ffff96848811cae0 RCX: 0000000000000001 [ 7692.206510] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 7692.212754] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7692.223086] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 7692.231160] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 7692.235875] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 7692.243455] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7692.248901] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 7692.256693] Call Trace: [ 7692.258874] [ 7692.264961] ? show_regs.cold.9+0x22/0x2f [ 7692.275428] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7692.278665] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7692.282354] ? watchdog+0x40/0x40 [ 7692.286519] ? hrtimer_interrupt+0x128/0x2c0 [ 7692.293851] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7692.297250] ? apic_timer_interrupt+0xf/0x20 [ 7692.304951] [ 7692.306120] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7692.310825] do_raw_spin_lock+0x177/0x190 [ 7692.316005] _raw_spin_lock+0x21/0x30 [ 7692.318333] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7692.324918] ? do_raw_spin_unlock+0x75/0x190 [ 7692.330117] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7692.333612] ? do_wait_intr+0xf0/0xf0 [ 7692.336139] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7692.346041] mdc_close+0x53b/0xd70 [mdc] [ 7692.349881] lmv_close+0x18d/0x490 [lmv] [ 7692.356623] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 7692.363334] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7692.368134] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7692.377023] ll_md_close+0x274/0x800 [lustre] [ 7692.386056] ll_file_release+0x128/0xc50 [lustre] [ 7692.393070] __fput+0x10d/0x440 [ 7692.396297] ____fput+0x12/0x20 [ 7692.398891] task_work_run+0xc6/0x110 [ 7692.401073] exit_to_usermode_loop+0x1dd/0x1f0 [ 7692.403706] do_syscall_64+0x3d6/0x440 [ 7692.405487] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7692.408916] RIP: 0033:0x7ff54af45cd8 [ 7692.411903] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7692.427610] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7692.431647] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 7692.437888] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 7692.444363] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 7692.449124] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 7692.453734] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7700.055023] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [ 7700.060025] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [ 7700.069852] Modules linked in: [ 7700.072242] Modules linked in: [ 7700.072246] lustre(O) [ 7700.073261] lustre(O) [ 7700.074361] osp(O) [ 7700.075316] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7700.075396] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7700.075402] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7700.075405] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7700.075418] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7700.075422] RSP: 0000:ffffaf9fc15234f0 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 7700.075427] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 7700.075429] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 7700.075432] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7700.075434] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 7700.075436] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 7700.075439] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 7700.075442] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7700.075445] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 7700.075453] Call Trace: [ 7700.075458] [ 7700.075463] ? show_regs.cold.9+0x22/0x2f [ 7700.075472] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7700.075478] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7700.075483] ? watchdog+0x40/0x40 [ 7700.075487] ? hrtimer_interrupt+0x128/0x2c0 [ 7700.075492] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7700.075499] ? apic_timer_interrupt+0xf/0x20 [ 7700.075503] [ 7700.075506] ? native_safe_halt+0x2e/0x50 [ 7700.075510] kvm_wait+0xcd/0xe0 [ 7700.075520] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7700.075529] do_raw_spin_lock+0x177/0x190 [ 7700.075533] _raw_spin_lock+0x21/0x30 [ 7700.075538] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7700.075867] ? do_raw_spin_unlock+0x75/0x190 [ 7700.075874] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7700.076067] ? do_wait_intr+0xf0/0xf0 [ 7700.076078] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7700.076279] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7700.076504] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7700.076720] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7700.076851] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7700.076888] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7700.076917] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7700.076946] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7700.077027] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7700.077235] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7700.077264] lmv_intent_open+0x338/0x1150 [lmv] [ 7700.077296] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7700.077375] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7700.077455] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7700.077477] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7700.077560] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7700.077642] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7700.077723] ? do_raw_spin_unlock+0x75/0x190 [ 7700.077730] ? _raw_spin_unlock+0x12/0x30 [ 7700.077749] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7700.077778] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7700.077798] ll_atomic_open+0x36a/0xd70 [lustre] [ 7700.077881] lookup_open+0x356/0xb40 [ 7700.077889] do_last+0x313/0xfc0 [ 7700.077894] ? nd_jump_root+0xe5/0x160 [ 7700.077898] ? path_init+0x437/0x520 [ 7700.077903] path_openat+0xf7/0x500 [ 7700.077908] ? do_raw_spin_unlock+0x75/0x190 [ 7700.077913] do_filp_open+0x99/0x140 [ 7700.077920] ? getname_flags+0x6e/0x330 [ 7700.077924] ? __check_object_size+0xff/0x256 [ 7700.077933] ? do_raw_spin_unlock+0x75/0x190 [ 7700.077937] ? _raw_spin_unlock+0x12/0x30 [ 7700.077942] do_sys_openat2+0x2b4/0x410 [ 7700.077946] do_sys_open+0x73/0xa0 [ 7700.077951] __x64_sys_openat+0x24/0x30 [ 7700.077955] do_syscall_64+0xc1/0x440 [ 7700.077961] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7700.077968] RIP: 0033:0x7ff54af45332 [ 7700.077977] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7700.077981] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7700.077987] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 7700.077990] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 7700.077992] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 7700.077995] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 7700.077997] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 7700.534493] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7700.610248] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7700.622182] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7700.631922] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7700.638450] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7700.649979] RSP: 0000:ffffaf9fc1b97bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7700.654354] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 7700.658479] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 7700.665785] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 7700.672810] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 7700.680981] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7700.687218] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 7700.691754] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7700.698770] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 7700.702617] Call Trace: [ 7700.704322] [ 7700.705385] ? show_regs.cold.9+0x22/0x2f [ 7700.707506] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7700.710453] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7700.712174] ? watchdog+0x40/0x40 [ 7700.713269] ? hrtimer_interrupt+0x128/0x2c0 [ 7700.720385] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7700.722318] ? apic_timer_interrupt+0xf/0x20 [ 7700.726835] [ 7700.731306] ? native_safe_halt+0x2e/0x50 [ 7700.735887] kvm_wait+0xcd/0xe0 [ 7700.739267] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7700.744126] do_raw_spin_lock+0x177/0x190 [ 7700.745824] _raw_spin_lock+0x21/0x30 [ 7700.749563] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7700.752514] obd_mod_free+0x46/0x140 [ptlrpc] [ 7700.758464] mdc_commit_open+0x91/0xb0 [mdc] [ 7700.762261] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7700.765946] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7700.777805] after_reply+0xe2c/0x1780 [ptlrpc] [ 7700.784421] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7700.793674] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7700.796385] ? do_wait_intr+0xf0/0xf0 [ 7700.806597] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7700.814353] kthread+0x1d1/0x200 [ 7700.817195] ? set_kthread_struct+0x70/0x70 [ 7700.824804] ret_from_fork+0x1f/0x30 [ 7704.059029] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 7704.071098] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7704.094362] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7704.098283] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7704.101561] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7704.103273] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7704.105092] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7704.111838] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7704.114273] RAX: 0000000000006b68 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 7704.116741] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 7704.118287] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 7704.123375] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 7704.127076] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 7704.130152] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 7704.134261] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7704.137072] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 7704.138770] Call Trace: [ 7704.139419] [ 7704.139920] ? show_regs.cold.9+0x22/0x2f [ 7704.141426] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7704.143366] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7704.145742] ? watchdog+0x40/0x40 [ 7704.147370] ? hrtimer_interrupt+0x128/0x2c0 [ 7704.149266] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7704.151404] ? apic_timer_interrupt+0xf/0x20 [ 7704.153072] [ 7704.153675] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7704.155469] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7704.157481] do_raw_spin_lock+0x177/0x190 [ 7704.159379] _raw_spin_lock+0x21/0x30 [ 7704.161800] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7704.164896] process_one_work+0x2c8/0x700 [ 7704.167935] worker_thread+0x296/0x6e0 [ 7704.169747] ? rescuer_thread+0x570/0x570 [ 7704.171034] kthread+0x1d1/0x200 [ 7704.172157] ? set_kthread_struct+0x70/0x70 [ 7704.173953] ret_from_fork+0x1f/0x30 [ 7704.543668] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5745s! [ 7704.550790] Showing busy workqueues and worker pools: [ 7704.556387] workqueue events: flags=0x0 [ 7704.557841] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7704.563640] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 7704.573914] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7704.580170] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 7704.594868] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7704.602349] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 7704.614502] workqueue events_long: flags=0x0 [ 7704.620577] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7704.630056] pending: lu_objects_flush [obdclass] [ 7704.638944] workqueue events_unbound: flags=0x2 [ 7704.644142] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7704.650325] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7704.662584] workqueue events_power_efficient: flags=0x80 [ 7704.675391] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7704.690325] pending: neigh_periodic_work [ 7704.694034] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7704.701766] pending: do_cache_clean, check_lifetime [ 7704.707264] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7704.709661] pending: neigh_periodic_work [ 7704.712309] workqueue writeback: flags=0x4a [ 7704.714669] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7704.718261] pending: wb_workfn, wb_workfn [ 7704.721379] workqueue xprtiod: flags=0xa [ 7704.724502] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7704.732397] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 7704.739830] workqueue ipv6_addrconf: flags=0x40008 [ 7704.747058] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7704.759663] pending: addrconf_verify_work [ 7704.762705] workqueue knbd0-recv: flags=0x1a [ 7704.765606] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7704.774199] in-flight: 657:recv_work [ 7704.779902] workqueue ptlrpc_pinger: flags=0x2 [ 7704.791317] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7704.801264] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 7704.803629] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5745s workers=4 idle: 20831 12642 28859 [ 7704.828356] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7695s workers=2 idle: 659 [ 7720.062451] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 7720.068075] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7720.129266] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7720.139124] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7720.150053] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7720.154088] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7720.173978] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7720.183184] RAX: 000000000000525c RBX: ffff96848811cae0 RCX: 0000000000000001 [ 7720.193487] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 7720.199349] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7720.208776] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 7720.213759] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 7720.220937] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 7720.230147] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7720.241483] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 7720.249890] Call Trace: [ 7720.252110] [ 7720.255974] ? show_regs.cold.9+0x22/0x2f [ 7720.259059] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7720.265683] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7720.267330] ? watchdog+0x40/0x40 [ 7720.271958] ? hrtimer_interrupt+0x128/0x2c0 [ 7720.274019] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7720.278913] ? apic_timer_interrupt+0xf/0x20 [ 7720.284692] [ 7720.288958] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7720.297166] do_raw_spin_lock+0x177/0x190 [ 7720.304719] _raw_spin_lock+0x21/0x30 [ 7720.305983] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7720.315116] ? do_raw_spin_unlock+0x75/0x190 [ 7720.320130] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7720.328069] ? do_wait_intr+0xf0/0xf0 [ 7720.331682] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7720.337195] mdc_close+0x53b/0xd70 [mdc] [ 7720.338628] lmv_close+0x18d/0x490 [lmv] [ 7720.340053] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 7720.345223] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7720.353046] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7720.359886] ll_md_close+0x274/0x800 [lustre] [ 7720.365117] ll_file_release+0x128/0xc50 [lustre] [ 7720.372657] __fput+0x10d/0x440 [ 7720.375076] ____fput+0x12/0x20 [ 7720.377049] task_work_run+0xc6/0x110 [ 7720.382025] exit_to_usermode_loop+0x1dd/0x1f0 [ 7720.394975] do_syscall_64+0x3d6/0x440 [ 7720.396379] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7720.403646] RIP: 0033:0x7ff54af45cd8 [ 7720.407986] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7720.432200] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7720.435051] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 7720.451558] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 7720.463224] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 7720.480715] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 7720.490443] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7728.060038] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [ 7728.063528] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7728.086809] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7728.092688] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7728.096181] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7728.099186] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7728.108518] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7728.111717] RAX: 0000000000003089 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 7728.115872] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 7728.119280] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7728.123600] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 7728.126835] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 7728.130258] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 7728.133806] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7728.136582] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 7728.139793] Call Trace: [ 7728.141230] [ 7728.142210] ? show_regs.cold.9+0x22/0x2f [ 7728.144173] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7728.146675] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7728.148800] ? watchdog+0x40/0x40 [ 7728.150567] ? hrtimer_interrupt+0x128/0x2c0 [ 7728.152659] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7728.155074] ? apic_timer_interrupt+0xf/0x20 [ 7728.157190] [ 7728.158238] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7728.161594] do_raw_spin_lock+0x177/0x190 [ 7728.163833] _raw_spin_lock+0x21/0x30 [ 7728.166271] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7728.170058] ? do_raw_spin_unlock+0x75/0x190 [ 7728.173248] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7728.177574] ? do_wait_intr+0xf0/0xf0 [ 7728.179609] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7728.183666] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7728.187601] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7728.191759] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7728.195532] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7728.199076] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7728.202848] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7728.205513] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7728.210560] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7728.218052] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7728.222585] lmv_intent_open+0x338/0x1150 [lmv] [ 7728.225485] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7728.229534] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7728.232852] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7728.234718] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7728.236790] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7728.238809] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7728.240906] ? do_raw_spin_unlock+0x75/0x190 [ 7728.242873] ? _raw_spin_unlock+0x12/0x30 [ 7728.244619] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7728.246684] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7728.248715] ll_atomic_open+0x36a/0xd70 [lustre] [ 7728.250867] lookup_open+0x356/0xb40 [ 7728.252900] do_last+0x313/0xfc0 [ 7728.254044] ? nd_jump_root+0xe5/0x160 [ 7728.255582] ? path_init+0x437/0x520 [ 7728.256947] path_openat+0xf7/0x500 [ 7728.258471] ? do_raw_spin_unlock+0x75/0x190 [ 7728.260670] do_filp_open+0x99/0x140 [ 7728.261911] ? getname_flags+0x6e/0x330 [ 7728.263662] ? __check_object_size+0xff/0x256 [ 7728.265443] ? do_raw_spin_unlock+0x75/0x190 [ 7728.266939] ? _raw_spin_unlock+0x12/0x30 [ 7728.268556] do_sys_openat2+0x2b4/0x410 [ 7728.269926] do_sys_open+0x73/0xa0 [ 7728.271687] __x64_sys_openat+0x24/0x30 [ 7728.273081] do_syscall_64+0xc1/0x440 [ 7728.274535] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7728.276628] RIP: 0033:0x7ff54af45332 [ 7728.277910] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7728.284537] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7728.287279] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 7728.289749] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 7728.292069] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 7728.294722] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 7728.297141] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 7732.059718] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 7732.061488] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [ 7732.068094] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7732.068173] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7732.068181] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7732.068189] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7732.079355] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7732.079451] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7732.079459] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7732.079462] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7732.079479] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7732.079484] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7732.079488] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 7732.079490] RDX: 0000000000007ce6 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 7732.079492] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 7732.079494] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 7732.079496] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7732.079499] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 7732.079502] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7732.079505] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 7732.079513] Call Trace: [ 7732.079519] [ 7732.079526] ? show_regs.cold.9+0x22/0x2f [ 7732.079536] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7732.079544] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7732.079548] ? watchdog+0x40/0x40 [ 7732.079553] ? hrtimer_interrupt+0x128/0x2c0 [ 7732.079557] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7732.079565] ? apic_timer_interrupt+0xf/0x20 [ 7732.079570] [ 7732.079584] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7732.079590] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7732.079594] do_raw_spin_lock+0x177/0x190 [ 7732.079599] _raw_spin_lock+0x21/0x30 [ 7732.079605] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7732.079934] obd_mod_free+0x46/0x140 [ptlrpc] [ 7732.080143] mdc_commit_open+0x91/0xb0 [mdc] [ 7732.080187] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7732.080379] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7732.080588] after_reply+0xe2c/0x1780 [ptlrpc] [ 7732.080798] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7732.080998] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7732.081194] ? do_wait_intr+0xf0/0xf0 [ 7732.081206] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7732.081405] kthread+0x1d1/0x200 [ 7732.081412] ? set_kthread_struct+0x70/0x70 [ 7732.081416] ret_from_fork+0x1f/0x30 [ 7732.234830] [ 7732.235453] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7732.237349] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7732.244392] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7732.246960] RAX: 0000000000007b12 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 7732.249540] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 7732.251827] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 7732.254193] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 7732.256727] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 7732.259239] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 7732.264864] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7732.269569] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 7732.272355] Call Trace: [ 7732.273206] [ 7732.274576] ? show_regs.cold.9+0x22/0x2f [ 7732.276245] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7732.278862] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7732.280922] ? watchdog+0x40/0x40 [ 7732.282405] ? hrtimer_interrupt+0x128/0x2c0 [ 7732.287171] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7732.289399] ? apic_timer_interrupt+0xf/0x20 [ 7732.294486] [ 7732.295230] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7732.298607] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7732.300713] do_raw_spin_lock+0x177/0x190 [ 7732.304755] _raw_spin_lock+0x21/0x30 [ 7732.306145] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7732.308745] process_one_work+0x2c8/0x700 [ 7732.310374] worker_thread+0x296/0x6e0 [ 7732.313814] ? rescuer_thread+0x570/0x570 [ 7732.318921] kthread+0x1d1/0x200 [ 7732.320399] ? set_kthread_struct+0x70/0x70 [ 7732.322234] ret_from_fork+0x1f/0x30 [ 7735.263209] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5776s! [ 7735.275971] Showing busy workqueues and worker pools: [ 7735.283792] workqueue events: flags=0x0 [ 7735.286733] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7735.295819] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 7735.308070] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7735.316829] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 7735.327019] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7735.331075] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 7735.338273] workqueue events_long: flags=0x0 [ 7735.346105] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7735.354866] pending: lu_objects_flush [obdclass] [ 7735.360870] workqueue events_unbound: flags=0x2 [ 7735.363123] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7735.368842] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7735.375880] workqueue events_power_efficient: flags=0x80 [ 7735.379112] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7735.382237] pending: neigh_periodic_work [ 7735.386894] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7735.391558] pending: do_cache_clean, check_lifetime [ 7735.393822] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7735.397693] pending: neigh_periodic_work [ 7735.399677] workqueue writeback: flags=0x4a [ 7735.401693] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7735.404108] pending: wb_workfn, wb_workfn [ 7735.405857] workqueue xprtiod: flags=0xa [ 7735.416208] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7735.422420] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 7735.426305] workqueue ipv6_addrconf: flags=0x40008 [ 7735.429358] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7735.431875] pending: addrconf_verify_work [ 7735.434000] workqueue knbd0-recv: flags=0x1a [ 7735.435158] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7735.438665] in-flight: 657:recv_work [ 7735.439818] workqueue ptlrpc_pinger: flags=0x2 [ 7735.441730] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7735.443625] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 7735.445821] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5776s workers=4 idle: 20831 12642 28859 [ 7735.448992] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7725s workers=2 idle: 659 [ 7748.058057] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 7748.061358] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7748.081621] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7748.087349] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7748.094899] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7748.097453] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7748.106305] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7748.108849] RAX: 0000000000005646 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 7748.111512] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 7748.114351] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7748.117860] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 7748.125560] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 7748.132450] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 7748.140800] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7748.143085] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 7748.150962] Call Trace: [ 7748.154859] [ 7748.156011] ? show_regs.cold.9+0x22/0x2f [ 7748.170006] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7748.177012] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7748.186534] ? watchdog+0x40/0x40 [ 7748.188348] ? hrtimer_interrupt+0x128/0x2c0 [ 7748.204183] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7748.209676] ? apic_timer_interrupt+0xf/0x20 [ 7748.220852] [ 7748.221590] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7748.228401] do_raw_spin_lock+0x177/0x190 [ 7748.235991] _raw_spin_lock+0x21/0x30 [ 7748.237591] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7748.243014] ? do_raw_spin_unlock+0x75/0x190 [ 7748.246003] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7748.248904] ? do_wait_intr+0xf0/0xf0 [ 7748.251228] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7748.256399] mdc_close+0x53b/0xd70 [mdc] [ 7748.258396] lmv_close+0x18d/0x490 [lmv] [ 7748.260460] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 7748.263980] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7748.267154] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7748.269233] ll_md_close+0x274/0x800 [lustre] [ 7748.271221] ll_file_release+0x128/0xc50 [lustre] [ 7748.272932] __fput+0x10d/0x440 [ 7748.274009] ____fput+0x12/0x20 [ 7748.274987] task_work_run+0xc6/0x110 [ 7748.276083] exit_to_usermode_loop+0x1dd/0x1f0 [ 7748.277515] do_syscall_64+0x3d6/0x440 [ 7748.278812] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7748.280568] RIP: 0033:0x7ff54af45cd8 [ 7748.281977] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7748.295041] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7748.299248] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 7748.302908] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 7748.307771] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 7748.312476] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 7748.322840] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7756.065979] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 7756.091623] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7756.222273] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7756.242042] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7756.261872] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7756.280160] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7756.324718] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7756.346664] RAX: 0000000000006e0e RBX: ffff96848811cae0 RCX: 0000000000000001 [ 7756.358455] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 7756.373570] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7756.390878] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 7756.405493] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 7756.407589] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 7756.432797] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7756.445557] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 7756.452226] Call Trace: [ 7756.453027] [ 7756.453719] ? show_regs.cold.9+0x22/0x2f [ 7756.458707] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7756.464247] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7756.468282] ? watchdog+0x40/0x40 [ 7756.470159] ? hrtimer_interrupt+0x128/0x2c0 [ 7756.474710] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7756.481157] ? apic_timer_interrupt+0xf/0x20 [ 7756.482708] [ 7756.484519] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7756.489345] do_raw_spin_lock+0x177/0x190 [ 7756.492377] _raw_spin_lock+0x21/0x30 [ 7756.497209] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7756.502288] ? do_raw_spin_unlock+0x75/0x190 [ 7756.507716] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7756.511573] ? do_wait_intr+0xf0/0xf0 [ 7756.518266] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7756.526855] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7756.529695] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7756.536122] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7756.543779] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7756.552460] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7756.556338] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7756.563159] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7756.566988] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7756.574932] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7756.580887] lmv_intent_open+0x338/0x1150 [lmv] [ 7756.586213] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7756.593943] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7756.600719] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7756.605700] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7756.648241] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7756.651926] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7756.657813] ? do_raw_spin_unlock+0x75/0x190 [ 7756.661226] ? _raw_spin_unlock+0x12/0x30 [ 7756.663738] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7756.670291] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7756.676182] ll_atomic_open+0x36a/0xd70 [lustre] [ 7756.679615] lookup_open+0x356/0xb40 [ 7756.683511] do_last+0x313/0xfc0 [ 7756.685809] ? nd_jump_root+0xe5/0x160 [ 7756.689902] ? path_init+0x437/0x520 [ 7756.693275] path_openat+0xf7/0x500 [ 7756.702626] ? do_raw_spin_unlock+0x75/0x190 [ 7756.704225] do_filp_open+0x99/0x140 [ 7756.706318] ? getname_flags+0x6e/0x330 [ 7756.709439] ? __check_object_size+0xff/0x256 [ 7756.715097] ? do_raw_spin_unlock+0x75/0x190 [ 7756.717783] ? _raw_spin_unlock+0x12/0x30 [ 7756.725985] do_sys_openat2+0x2b4/0x410 [ 7756.730658] do_sys_open+0x73/0xa0 [ 7756.737092] __x64_sys_openat+0x24/0x30 [ 7756.743720] do_syscall_64+0xc1/0x440 [ 7756.751153] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7756.754904] RIP: 0033:0x7ff54af45332 [ 7756.758804] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7756.774111] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7756.777850] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 7756.783748] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 7756.789837] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 7756.796860] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 7756.807734] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 7760.058079] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [ 7760.061724] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 7760.078089] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7760.078180] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7760.078188] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7760.078190] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7760.078204] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7760.078208] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7760.078213] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 7760.078215] RDX: 0000000000000d51 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 7760.078217] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 7760.078219] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 7760.078221] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7760.078225] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 7760.078227] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7760.078229] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 7760.078237] Call Trace: [ 7760.078243] [ 7760.078249] ? show_regs.cold.9+0x22/0x2f [ 7760.078258] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7760.080000] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7760.080017] ? watchdog+0x40/0x40 [ 7760.080026] ? hrtimer_interrupt+0x128/0x2c0 [ 7760.080030] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7760.080038] ? apic_timer_interrupt+0xf/0x20 [ 7760.080042] [ 7760.080046] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7760.080052] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7760.080057] do_raw_spin_lock+0x177/0x190 [ 7760.080063] _raw_spin_lock+0x21/0x30 [ 7760.080069] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7760.080405] obd_mod_free+0x46/0x140 [ptlrpc] [ 7760.080615] mdc_commit_open+0x91/0xb0 [mdc] [ 7760.080654] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7760.080873] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7760.108221] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) [ 7760.197553] after_reply+0xe2c/0x1780 [ptlrpc] [ 7760.197900] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7760.198096] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7760.202700] ? do_wait_intr+0xf0/0xf0 [ 7760.202714] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7760.203387] kthread+0x1d1/0x200 [ 7760.203395] ? set_kthread_struct+0x70/0x70 [ 7760.203400] ret_from_fork+0x1f/0x30 [ 7760.777670] mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7760.836704] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7760.849755] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7760.865616] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7760.869571] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7760.883629] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7760.943629] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7760.960964] RAX: 0000000000005305 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 7760.977905] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 7760.998431] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 7761.004568] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 7761.021955] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 7761.042477] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 7761.056540] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7761.072734] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 7761.087091] Call Trace: [ 7761.090305] [ 7761.096950] ? show_regs.cold.9+0x22/0x2f [ 7761.104021] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7761.112170] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7761.120506] ? watchdog+0x40/0x40 [ 7761.127255] ? hrtimer_interrupt+0x128/0x2c0 [ 7761.136680] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7761.165133] ? apic_timer_interrupt+0xf/0x20 [ 7761.166495] [ 7761.167161] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7761.189627] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7761.202892] do_raw_spin_lock+0x177/0x190 [ 7761.209553] _raw_spin_lock+0x21/0x30 [ 7761.211824] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7761.226885] process_one_work+0x2c8/0x700 [ 7761.236133] worker_thread+0x296/0x6e0 [ 7761.241608] ? rescuer_thread+0x570/0x570 [ 7761.250728] kthread+0x1d1/0x200 [ 7761.258710] ? set_kthread_struct+0x70/0x70 [ 7761.273210] ret_from_fork+0x1f/0x30 [ 7765.479083] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5806s! [ 7765.499936] Showing busy workqueues and worker pools: [ 7765.507148] workqueue events: flags=0x0 [ 7765.508772] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7765.517289] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 7765.528167] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7765.534341] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 7765.545730] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7765.552324] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 7765.555760] workqueue events_long: flags=0x0 [ 7765.565141] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7765.579306] pending: lu_objects_flush [obdclass] [ 7765.587771] workqueue events_unbound: flags=0x2 [ 7765.596189] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7765.607041] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7765.623261] workqueue events_power_efficient: flags=0x80 [ 7765.630515] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7765.638837] pending: neigh_periodic_work [ 7765.646465] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7765.660713] pending: do_cache_clean, check_lifetime [ 7765.665859] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7765.679491] pending: neigh_periodic_work [ 7765.681055] workqueue writeback: flags=0x4a [ 7765.684671] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7765.690325] pending: wb_workfn, wb_workfn [ 7765.696553] workqueue xprtiod: flags=0xa [ 7765.701700] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7765.706991] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 7765.714361] workqueue ipv6_addrconf: flags=0x40008 [ 7765.716499] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7765.719745] pending: addrconf_verify_work [ 7765.723156] workqueue knbd0-recv: flags=0x1a [ 7765.735351] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7765.740194] in-flight: 657:recv_work [ 7765.745357] workqueue ptlrpc_pinger: flags=0x2 [ 7765.750689] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7765.761173] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 7765.772847] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5806s workers=4 idle: 20831 12642 28859 [ 7765.793083] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7756s workers=2 idle: 659 [ 7774.009057] rcu: INFO: rcu_sched self-detected stall on CPU [ 7774.011423] rcu: 0-...!: (3008434 ticks this GP) idle=98a/1/0x4000000000000002 softirq=125154/125154 fqs=959 [ 7774.018637] (t=5820169 jiffies g=253585 q=4717) [ 7774.022324] rcu: rcu_sched kthread starved for 5815129 jiffies! g253585 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [ 7774.038284] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 7774.044784] rcu: RCU grace-period kthread stack dump: [ 7774.047823] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 7774.176739] Call Trace: [ 7774.177607] __schedule+0x351/0xcb0 [ 7774.189912] schedule+0xc0/0x180 [ 7774.196324] schedule_timeout+0xb4/0x190 [ 7774.197798] ? do_raw_spin_unlock+0x75/0x190 [ 7774.202137] ? __next_timer_interrupt+0x160/0x160 [ 7774.208559] ? prepare_to_swait_event+0xf3/0x140 [ 7774.210425] rcu_gp_kthread+0x8dd/0x15b0 [ 7774.211792] ? force_qs_rnp+0x2c0/0x2c0 [ 7774.218128] kthread+0x1d1/0x200 [ 7774.219273] ? set_kthread_struct+0x70/0x70 [ 7774.228204] ret_from_fork+0x1f/0x30 [ 7774.231451] rcu: Stack dump where RCU GP kthread last ran: [ 7774.234379] NMI backtrace for cpu 0 [ 7774.235508] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7774.250104] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7774.253899] Call Trace: [ 7774.258978] [ 7774.259691] ? dump_stack+0xbb/0x10e [ 7774.260813] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 7774.267601] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 7774.271771] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 7774.279332] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 7774.285705] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 7774.289478] ? rcu_check_gp_kthread_starvation+0x16f/0x188 [ 7774.295280] ? rcu_sched_clock_irq.cold.51+0xce/0x307 [ 7774.299384] ? update_process_times+0x6d/0xc0 [ 7774.305312] ? tick_sched_handle.isra.18+0x27/0x90 [ 7774.311077] ? tick_sched_timer+0x53/0xb0 [ 7774.312485] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7774.318823] ? tick_sched_handle.isra.18+0x90/0x90 [ 7774.321052] ? hrtimer_interrupt+0x128/0x2c0 [ 7774.328645] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7774.334779] ? apic_timer_interrupt+0xf/0x20 [ 7774.339091] [ 7774.339804] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7774.348448] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7774.358935] ? do_raw_spin_lock+0x177/0x190 [ 7774.360414] ? _raw_spin_lock+0x21/0x30 [ 7774.366336] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7774.369675] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 7774.372107] ? mdc_commit_open+0x91/0xb0 [mdc] [ 7774.380238] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7774.385520] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7774.392567] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 7774.404913] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7774.416995] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7774.425750] ? do_wait_intr+0xf0/0xf0 [ 7774.431818] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7774.440929] ? kthread+0x1d1/0x200 [ 7774.446084] ? set_kthread_struct+0x70/0x70 [ 7774.453701] ? ret_from_fork+0x1f/0x30 [ 7774.459609] NMI backtrace for cpu 0 [ 7774.461008] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7774.475355] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7774.487970] Call Trace: [ 7774.492628] [ 7774.495399] ? dump_stack+0xbb/0x10e [ 7774.501985] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 7774.513806] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 7774.515557] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 7774.521324] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 7774.527983] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 7774.534477] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 7774.539942] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 7774.544793] ? update_process_times+0x6d/0xc0 [ 7774.548227] ? tick_sched_handle.isra.18+0x27/0x90 [ 7774.555150] ? tick_sched_timer+0x53/0xb0 [ 7774.560155] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7774.565167] ? tick_sched_handle.isra.18+0x90/0x90 [ 7774.571863] ? hrtimer_interrupt+0x128/0x2c0 [ 7774.573943] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7774.577217] ? apic_timer_interrupt+0xf/0x20 [ 7774.582832] [ 7774.587702] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7774.596170] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7774.605833] ? do_raw_spin_lock+0x177/0x190 [ 7774.611316] ? _raw_spin_lock+0x21/0x30 [ 7774.616036] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7774.618699] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 7774.626232] ? mdc_commit_open+0x91/0xb0 [mdc] [ 7774.633332] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7774.644419] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7774.652821] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 7774.658311] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7774.669281] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7774.677184] ? do_wait_intr+0xf0/0xf0 [ 7774.681723] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7774.687137] ? kthread+0x1d1/0x200 [ 7774.691114] ? set_kthread_struct+0x70/0x70 [ 7774.699155] ? ret_from_fork+0x1f/0x30 [ 7776.064306] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [ 7776.074208] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7776.128070] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7776.137050] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7776.157405] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7776.159551] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7776.181745] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 7776.195589] RAX: 0000000000005685 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 7776.202300] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 7776.206817] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7776.218711] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 7776.229116] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 7776.239055] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 7776.247033] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7776.252979] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 7776.258209] Call Trace: [ 7776.259195] [ 7776.259837] ? show_regs.cold.9+0x22/0x2f [ 7776.274313] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7776.279114] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7776.282633] ? watchdog+0x40/0x40 [ 7776.284603] ? hrtimer_interrupt+0x128/0x2c0 [ 7776.291317] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7776.293711] ? apic_timer_interrupt+0xf/0x20 [ 7776.305616] [ 7776.308257] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7776.310179] do_raw_spin_lock+0x177/0x190 [ 7776.315991] _raw_spin_lock+0x21/0x30 [ 7776.320704] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7776.332789] ? do_raw_spin_unlock+0x75/0x190 [ 7776.341126] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7776.345021] ? do_wait_intr+0xf0/0xf0 [ 7776.348429] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7776.351373] mdc_close+0x53b/0xd70 [mdc] [ 7776.353789] lmv_close+0x18d/0x490 [lmv] [ 7776.358170] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 7776.364718] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7776.371666] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7776.378647] ll_md_close+0x274/0x800 [lustre] [ 7776.380432] ll_file_release+0x128/0xc50 [lustre] [ 7776.387781] __fput+0x10d/0x440 [ 7776.391131] ____fput+0x12/0x20 [ 7776.392323] task_work_run+0xc6/0x110 [ 7776.408113] exit_to_usermode_loop+0x1dd/0x1f0 [ 7776.414839] do_syscall_64+0x3d6/0x440 [ 7776.416769] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7776.423233] RIP: 0033:0x7ff54af45cd8 [ 7776.427132] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7776.456654] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7776.463149] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 7776.470090] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 7776.477901] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 7776.484193] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 7776.490035] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7784.064870] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 7784.068180] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7784.143193] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7784.156196] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7784.168629] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7784.177440] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7784.200495] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7784.208296] RAX: 0000000000004095 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 7784.215472] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 7784.220195] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7784.224375] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 7784.228977] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 7784.232912] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 7784.239002] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7784.242467] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 7784.246851] Call Trace: [ 7784.252728] [ 7784.254303] ? show_regs.cold.9+0x22/0x2f [ 7784.261185] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7784.265872] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7784.269452] ? watchdog+0x40/0x40 [ 7784.270585] ? hrtimer_interrupt+0x128/0x2c0 [ 7784.277933] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7784.282474] ? apic_timer_interrupt+0xf/0x20 [ 7784.287081] [ 7784.288181] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7784.291879] do_raw_spin_lock+0x177/0x190 [ 7784.294314] _raw_spin_lock+0x21/0x30 [ 7784.296905] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7784.304093] ? do_raw_spin_unlock+0x75/0x190 [ 7784.305497] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7784.309943] ? do_wait_intr+0xf0/0xf0 [ 7784.315467] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7784.319902] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7784.332032] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7784.337050] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7784.347199] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7784.353579] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7784.356740] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7784.365845] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7784.367765] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7784.380705] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7784.387353] lmv_intent_open+0x338/0x1150 [lmv] [ 7784.390412] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7784.399199] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7784.405399] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7784.411215] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7784.416655] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7784.418626] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7784.433477] ? do_raw_spin_unlock+0x75/0x190 [ 7784.438485] ? _raw_spin_unlock+0x12/0x30 [ 7784.442967] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7784.448472] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7784.454663] ll_atomic_open+0x36a/0xd70 [lustre] [ 7784.457397] lookup_open+0x356/0xb40 [ 7784.461360] do_last+0x313/0xfc0 [ 7784.465049] ? nd_jump_root+0xe5/0x160 [ 7784.467050] ? path_init+0x437/0x520 [ 7784.471051] path_openat+0xf7/0x500 [ 7784.473085] ? do_raw_spin_unlock+0x75/0x190 [ 7784.475437] do_filp_open+0x99/0x140 [ 7784.478070] ? getname_flags+0x6e/0x330 [ 7784.479456] ? __check_object_size+0xff/0x256 [ 7784.482009] ? do_raw_spin_unlock+0x75/0x190 [ 7784.484171] ? _raw_spin_unlock+0x12/0x30 [ 7784.486314] do_sys_openat2+0x2b4/0x410 [ 7784.488379] do_sys_open+0x73/0xa0 [ 7784.490556] __x64_sys_openat+0x24/0x30 [ 7784.493857] do_syscall_64+0xc1/0x440 [ 7784.495929] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7784.498514] RIP: 0033:0x7ff54af45332 [ 7784.500599] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7784.516457] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7784.521303] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 7784.524345] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 7784.527812] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 7784.531158] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 7784.535039] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 7788.059397] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 7788.078807] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7788.137776] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7788.151501] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7788.171937] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7788.187633] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7788.197685] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7788.234430] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7788.249311] RAX: 0000000000007007 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 7788.261278] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 7788.271477] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 7788.275298] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 7788.278503] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 7788.281627] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 7788.286116] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7788.288659] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 7788.291853] Call Trace: [ 7788.293783] [ 7788.295476] ? show_regs.cold.9+0x22/0x2f [ 7788.300777] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7788.307831] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7788.320554] ? watchdog+0x40/0x40 [ 7788.322676] ? hrtimer_interrupt+0x128/0x2c0 [ 7788.327815] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7788.331515] ? apic_timer_interrupt+0xf/0x20 [ 7788.334573] [ 7788.335582] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7788.338615] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7788.345997] do_raw_spin_lock+0x177/0x190 [ 7788.351324] _raw_spin_lock+0x21/0x30 [ 7788.352741] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7788.360316] process_one_work+0x2c8/0x700 [ 7788.363343] worker_thread+0x296/0x6e0 [ 7788.369997] ? rescuer_thread+0x570/0x570 [ 7788.371343] kthread+0x1d1/0x200 [ 7788.374657] ? set_kthread_struct+0x70/0x70 [ 7788.377697] ret_from_fork+0x1f/0x30 [ 7796.193098] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5837s! [ 7796.202015] Showing busy workqueues and worker pools: [ 7796.205695] workqueue events: flags=0x0 [ 7796.209449] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7796.216619] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 7796.225480] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7796.233506] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 7796.243831] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7796.251587] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 7796.259179] workqueue events_long: flags=0x0 [ 7796.261710] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7796.268975] pending: lu_objects_flush [obdclass] [ 7796.275981] workqueue events_unbound: flags=0x2 [ 7796.285447] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7796.294611] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7796.314043] workqueue events_power_efficient: flags=0x80 [ 7796.322454] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7796.327406] pending: neigh_periodic_work [ 7796.332011] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7796.337678] pending: do_cache_clean, check_lifetime [ 7796.345931] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7796.350966] pending: neigh_periodic_work [ 7796.356749] workqueue writeback: flags=0x4a [ 7796.363101] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7796.370083] pending: wb_workfn, wb_workfn [ 7796.374136] workqueue xprtiod: flags=0xa [ 7796.379497] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7796.383968] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 7796.393940] workqueue ipv6_addrconf: flags=0x40008 [ 7796.400071] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7796.402957] pending: addrconf_verify_work [ 7796.408462] workqueue knbd0-recv: flags=0x1a [ 7796.412675] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7796.423870] in-flight: 657:recv_work [ 7796.428603] workqueue ptlrpc_pinger: flags=0x2 [ 7796.434657] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7796.440042] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 7796.444699] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5837s workers=4 idle: 20831 12642 28859 [ 7796.453645] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7786s workers=2 idle: 659 [ 7804.058062] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [ 7804.062044] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7804.110356] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7804.128627] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7804.138233] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7804.145575] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7804.163379] RSP: 0018:ffffaf9fc1543ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7804.175214] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 7804.181662] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 7804.186014] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7804.197799] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 7804.214159] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 7804.221790] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 7804.228650] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7804.235763] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 7804.242635] Call Trace: [ 7804.244375] [ 7804.246961] ? show_regs.cold.9+0x22/0x2f [ 7804.252490] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7804.256402] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7804.263888] ? watchdog+0x40/0x40 [ 7804.267859] ? hrtimer_interrupt+0x128/0x2c0 [ 7804.269900] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7804.277315] ? apic_timer_interrupt+0xf/0x20 [ 7804.283336] [ 7804.285323] ? native_safe_halt+0x2e/0x50 [ 7804.287966] kvm_wait+0xcd/0xe0 [ 7804.292350] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7804.298758] do_raw_spin_lock+0x177/0x190 [ 7804.303284] _raw_spin_lock+0x21/0x30 [ 7804.305386] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7804.312423] ? do_raw_spin_unlock+0x75/0x190 [ 7804.317743] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7804.323545] ? do_wait_intr+0xf0/0xf0 [ 7804.327001] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7804.331087] mdc_close+0x53b/0xd70 [mdc] [ 7804.337210] lmv_close+0x18d/0x490 [lmv] [ 7804.341016] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 7804.347514] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7804.353769] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7804.361211] ll_md_close+0x274/0x800 [lustre] [ 7804.366215] ll_file_release+0x128/0xc50 [lustre] [ 7804.371508] __fput+0x10d/0x440 [ 7804.373628] ____fput+0x12/0x20 [ 7804.375351] task_work_run+0xc6/0x110 [ 7804.377236] exit_to_usermode_loop+0x1dd/0x1f0 [ 7804.384315] do_syscall_64+0x3d6/0x440 [ 7804.388424] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7804.397669] RIP: 0033:0x7ff54af45cd8 [ 7804.400096] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7804.425495] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7804.434475] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 7804.441177] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 7804.446104] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 7804.454281] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 7804.462206] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7812.060070] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 7812.068406] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7812.115350] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7812.124992] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7812.137573] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7812.139814] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7812.147645] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7812.152024] RAX: 0000000000007989 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 7812.162708] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 7812.174246] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7812.182316] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 7812.198162] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 7812.208411] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 7812.218529] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7812.226535] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 7812.239164] Call Trace: [ 7812.245248] [ 7812.247758] ? show_regs.cold.9+0x22/0x2f [ 7812.249193] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7812.261294] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7812.267699] ? watchdog+0x40/0x40 [ 7812.271307] ? hrtimer_interrupt+0x128/0x2c0 [ 7812.276074] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7812.282123] ? apic_timer_interrupt+0xf/0x20 [ 7812.290114] [ 7812.290923] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7812.303548] do_raw_spin_lock+0x177/0x190 [ 7812.308265] _raw_spin_lock+0x21/0x30 [ 7812.316844] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7812.322157] ? do_raw_spin_unlock+0x75/0x190 [ 7812.330350] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7812.336612] ? do_wait_intr+0xf0/0xf0 [ 7812.342898] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7812.350605] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7812.353419] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7812.357077] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7812.361882] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7812.373991] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7812.377533] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7812.382204] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7812.390439] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7812.394919] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7812.400329] lmv_intent_open+0x338/0x1150 [lmv] [ 7812.402015] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7812.406592] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7812.411303] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7812.414315] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7812.417745] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7812.421902] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7812.434584] ? do_raw_spin_unlock+0x75/0x190 [ 7812.437942] ? _raw_spin_unlock+0x12/0x30 [ 7812.441939] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7812.447927] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7812.457543] ll_atomic_open+0x36a/0xd70 [lustre] [ 7812.464308] lookup_open+0x356/0xb40 [ 7812.471003] do_last+0x313/0xfc0 [ 7812.476697] ? nd_jump_root+0xe5/0x160 [ 7812.482745] ? path_init+0x437/0x520 [ 7812.485267] path_openat+0xf7/0x500 [ 7812.492562] ? do_raw_spin_unlock+0x75/0x190 [ 7812.493956] do_filp_open+0x99/0x140 [ 7812.495072] ? getname_flags+0x6e/0x330 [ 7812.510213] ? __check_object_size+0xff/0x256 [ 7812.520636] ? do_raw_spin_unlock+0x75/0x190 [ 7812.522900] ? _raw_spin_unlock+0x12/0x30 [ 7812.524775] do_sys_openat2+0x2b4/0x410 [ 7812.528911] do_sys_open+0x73/0xa0 [ 7812.530148] __x64_sys_openat+0x24/0x30 [ 7812.531504] do_syscall_64+0xc1/0x440 [ 7812.539274] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7812.541034] RIP: 0033:0x7ff54af45332 [ 7812.550848] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7812.572102] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7812.579741] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 7812.588507] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 7812.601592] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 7812.610508] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 7812.615118] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 7816.059689] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 7816.074024] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7816.150343] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7816.170229] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7816.183290] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7816.192750] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7816.199344] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7816.229341] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7816.249135] RAX: 0000000000006687 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 7816.260848] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 7816.271628] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 7816.295276] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 7816.302274] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 7816.318845] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 7816.334911] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7816.352939] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 7816.369102] Call Trace: [ 7816.376331] [ 7816.380925] ? show_regs.cold.9+0x22/0x2f [ 7816.384448] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7816.391131] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7816.401413] ? watchdog+0x40/0x40 [ 7816.415095] ? hrtimer_interrupt+0x128/0x2c0 [ 7816.416378] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7816.417801] ? apic_timer_interrupt+0xf/0x20 [ 7816.431345] [ 7816.432102] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7816.439823] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7816.444760] do_raw_spin_lock+0x177/0x190 [ 7816.452906] _raw_spin_lock+0x21/0x30 [ 7816.454252] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7816.458654] process_one_work+0x2c8/0x700 [ 7816.461697] worker_thread+0x296/0x6e0 [ 7816.463765] ? rescuer_thread+0x570/0x570 [ 7816.465370] kthread+0x1d1/0x200 [ 7816.474972] ? set_kthread_struct+0x70/0x70 [ 7816.477025] ret_from_fork+0x1f/0x30 [ 7824.055061] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 7824.059280] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7824.094148] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7824.101209] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7824.105774] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7824.109793] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7824.122642] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7824.125204] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 7824.127841] RDX: 0000000000006d00 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 7824.137655] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 7824.147171] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 7824.155060] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7824.164695] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 7824.176870] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7824.193967] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 7824.196318] Call Trace: [ 7824.208086] [ 7824.211234] ? show_regs.cold.9+0x22/0x2f [ 7824.218267] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7824.228697] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7824.233133] ? watchdog+0x40/0x40 [ 7824.237158] ? hrtimer_interrupt+0x128/0x2c0 [ 7824.240288] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7824.241897] ? apic_timer_interrupt+0xf/0x20 [ 7824.246051] [ 7824.247141] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7824.252973] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7824.258550] do_raw_spin_lock+0x177/0x190 [ 7824.260013] _raw_spin_lock+0x21/0x30 [ 7824.261340] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7824.274145] obd_mod_free+0x46/0x140 [ptlrpc] [ 7824.282044] mdc_commit_open+0x91/0xb0 [mdc] [ 7824.287172] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7824.288871] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7824.290333] after_reply+0xe2c/0x1780 [ptlrpc] [ 7824.295864] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7824.301376] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7824.304029] ? do_wait_intr+0xf0/0xf0 [ 7824.305295] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7824.308393] kthread+0x1d1/0x200 [ 7824.309437] ? set_kthread_struct+0x70/0x70 [ 7824.312758] ret_from_fork+0x1f/0x30 [ 7826.912608] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5868s! [ 7826.920154] Showing busy workqueues and worker pools: [ 7826.926314] workqueue events: flags=0x0 [ 7826.935902] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7826.950873] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 7826.960780] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7826.972285] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 7826.990662] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7826.999839] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 7827.010673] workqueue events_long: flags=0x0 [ 7827.012249] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7827.019203] pending: lu_objects_flush [obdclass] [ 7827.028044] workqueue events_unbound: flags=0x2 [ 7827.032029] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7827.039732] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7827.050899] workqueue events_power_efficient: flags=0x80 [ 7827.055169] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7827.061911] pending: neigh_periodic_work [ 7827.071086] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7827.084306] pending: do_cache_clean, check_lifetime [ 7827.094721] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7827.101031] pending: neigh_periodic_work [ 7827.108559] workqueue writeback: flags=0x4a [ 7827.110022] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7827.122459] pending: wb_workfn, wb_workfn [ 7827.124019] workqueue xprtiod: flags=0xa [ 7827.137695] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7827.145200] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 7827.154291] workqueue ipv6_addrconf: flags=0x40008 [ 7827.157649] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7827.164584] pending: addrconf_verify_work [ 7827.166916] workqueue knbd0-recv: flags=0x1a [ 7827.169907] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7827.178130] in-flight: 657:recv_work [ 7827.180094] workqueue ptlrpc_pinger: flags=0x2 [ 7827.185630] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7827.191683] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 7827.196289] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5868s workers=4 idle: 20831 12642 28859 [ 7827.206242] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7817s workers=2 idle: 659 [ 7832.063349] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [ 7832.071031] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7832.097066] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7832.102950] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7832.107249] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7832.110753] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7832.120665] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7832.125939] RAX: 0000000000007915 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 7832.131397] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 7832.134958] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7832.139258] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 7832.142802] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 7832.145461] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 7832.147683] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7832.152596] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 7832.157645] Call Trace: [ 7832.159231] [ 7832.160392] ? show_regs.cold.9+0x22/0x2f [ 7832.162667] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7832.168919] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7832.175496] ? watchdog+0x40/0x40 [ 7832.179179] ? hrtimer_interrupt+0x128/0x2c0 [ 7832.184870] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7832.191968] ? apic_timer_interrupt+0xf/0x20 [ 7832.195892] [ 7832.198509] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7832.201881] do_raw_spin_lock+0x177/0x190 [ 7832.204478] _raw_spin_lock+0x21/0x30 [ 7832.206673] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7832.210190] ? do_raw_spin_unlock+0x75/0x190 [ 7832.216313] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7832.219870] ? do_wait_intr+0xf0/0xf0 [ 7832.225544] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7832.233211] mdc_close+0x53b/0xd70 [mdc] [ 7832.234750] lmv_close+0x18d/0x490 [lmv] [ 7832.237061] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 7832.239088] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7832.243016] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7832.249094] ll_md_close+0x274/0x800 [lustre] [ 7832.252431] ll_file_release+0x128/0xc50 [lustre] [ 7832.257919] __fput+0x10d/0x440 [ 7832.263190] ____fput+0x12/0x20 [ 7832.270144] task_work_run+0xc6/0x110 [ 7832.273061] exit_to_usermode_loop+0x1dd/0x1f0 [ 7832.276272] do_syscall_64+0x3d6/0x440 [ 7832.279024] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7832.287523] RIP: 0033:0x7ff54af45cd8 [ 7832.292436] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7832.312419] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7832.315043] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 7832.321094] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 7832.327611] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 7832.333230] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 7832.337812] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7840.060645] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 7840.063717] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7840.116699] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7840.126581] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7840.133362] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7840.136465] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7840.157839] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7840.167891] RAX: 0000000000003254 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 7840.174716] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 7840.182272] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7840.190081] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 7840.193553] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 7840.197954] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 7840.215253] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7840.218664] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 7840.224826] Call Trace: [ 7840.227276] [ 7840.227778] ? show_regs.cold.9+0x22/0x2f [ 7840.228723] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7840.229935] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7840.233028] ? watchdog+0x40/0x40 [ 7840.236122] ? hrtimer_interrupt+0x128/0x2c0 [ 7840.239126] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7840.241590] ? apic_timer_interrupt+0xf/0x20 [ 7840.244040] [ 7840.244901] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7840.247807] do_raw_spin_lock+0x177/0x190 [ 7840.249604] _raw_spin_lock+0x21/0x30 [ 7840.251343] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7840.254161] ? do_raw_spin_unlock+0x75/0x190 [ 7840.256516] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7840.259506] ? do_wait_intr+0xf0/0xf0 [ 7840.261679] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7840.264796] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7840.268266] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7840.271478] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7840.274537] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7840.277020] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7840.279670] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7840.282091] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7840.285150] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7840.288614] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7840.291106] lmv_intent_open+0x338/0x1150 [lmv] [ 7840.293068] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7840.295703] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7840.298627] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7840.300076] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7840.302154] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7840.304755] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7840.306921] ? do_raw_spin_unlock+0x75/0x190 [ 7840.309128] ? _raw_spin_unlock+0x12/0x30 [ 7840.310668] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7840.313117] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7840.314990] ll_atomic_open+0x36a/0xd70 [lustre] [ 7840.316580] lookup_open+0x356/0xb40 [ 7840.317651] do_last+0x313/0xfc0 [ 7840.318786] ? nd_jump_root+0xe5/0x160 [ 7840.320571] ? path_init+0x437/0x520 [ 7840.322166] path_openat+0xf7/0x500 [ 7840.323757] ? do_raw_spin_unlock+0x75/0x190 [ 7840.325668] do_filp_open+0x99/0x140 [ 7840.327345] ? getname_flags+0x6e/0x330 [ 7840.329043] ? __check_object_size+0xff/0x256 [ 7840.331118] ? do_raw_spin_unlock+0x75/0x190 [ 7840.333328] ? _raw_spin_unlock+0x12/0x30 [ 7840.335478] do_sys_openat2+0x2b4/0x410 [ 7840.337154] do_sys_open+0x73/0xa0 [ 7840.339000] __x64_sys_openat+0x24/0x30 [ 7840.340623] do_syscall_64+0xc1/0x440 [ 7840.342391] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7840.344635] RIP: 0033:0x7ff54af45332 [ 7840.346275] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7840.353758] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7840.357536] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 7840.360827] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 7840.364391] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 7840.367609] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 7840.370725] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 7844.063566] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 7844.076066] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7844.104804] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7844.111653] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7844.117277] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7844.123647] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7844.125773] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7844.135064] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7844.138127] RAX: 0000000000007f14 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 7844.141030] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 7844.143729] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 7844.151677] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 7844.155005] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 7844.156911] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 7844.164297] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7844.166407] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 7844.169072] Call Trace: [ 7844.170124] [ 7844.171139] ? show_regs.cold.9+0x22/0x2f [ 7844.172494] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7844.178534] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7844.181702] ? watchdog+0x40/0x40 [ 7844.183134] ? hrtimer_interrupt+0x128/0x2c0 [ 7844.187112] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7844.188677] ? apic_timer_interrupt+0xf/0x20 [ 7844.189851] [ 7844.194484] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7844.196961] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7844.198913] do_raw_spin_lock+0x177/0x190 [ 7844.200710] _raw_spin_lock+0x21/0x30 [ 7844.203763] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7844.205839] process_one_work+0x2c8/0x700 [ 7844.209568] worker_thread+0x296/0x6e0 [ 7844.213833] ? rescuer_thread+0x570/0x570 [ 7844.216921] kthread+0x1d1/0x200 [ 7844.218672] ? set_kthread_struct+0x70/0x70 [ 7844.220824] ret_from_fork+0x1f/0x30 [ 7852.055051] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 7852.058760] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7852.078665] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7852.082826] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7852.086638] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7852.088633] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7852.110734] RSP: 0000:ffffaf9fc1b97bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7852.119571] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 7852.125901] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 7852.130709] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 7852.135792] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 7852.141755] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7852.148925] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 7852.155570] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7852.158008] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 7852.161504] Call Trace: [ 7852.162672] [ 7852.163667] ? show_regs.cold.9+0x22/0x2f [ 7852.165538] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7852.167551] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7852.169736] ? watchdog+0x40/0x40 [ 7852.171356] ? hrtimer_interrupt+0x128/0x2c0 [ 7852.173284] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7852.174793] ? apic_timer_interrupt+0xf/0x20 [ 7852.175779] [ 7852.176516] ? native_safe_halt+0x2e/0x50 [ 7852.177779] kvm_wait+0xcd/0xe0 [ 7852.178964] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7852.180655] do_raw_spin_lock+0x177/0x190 [ 7852.181849] _raw_spin_lock+0x21/0x30 [ 7852.182870] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7852.184320] obd_mod_free+0x46/0x140 [ptlrpc] [ 7852.185538] mdc_commit_open+0x91/0xb0 [mdc] [ 7852.186757] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7852.188248] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7852.189605] after_reply+0xe2c/0x1780 [ptlrpc] [ 7852.190972] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7852.192609] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7852.194685] ? do_wait_intr+0xf0/0xf0 [ 7852.196356] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7852.198788] kthread+0x1d1/0x200 [ 7852.199944] ? set_kthread_struct+0x70/0x70 [ 7852.201276] ret_from_fork+0x1f/0x30 [ 7857.631065] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5898s! [ 7857.634483] Showing busy workqueues and worker pools: [ 7857.636797] workqueue events: flags=0x0 [ 7857.638631] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7857.641823] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 7857.645876] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7857.648510] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 7857.653560] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7857.655881] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 7857.658704] workqueue events_long: flags=0x0 [ 7857.660430] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7857.662415] pending: lu_objects_flush [obdclass] [ 7857.664380] workqueue events_unbound: flags=0x2 [ 7857.666246] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7857.668868] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7857.673403] workqueue events_power_efficient: flags=0x80 [ 7857.675817] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7857.678557] pending: neigh_periodic_work [ 7857.680437] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7857.683396] pending: do_cache_clean, check_lifetime [ 7857.685848] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7857.688876] pending: neigh_periodic_work [ 7857.691093] workqueue writeback: flags=0x4a [ 7857.693267] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7857.696408] pending: wb_workfn, wb_workfn [ 7857.698570] workqueue xprtiod: flags=0xa [ 7857.700742] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7857.703723] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 7857.707846] workqueue ipv6_addrconf: flags=0x40008 [ 7857.710012] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7857.712887] pending: addrconf_verify_work [ 7857.715052] workqueue knbd0-recv: flags=0x1a [ 7857.717016] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7857.719909] in-flight: 657:recv_work [ 7857.721920] workqueue ptlrpc_pinger: flags=0x2 [ 7857.723821] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7857.726526] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 7857.729452] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5898s workers=4 idle: 20831 12642 28859 [ 7857.731977] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7847s workers=2 idle: 659 [ 7860.058065] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [ 7860.064666] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7860.090826] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7860.097475] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7860.101406] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7860.104555] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7860.112983] RSP: 0018:ffffaf9fc1543ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7860.115745] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 7860.118718] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 7860.121611] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7860.125284] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 7860.128234] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 7860.131353] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 7860.134417] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7860.136509] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 7860.140204] Call Trace: [ 7860.141118] [ 7860.141988] ? show_regs.cold.9+0x22/0x2f [ 7860.143515] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7860.145607] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7860.147235] ? watchdog+0x40/0x40 [ 7860.149783] ? hrtimer_interrupt+0x128/0x2c0 [ 7860.154391] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7860.160781] ? apic_timer_interrupt+0xf/0x20 [ 7860.163449] [ 7860.165235] ? native_safe_halt+0x2e/0x50 [ 7860.171334] kvm_wait+0xcd/0xe0 [ 7860.173494] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7860.176228] do_raw_spin_lock+0x177/0x190 [ 7860.178155] _raw_spin_lock+0x21/0x30 [ 7860.187906] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7860.195487] ? do_raw_spin_unlock+0x75/0x190 [ 7860.199994] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7860.208268] ? do_wait_intr+0xf0/0xf0 [ 7860.220379] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7860.223550] mdc_close+0x53b/0xd70 [mdc] [ 7860.235294] lmv_close+0x18d/0x490 [lmv] [ 7860.242563] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 7860.253968] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7860.261439] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7860.263039] ll_md_close+0x274/0x800 [lustre] [ 7860.269594] ll_file_release+0x128/0xc50 [lustre] [ 7860.273470] __fput+0x10d/0x440 [ 7860.277494] ____fput+0x12/0x20 [ 7860.278965] task_work_run+0xc6/0x110 [ 7860.284643] exit_to_usermode_loop+0x1dd/0x1f0 [ 7860.288704] do_syscall_64+0x3d6/0x440 [ 7860.292696] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7860.298309] RIP: 0033:0x7ff54af45cd8 [ 7860.302741] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7860.320307] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7860.327647] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 7860.335411] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 7860.348705] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 7860.351151] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 7860.359994] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7868.061185] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 7868.070308] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7868.115520] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7868.125619] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7868.135365] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7868.141739] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7868.161095] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7868.171379] RAX: 000000000000708d RBX: ffff96848811cae0 RCX: 0000000000000001 [ 7868.177331] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 7868.186335] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7868.194976] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 7868.201512] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 7868.208402] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 7868.217487] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7868.222981] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 7868.229162] Call Trace: [ 7868.231946] [ 7868.232658] ? show_regs.cold.9+0x22/0x2f [ 7868.236391] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7868.240968] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7868.243202] ? watchdog+0x40/0x40 [ 7868.248209] ? hrtimer_interrupt+0x128/0x2c0 [ 7868.249741] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7868.255017] ? apic_timer_interrupt+0xf/0x20 [ 7868.259038] [ 7868.261215] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7868.265249] do_raw_spin_lock+0x177/0x190 [ 7868.270588] _raw_spin_lock+0x21/0x30 [ 7868.272224] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7868.278608] ? do_raw_spin_unlock+0x75/0x190 [ 7868.281807] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7868.289582] ? do_wait_intr+0xf0/0xf0 [ 7868.291329] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7868.295071] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7868.301548] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7868.303708] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7868.309211] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7868.312397] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7868.316282] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7868.318619] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7868.323337] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7868.329396] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7868.333260] lmv_intent_open+0x338/0x1150 [lmv] [ 7868.338264] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7868.342618] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7868.347384] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7868.353039] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7868.355295] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7868.360520] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7868.365311] ? do_raw_spin_unlock+0x75/0x190 [ 7868.368393] ? _raw_spin_unlock+0x12/0x30 [ 7868.374548] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7868.376725] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7868.379349] ll_atomic_open+0x36a/0xd70 [lustre] [ 7868.384293] lookup_open+0x356/0xb40 [ 7868.387482] do_last+0x313/0xfc0 [ 7868.389064] ? nd_jump_root+0xe5/0x160 [ 7868.392688] ? path_init+0x437/0x520 [ 7868.396614] path_openat+0xf7/0x500 [ 7868.401366] ? do_raw_spin_unlock+0x75/0x190 [ 7868.405707] do_filp_open+0x99/0x140 [ 7868.409196] ? getname_flags+0x6e/0x330 [ 7868.411366] ? __check_object_size+0xff/0x256 [ 7868.413647] ? do_raw_spin_unlock+0x75/0x190 [ 7868.415931] ? _raw_spin_unlock+0x12/0x30 [ 7868.418375] do_sys_openat2+0x2b4/0x410 [ 7868.420316] do_sys_open+0x73/0xa0 [ 7868.422153] __x64_sys_openat+0x24/0x30 [ 7868.425128] do_syscall_64+0xc1/0x440 [ 7868.427604] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7868.430286] RIP: 0033:0x7ff54af45332 [ 7868.433569] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7868.455921] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7868.464687] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 7868.469501] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 7868.475832] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 7868.485117] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 7868.493829] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 7872.059031] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 7872.061425] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7872.115497] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7872.133842] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7872.147690] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7872.169361] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7872.176700] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7872.210852] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7872.221139] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 7872.231597] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 7872.240802] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 7872.251595] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 7872.253809] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 7872.265582] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 7872.267972] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7872.273566] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 7872.281275] Call Trace: [ 7872.284309] [ 7872.285419] ? show_regs.cold.9+0x22/0x2f [ 7872.287827] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7872.292224] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7872.295504] ? watchdog+0x40/0x40 [ 7872.297512] ? hrtimer_interrupt+0x128/0x2c0 [ 7872.301805] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7872.303265] ? apic_timer_interrupt+0xf/0x20 [ 7872.305717] [ 7872.307542] ? native_safe_halt+0x2e/0x50 [ 7872.310366] kvm_wait+0xcd/0xe0 [ 7872.312732] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7872.315580] do_raw_spin_lock+0x177/0x190 [ 7872.317524] _raw_spin_lock+0x21/0x30 [ 7872.324196] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7872.325965] process_one_work+0x2c8/0x700 [ 7872.329499] worker_thread+0x296/0x6e0 [ 7872.333423] ? rescuer_thread+0x570/0x570 [ 7872.336254] kthread+0x1d1/0x200 [ 7872.337278] ? set_kthread_struct+0x70/0x70 [ 7872.341423] ret_from_fork+0x1f/0x30 [ 7884.055394] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 7884.060989] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7884.111503] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7884.129012] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7884.139973] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7884.145543] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7884.163414] RSP: 0000:ffffaf9fc1b97bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7884.175855] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 7884.189770] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 7884.197447] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 7884.206956] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 7884.215975] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7884.223520] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 7884.231499] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7884.237270] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 7884.243297] Call Trace: [ 7884.245231] [ 7884.250315] ? show_regs.cold.9+0x22/0x2f [ 7884.260898] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7884.263881] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7884.266863] ? watchdog+0x40/0x40 [ 7884.268906] ? hrtimer_interrupt+0x128/0x2c0 [ 7884.271930] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7884.276110] ? apic_timer_interrupt+0xf/0x20 [ 7884.286439] [ 7884.289445] ? native_safe_halt+0x2e/0x50 [ 7884.294450] kvm_wait+0xcd/0xe0 [ 7884.296486] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7884.299928] do_raw_spin_lock+0x177/0x190 [ 7884.302744] _raw_spin_lock+0x21/0x30 [ 7884.306006] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7884.315245] obd_mod_free+0x46/0x140 [ptlrpc] [ 7884.322613] mdc_commit_open+0x91/0xb0 [mdc] [ 7884.328480] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7884.333923] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7884.338006] after_reply+0xe2c/0x1780 [ptlrpc] [ 7884.342163] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7884.345858] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7884.361182] ? do_wait_intr+0xf0/0xf0 [ 7884.370793] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7884.387566] kthread+0x1d1/0x200 [ 7884.392261] ? set_kthread_struct+0x70/0x70 [ 7884.398677] ret_from_fork+0x1f/0x30 [ 7887.839098] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5928s! [ 7887.843088] Showing busy workqueues and worker pools: [ 7887.845617] workqueue events: flags=0x0 [ 7887.862567] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7887.881897] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 7887.890609] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7887.895555] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 7887.904963] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7887.908667] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 7887.917553] workqueue events_long: flags=0x0 [ 7887.921663] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7887.927527] pending: lu_objects_flush [obdclass] [ 7887.933445] workqueue events_unbound: flags=0x2 [ 7887.936213] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7887.944782] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7887.951910] workqueue events_power_efficient: flags=0x80 [ 7887.954992] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7887.962278] pending: neigh_periodic_work [ 7887.968971] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7887.976021] pending: do_cache_clean, check_lifetime [ 7887.984495] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7887.989806] pending: neigh_periodic_work [ 7887.993486] workqueue writeback: flags=0x4a [ 7887.999698] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7888.006476] pending: wb_workfn, wb_workfn [ 7888.008512] workqueue xprtiod: flags=0xa [ 7888.017310] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7888.026719] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 7888.033291] workqueue ipv6_addrconf: flags=0x40008 [ 7888.039297] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7888.046703] pending: addrconf_verify_work [ 7888.049715] workqueue knbd0-recv: flags=0x1a [ 7888.052756] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7888.058050] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [ 7888.059844] in-flight: [ 7888.075495] Modules linked in: [ 7888.075497] 657:recv_work [ 7888.077786] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7888.077855] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7888.077862] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7888.077864] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7888.077877] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7888.077881] RSP: 0018:ffffaf9fc1543ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7888.077885] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 7888.077887] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 7888.077889] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7888.077892] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 7888.077894] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 7888.077896] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 7888.077900] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7888.077902] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 7888.078039] Call Trace: [ 7888.078047] [ 7888.078053] ? show_regs.cold.9+0x22/0x2f [ 7888.078324] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7888.078333] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7888.078337] ? watchdog+0x40/0x40 [ 7888.078342] ? hrtimer_interrupt+0x128/0x2c0 [ 7888.078346] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7888.078352] ? apic_timer_interrupt+0xf/0x20 [ 7888.078357] [ 7888.078360] ? native_safe_halt+0x2e/0x50 [ 7888.078364] kvm_wait+0xcd/0xe0 [ 7888.078371] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7888.078380] do_raw_spin_lock+0x177/0x190 [ 7888.078385] _raw_spin_lock+0x21/0x30 [ 7888.078389] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7888.125710] [ 7888.126265] workqueue ptlrpc_pinger: flags=0x2 [ 7888.126273] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7888.126283] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 7888.126561] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5929s workers=4 idle: 20831 12642 28859 [ 7888.126575] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7878s workers=2 idle: 659 [ 7888.284186] ? do_raw_spin_unlock+0x75/0x190 [ 7888.287720] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7888.295020] ? do_wait_intr+0xf0/0xf0 [ 7888.300721] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7888.306719] mdc_close+0x53b/0xd70 [mdc] [ 7888.309168] lmv_close+0x18d/0x490 [lmv] [ 7888.311745] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 7888.315169] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7888.319733] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7888.324152] ll_md_close+0x274/0x800 [lustre] [ 7888.326625] ll_file_release+0x128/0xc50 [lustre] [ 7888.329660] __fput+0x10d/0x440 [ 7888.331558] ____fput+0x12/0x20 [ 7888.333455] task_work_run+0xc6/0x110 [ 7888.336128] exit_to_usermode_loop+0x1dd/0x1f0 [ 7888.339484] do_syscall_64+0x3d6/0x440 [ 7888.348327] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7888.353620] RIP: 0033:0x7ff54af45cd8 [ 7888.358215] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7888.377373] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7888.385567] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 7888.398376] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 7888.406596] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 7888.411190] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 7888.420415] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7896.060025] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 7896.063270] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7896.133775] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7896.144554] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7896.165551] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7896.172852] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7896.201481] RSP: 0000:ffffaf9fc15234f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7896.219454] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 7896.238568] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 7896.252351] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7896.264522] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 7896.288749] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 7896.311364] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 7896.326736] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7896.340865] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 7896.360789] Call Trace: [ 7896.367495] [ 7896.370389] ? show_regs.cold.9+0x22/0x2f [ 7896.374425] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7896.377268] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7896.379180] ? watchdog+0x40/0x40 [ 7896.384190] ? hrtimer_interrupt+0x128/0x2c0 [ 7896.389538] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7896.398140] ? apic_timer_interrupt+0xf/0x20 [ 7896.400572] [ 7896.403053] ? native_safe_halt+0x2e/0x50 [ 7896.410452] kvm_wait+0xcd/0xe0 [ 7896.416094] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7896.422415] do_raw_spin_lock+0x177/0x190 [ 7896.426347] _raw_spin_lock+0x21/0x30 [ 7896.429705] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7896.438153] ? do_raw_spin_unlock+0x75/0x190 [ 7896.444119] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7896.450947] ? do_wait_intr+0xf0/0xf0 [ 7896.457406] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7896.469068] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7896.481369] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7896.506027] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7896.514335] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7896.519551] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7896.522331] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7896.524359] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7896.528173] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7896.538505] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7896.546799] lmv_intent_open+0x338/0x1150 [lmv] [ 7896.556427] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7896.558001] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7896.566873] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7896.572797] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7896.577109] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7896.582926] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7896.590562] ? do_raw_spin_unlock+0x75/0x190 [ 7896.597560] ? _raw_spin_unlock+0x12/0x30 [ 7896.601444] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7896.602657] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7896.608093] ll_atomic_open+0x36a/0xd70 [lustre] [ 7896.611491] lookup_open+0x356/0xb40 [ 7896.613920] do_last+0x313/0xfc0 [ 7896.614843] ? nd_jump_root+0xe5/0x160 [ 7896.616216] ? path_init+0x437/0x520 [ 7896.623629] path_openat+0xf7/0x500 [ 7896.625446] ? do_raw_spin_unlock+0x75/0x190 [ 7896.627726] do_filp_open+0x99/0x140 [ 7896.629870] ? getname_flags+0x6e/0x330 [ 7896.631169] ? __check_object_size+0xff/0x256 [ 7896.632323] ? do_raw_spin_unlock+0x75/0x190 [ 7896.633761] ? _raw_spin_unlock+0x12/0x30 [ 7896.635379] do_sys_openat2+0x2b4/0x410 [ 7896.639109] do_sys_open+0x73/0xa0 [ 7896.641191] __x64_sys_openat+0x24/0x30 [ 7896.643183] do_syscall_64+0xc1/0x440 [ 7896.644598] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7896.646981] RIP: 0033:0x7ff54af45332 [ 7896.647992] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7896.654850] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7896.657057] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 7896.659082] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 7896.661300] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 7896.663369] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 7896.665621] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 7900.059995] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 7900.070137] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7900.099333] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7900.105413] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7900.113145] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7900.117291] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7900.120900] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7900.133897] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7900.137429] RAX: 0000000000007c84 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 7900.141573] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 7900.144983] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 7900.150261] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 7900.153105] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 7900.156569] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 7900.162293] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7900.165182] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 7900.168481] Call Trace: [ 7900.170152] [ 7900.171171] ? show_regs.cold.9+0x22/0x2f [ 7900.173116] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7900.176152] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7900.178358] ? watchdog+0x40/0x40 [ 7900.180204] ? hrtimer_interrupt+0x128/0x2c0 [ 7900.183075] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7900.186774] ? apic_timer_interrupt+0xf/0x20 [ 7900.189394] [ 7900.191155] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7900.193360] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7900.196516] do_raw_spin_lock+0x177/0x190 [ 7900.198521] _raw_spin_lock+0x21/0x30 [ 7900.200322] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7900.203309] process_one_work+0x2c8/0x700 [ 7900.205231] worker_thread+0x296/0x6e0 [ 7900.207121] ? rescuer_thread+0x570/0x570 [ 7900.209341] kthread+0x1d1/0x200 [ 7900.211282] ? set_kthread_struct+0x70/0x70 [ 7900.214065] ret_from_fork+0x1f/0x30 [ 7916.055039] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 7916.058230] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 7916.060219] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7916.060295] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7916.060303] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7916.060306] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7916.060321] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7916.060326] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7916.060330] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 7916.060332] RDX: 00000000000075fb RSI: 0000000000000003 RDI: ffff96848811cae0 [ 7916.060334] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 7916.060336] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 7916.060338] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7916.060342] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 7916.060346] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7916.060348] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 7916.060362] Call Trace: [ 7916.060369] [ 7916.060376] ? show_regs.cold.9+0x22/0x2f [ 7916.060579] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7916.060587] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7916.060591] ? watchdog+0x40/0x40 [ 7916.060596] ? hrtimer_interrupt+0x128/0x2c0 [ 7916.060600] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7916.060609] ? apic_timer_interrupt+0xf/0x20 [ 7916.060614] [ 7916.060617] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7916.060622] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7916.060628] do_raw_spin_lock+0x177/0x190 [ 7916.060633] _raw_spin_lock+0x21/0x30 [ 7916.060668] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7916.060964] obd_mod_free+0x46/0x140 [ptlrpc] [ 7916.061161] mdc_commit_open+0x91/0xb0 [mdc] [ 7916.061197] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7916.069688] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7916.069765] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7916.069774] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7916.069777] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7916.069791] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7916.069795] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7916.069799] RAX: 0000000000007d15 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 7916.069802] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 7916.069804] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7916.069806] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 7916.069808] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 7916.069811] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 7916.069814] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7916.069817] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 7916.069827] Call Trace: [ 7916.069832] [ 7916.069838] ? show_regs.cold.9+0x22/0x2f [ 7916.069848] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7916.069855] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7916.069859] ? watchdog+0x40/0x40 [ 7916.069863] ? hrtimer_interrupt+0x128/0x2c0 [ 7916.069866] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7916.069873] ? apic_timer_interrupt+0xf/0x20 [ 7916.069878] [ 7916.069881] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7916.069888] do_raw_spin_lock+0x177/0x190 [ 7916.069893] _raw_spin_lock+0x21/0x30 [ 7916.069898] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7916.070186] ? do_raw_spin_unlock+0x75/0x190 [ 7916.070193] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7916.071012] ? do_wait_intr+0xf0/0xf0 [ 7916.071020] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7916.071227] mdc_close+0x53b/0xd70 [mdc] [ 7916.071267] lmv_close+0x18d/0x490 [lmv] [ 7916.071298] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 7916.094670] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7916.094983] after_reply+0xe2c/0x1780 [ptlrpc] [ 7916.095329] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7916.095619] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7916.095832] ? do_wait_intr+0xf0/0xf0 [ 7916.095843] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7916.096043] kthread+0x1d1/0x200 [ 7916.096049] ? set_kthread_struct+0x70/0x70 [ 7916.096053] ret_from_fork+0x1f/0x30 [ 7916.471455] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7916.474903] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7916.478160] ll_md_close+0x274/0x800 [lustre] [ 7916.481887] ll_file_release+0x128/0xc50 [lustre] [ 7916.483549] __fput+0x10d/0x440 [ 7916.487024] ____fput+0x12/0x20 [ 7916.489366] task_work_run+0xc6/0x110 [ 7916.491730] exit_to_usermode_loop+0x1dd/0x1f0 [ 7916.494675] do_syscall_64+0x3d6/0x440 [ 7916.502739] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7916.504530] RIP: 0033:0x7ff54af45cd8 [ 7916.508930] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7916.519010] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7916.522400] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 7916.526792] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 7916.530469] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 7916.534437] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 7916.538420] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7918.565508] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5959s! [ 7918.573131] Showing busy workqueues and worker pools: [ 7918.575856] workqueue events: flags=0x0 [ 7918.582862] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7918.589865] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 7918.597251] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7918.602863] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 7918.617185] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7918.623406] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 7918.633465] workqueue events_long: flags=0x0 [ 7918.641776] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7918.650133] pending: lu_objects_flush [obdclass] [ 7918.658403] workqueue events_unbound: flags=0x2 [ 7918.664597] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7918.672673] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7918.690696] workqueue events_power_efficient: flags=0x80 [ 7918.698387] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7918.718089] pending: neigh_periodic_work [ 7918.725815] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7918.729211] pending: do_cache_clean, check_lifetime [ 7918.738277] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7918.743841] pending: neigh_periodic_work [ 7918.745468] workqueue writeback: flags=0x4a [ 7918.756104] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7918.765639] pending: wb_workfn, wb_workfn [ 7918.768739] workqueue xprtiod: flags=0xa [ 7918.775941] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7918.783251] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 7918.801908] workqueue ipv6_addrconf: flags=0x40008 [ 7918.814533] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7918.826685] pending: addrconf_verify_work [ 7918.839626] workqueue knbd0-recv: flags=0x1a [ 7918.850633] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7918.860690] in-flight: 657:recv_work [ 7918.868467] workqueue ptlrpc_pinger: flags=0x2 [ 7918.883199] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7918.898929] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 7918.914813] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5960s workers=4 idle: 20831 12642 28859 [ 7918.937186] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7909s workers=2 idle: 659 [ 7924.060013] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 7924.063777] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7924.096811] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7924.110716] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7924.115833] RIP: 0010:__pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 7924.118757] Code: d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 05 18 d4 dc 02 01 e0 48 83 05 06 d4 dc 02 01 48 83 05 be d5 dc 02 01 8b 45 08 85 [ 7924.134365] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7924.138027] RAX: 00000000000033b7 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 7924.143031] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 7924.146703] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7924.150312] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 7924.153555] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 7924.158663] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 7924.164047] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7924.167595] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 7924.172754] Call Trace: [ 7924.174230] [ 7924.179472] ? show_regs.cold.9+0x22/0x2f [ 7924.181742] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7924.187736] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7924.192466] ? watchdog+0x40/0x40 [ 7924.194578] ? hrtimer_interrupt+0x128/0x2c0 [ 7924.196770] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7924.200733] ? apic_timer_interrupt+0xf/0x20 [ 7924.203532] [ 7924.204704] ? __pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 7924.207784] do_raw_spin_lock+0x177/0x190 [ 7924.210222] _raw_spin_lock+0x21/0x30 [ 7924.211823] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7924.214751] ? do_raw_spin_unlock+0x75/0x190 [ 7924.217240] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7924.219813] ? do_wait_intr+0xf0/0xf0 [ 7924.221980] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7924.224616] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7924.227393] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7924.230650] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7924.241354] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7924.255101] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7924.262255] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7924.264393] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7924.268104] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7924.274965] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7924.280819] lmv_intent_open+0x338/0x1150 [lmv] [ 7924.286757] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7924.294241] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7924.300014] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7924.306466] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7924.311806] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7924.316069] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7924.319018] ? do_raw_spin_unlock+0x75/0x190 [ 7924.321505] ? _raw_spin_unlock+0x12/0x30 [ 7924.326279] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7924.335731] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7924.341745] ll_atomic_open+0x36a/0xd70 [lustre] [ 7924.348507] lookup_open+0x356/0xb40 [ 7924.349620] do_last+0x313/0xfc0 [ 7924.352518] ? nd_jump_root+0xe5/0x160 [ 7924.354831] ? path_init+0x437/0x520 [ 7924.356785] path_openat+0xf7/0x500 [ 7924.359574] ? do_raw_spin_unlock+0x75/0x190 [ 7924.364754] do_filp_open+0x99/0x140 [ 7924.366906] ? getname_flags+0x6e/0x330 [ 7924.374690] ? __check_object_size+0xff/0x256 [ 7924.376028] ? do_raw_spin_unlock+0x75/0x190 [ 7924.377459] ? _raw_spin_unlock+0x12/0x30 [ 7924.382497] do_sys_openat2+0x2b4/0x410 [ 7924.384665] do_sys_open+0x73/0xa0 [ 7924.386831] __x64_sys_openat+0x24/0x30 [ 7924.390698] do_syscall_64+0xc1/0x440 [ 7924.392329] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7924.398115] RIP: 0033:0x7ff54af45332 [ 7924.401805] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7924.429039] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7924.433961] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 7924.443228] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 7924.448793] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 7924.461325] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 7924.468008] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 7928.059126] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 7928.063087] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7928.109140] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7928.122273] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7928.135936] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7928.141929] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7928.148998] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7928.160995] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7928.164410] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 7928.167356] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 7928.170092] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 7928.173032] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 7928.175848] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 7928.178456] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 7928.181407] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7928.183405] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 7928.186037] Call Trace: [ 7928.187034] [ 7928.187847] ? show_regs.cold.9+0x22/0x2f [ 7928.189300] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7928.191005] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7928.192596] ? watchdog+0x40/0x40 [ 7928.193914] ? hrtimer_interrupt+0x128/0x2c0 [ 7928.195543] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7928.197259] ? apic_timer_interrupt+0xf/0x20 [ 7928.198881] [ 7928.199611] ? native_safe_halt+0x2e/0x50 [ 7928.201069] kvm_wait+0xcd/0xe0 [ 7928.202274] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7928.204310] do_raw_spin_lock+0x177/0x190 [ 7928.205904] _raw_spin_lock+0x21/0x30 [ 7928.207285] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7928.209027] process_one_work+0x2c8/0x700 [ 7928.210504] worker_thread+0x296/0x6e0 [ 7928.211952] ? rescuer_thread+0x570/0x570 [ 7928.213449] kthread+0x1d1/0x200 [ 7928.214646] ? set_kthread_struct+0x70/0x70 [ 7928.216127] ret_from_fork+0x1f/0x30 [ 7944.055245] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 7944.058275] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7944.059578] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 7944.080064] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7944.080079] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7944.080082] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7944.080096] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7944.080101] RSP: 0000:ffffaf9fc1b97bb8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 7944.080107] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 7944.080109] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 7944.080111] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 7944.080114] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 7944.080116] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7944.080119] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 7944.080122] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7944.080124] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 7944.085726] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7944.085814] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7944.085822] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7944.085825] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7944.085840] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7944.085844] RSP: 0018:ffffaf9fc1543ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7944.085849] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 7944.085851] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 7944.085854] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7944.085856] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 7944.085859] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 7944.085867] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 7944.085870] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7944.085872] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 7944.085879] Call Trace: [ 7944.085885] [ 7944.085891] ? show_regs.cold.9+0x22/0x2f [ 7944.085902] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7944.085909] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7944.085912] ? watchdog+0x40/0x40 [ 7944.085917] ? hrtimer_interrupt+0x128/0x2c0 [ 7944.085920] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7944.085928] ? apic_timer_interrupt+0xf/0x20 [ 7944.085932] [ 7944.085935] ? native_safe_halt+0x2e/0x50 [ 7944.085939] kvm_wait+0xcd/0xe0 [ 7944.085947] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7944.085955] do_raw_spin_lock+0x177/0x190 [ 7944.085960] _raw_spin_lock+0x21/0x30 [ 7944.085964] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7944.087462] ? do_raw_spin_unlock+0x75/0x190 [ 7944.087471] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7944.087668] ? do_wait_intr+0xf0/0xf0 [ 7944.087677] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7944.087874] mdc_close+0x53b/0xd70 [mdc] [ 7944.087919] lmv_close+0x18d/0x490 [lmv] [ 7944.087956] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 7944.088124] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7944.110976] Call Trace: [ 7944.110990] [ 7944.110996] ? show_regs.cold.9+0x22/0x2f [ 7944.111010] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7944.111017] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7944.111022] ? watchdog+0x40/0x40 [ 7944.111027] ? hrtimer_interrupt+0x128/0x2c0 [ 7944.111031] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7944.111039] ? apic_timer_interrupt+0xf/0x20 [ 7944.111044] [ 7944.111048] ? native_safe_halt+0x2e/0x50 [ 7944.111055] kvm_wait+0xcd/0xe0 [ 7944.111064] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7944.111071] do_raw_spin_lock+0x177/0x190 [ 7944.111075] _raw_spin_lock+0x21/0x30 [ 7944.111082] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7944.111353] obd_mod_free+0x46/0x140 [ptlrpc] [ 7944.111559] mdc_commit_open+0x91/0xb0 [mdc] [ 7944.111601] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7944.111794] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7944.111990] after_reply+0xe2c/0x1780 [ptlrpc] [ 7944.112185] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7944.112393] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7944.112594] ? do_wait_intr+0xf0/0xf0 [ 7944.112605] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7944.112805] kthread+0x1d1/0x200 [ 7944.112810] ? set_kthread_struct+0x70/0x70 [ 7944.112814] ret_from_fork+0x1f/0x30 [ 7944.493986] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7944.496470] ll_md_close+0x274/0x800 [lustre] [ 7944.498937] ll_file_release+0x128/0xc50 [lustre] [ 7944.501513] __fput+0x10d/0x440 [ 7944.502475] ____fput+0x12/0x20 [ 7944.503358] task_work_run+0xc6/0x110 [ 7944.505109] exit_to_usermode_loop+0x1dd/0x1f0 [ 7944.507548] do_syscall_64+0x3d6/0x440 [ 7944.509579] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7944.516900] RIP: 0033:0x7ff54af45cd8 [ 7944.521656] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7944.545066] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7944.552602] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 7944.559451] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 7944.571175] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 7944.577563] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 7944.586530] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7949.279032] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5990s! [ 7949.282504] Showing busy workqueues and worker pools: [ 7949.284820] workqueue events: flags=0x0 [ 7949.286285] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7949.289384] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 7949.294512] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7949.298929] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 7949.305445] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7949.310098] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 7949.315433] workqueue events_long: flags=0x0 [ 7949.316886] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7949.320436] pending: lu_objects_flush [obdclass] [ 7949.322538] workqueue events_unbound: flags=0x2 [ 7949.323918] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7949.326382] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7949.337333] workqueue events_power_efficient: flags=0x80 [ 7949.341391] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7949.350846] pending: neigh_periodic_work [ 7949.352055] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7949.354530] pending: do_cache_clean, check_lifetime [ 7949.360663] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7949.364509] pending: neigh_periodic_work [ 7949.366111] workqueue writeback: flags=0x4a [ 7949.368138] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7949.370587] pending: wb_workfn, wb_workfn [ 7949.371750] workqueue xprtiod: flags=0xa [ 7949.373013] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7949.375486] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 7949.382757] workqueue ipv6_addrconf: flags=0x40008 [ 7949.385092] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7949.387904] pending: addrconf_verify_work [ 7949.389658] workqueue knbd0-recv: flags=0x1a [ 7949.391538] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7949.394031] in-flight: 657:recv_work [ 7949.395854] workqueue ptlrpc_pinger: flags=0x2 [ 7949.397775] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7949.399959] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 7949.402488] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5990s workers=4 idle: 20831 12642 28859 [ 7949.405570] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7939s workers=2 idle: 659 [ 7952.067246] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 7952.081343] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7952.133431] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7952.149924] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7952.160675] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7952.161997] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7952.174407] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7952.180146] RAX: 0000000000000f97 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 7952.187431] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 7952.194942] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7952.213626] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 7952.225150] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 7952.240692] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 7952.242943] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7952.244263] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 7952.263941] Call Trace: [ 7952.265180] [ 7952.266385] ? show_regs.cold.9+0x22/0x2f [ 7952.267961] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7952.269101] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7952.270195] ? watchdog+0x40/0x40 [ 7952.270928] ? hrtimer_interrupt+0x128/0x2c0 [ 7952.281071] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7952.284106] ? apic_timer_interrupt+0xf/0x20 [ 7952.290635] [ 7952.291156] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7952.293872] do_raw_spin_lock+0x177/0x190 [ 7952.296363] _raw_spin_lock+0x21/0x30 [ 7952.299408] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7952.302628] ? do_raw_spin_unlock+0x75/0x190 [ 7952.304727] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7952.307004] ? do_wait_intr+0xf0/0xf0 [ 7952.312684] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7952.314175] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7952.317352] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7952.319697] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7952.322969] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7952.324709] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7952.325998] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7952.327639] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7952.329313] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7952.331076] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7952.332712] lmv_intent_open+0x338/0x1150 [lmv] [ 7952.334929] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7952.336946] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7952.338637] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7952.340860] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7952.343510] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7952.344805] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7952.345893] ? do_raw_spin_unlock+0x75/0x190 [ 7952.346722] ? _raw_spin_unlock+0x12/0x30 [ 7952.348005] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7952.349627] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7952.350911] ll_atomic_open+0x36a/0xd70 [lustre] [ 7952.353417] lookup_open+0x356/0xb40 [ 7952.354211] do_last+0x313/0xfc0 [ 7952.354994] ? nd_jump_root+0xe5/0x160 [ 7952.355983] ? path_init+0x437/0x520 [ 7952.356822] path_openat+0xf7/0x500 [ 7952.357863] ? do_raw_spin_unlock+0x75/0x190 [ 7952.359181] do_filp_open+0x99/0x140 [ 7952.361281] ? getname_flags+0x6e/0x330 [ 7952.362394] ? __check_object_size+0xff/0x256 [ 7952.363862] ? do_raw_spin_unlock+0x75/0x190 [ 7952.365976] ? _raw_spin_unlock+0x12/0x30 [ 7952.368788] do_sys_openat2+0x2b4/0x410 [ 7952.371650] do_sys_open+0x73/0xa0 [ 7952.372520] __x64_sys_openat+0x24/0x30 [ 7952.373389] do_syscall_64+0xc1/0x440 [ 7952.374207] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7952.376211] RIP: 0033:0x7ff54af45332 [ 7952.377557] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7952.383631] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7952.387463] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 7952.394847] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 7952.403345] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 7952.411641] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 7952.413276] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 7954.012030] rcu: INFO: rcu_sched self-detected stall on CPU [ 7954.015511] rcu: 0-...!: (3094378 ticks this GP) idle=98a/1/0x4000000000000002 softirq=125154/125154 fqs=959 [ 7954.031302] (t=6000172 jiffies g=253585 q=4792) [ 7954.039811] rcu: rcu_sched kthread starved for 5995132 jiffies! g253585 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [ 7954.046959] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 7954.053514] rcu: RCU grace-period kthread stack dump: [ 7954.055580] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 7954.058578] Call Trace: [ 7954.059562] __schedule+0x351/0xcb0 [ 7954.061107] schedule+0xc0/0x180 [ 7954.062465] schedule_timeout+0xb4/0x190 [ 7954.064218] ? do_raw_spin_unlock+0x75/0x190 [ 7954.066130] ? __next_timer_interrupt+0x160/0x160 [ 7954.068143] ? prepare_to_swait_event+0xf3/0x140 [ 7954.070279] rcu_gp_kthread+0x8dd/0x15b0 [ 7954.071651] ? force_qs_rnp+0x2c0/0x2c0 [ 7954.073306] kthread+0x1d1/0x200 [ 7954.074247] ? set_kthread_struct+0x70/0x70 [ 7954.075390] ret_from_fork+0x1f/0x30 [ 7954.076353] rcu: Stack dump where RCU GP kthread last ran: [ 7954.078667] NMI backtrace for cpu 0 [ 7954.080440] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7954.085234] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7954.089017] Call Trace: [ 7954.090171] [ 7954.091102] ? dump_stack+0xbb/0x10e [ 7954.093096] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 7954.094803] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 7954.096773] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 7954.098681] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 7954.100019] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 7954.101632] ? rcu_check_gp_kthread_starvation+0x16f/0x188 [ 7954.103531] ? rcu_sched_clock_irq.cold.51+0xce/0x307 [ 7954.104766] ? update_process_times+0x6d/0xc0 [ 7954.105869] ? tick_sched_handle.isra.18+0x27/0x90 [ 7954.107450] ? tick_sched_timer+0x53/0xb0 [ 7954.110064] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7954.113543] ? tick_sched_handle.isra.18+0x90/0x90 [ 7954.117684] ? hrtimer_interrupt+0x128/0x2c0 [ 7954.125777] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7954.130002] ? apic_timer_interrupt+0xf/0x20 [ 7954.135478] [ 7954.136425] ? native_safe_halt+0x2e/0x50 [ 7954.138016] ? kvm_wait+0xcd/0xe0 [ 7954.139677] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7954.145215] ? do_raw_spin_lock+0x177/0x190 [ 7954.146696] ? _raw_spin_lock+0x21/0x30 [ 7954.152284] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7954.159484] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 7954.161284] ? mdc_commit_open+0x91/0xb0 [mdc] [ 7954.164872] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7954.167510] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7954.172418] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 7954.177670] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7954.181092] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7954.188042] ? do_wait_intr+0xf0/0xf0 [ 7954.189880] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7954.192836] ? kthread+0x1d1/0x200 [ 7954.196773] ? set_kthread_struct+0x70/0x70 [ 7954.199063] ? ret_from_fork+0x1f/0x30 [ 7954.200617] NMI backtrace for cpu 0 [ 7954.202038] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7954.207190] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7954.215840] Call Trace: [ 7954.218872] [ 7954.221862] ? dump_stack+0xbb/0x10e [ 7954.228353] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 7954.233033] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 7954.239799] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 7954.244004] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 7954.245784] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 7954.254458] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 7954.259878] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 7954.261783] ? update_process_times+0x6d/0xc0 [ 7954.266227] ? tick_sched_handle.isra.18+0x27/0x90 [ 7954.267817] ? tick_sched_timer+0x53/0xb0 [ 7954.270783] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7954.275375] ? tick_sched_handle.isra.18+0x90/0x90 [ 7954.277588] ? hrtimer_interrupt+0x128/0x2c0 [ 7954.279970] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7954.283336] ? apic_timer_interrupt+0xf/0x20 [ 7954.284777] [ 7954.285380] ? native_safe_halt+0x2e/0x50 [ 7954.286323] ? kvm_wait+0xcd/0xe0 [ 7954.287115] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7954.293180] ? do_raw_spin_lock+0x177/0x190 [ 7954.300620] ? _raw_spin_lock+0x21/0x30 [ 7954.303641] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7954.305371] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 7954.310435] ? mdc_commit_open+0x91/0xb0 [mdc] [ 7954.314711] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7954.320624] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7954.326021] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 7954.327490] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7954.333180] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7954.334962] ? do_wait_intr+0xf0/0xf0 [ 7954.340983] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7954.343212] ? kthread+0x1d1/0x200 [ 7954.344468] ? set_kthread_struct+0x70/0x70 [ 7954.345965] ? ret_from_fork+0x1f/0x30 [ 7956.066071] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 7956.085307] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7956.139563] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7956.157105] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7956.162935] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7956.166486] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7956.175191] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7956.202494] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7956.208745] RAX: 0000000000007e84 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 7956.214367] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 7956.222013] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 7956.226520] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 7956.234824] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 7956.243290] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 7956.248362] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7956.253789] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 7956.260429] Call Trace: [ 7956.262784] [ 7956.264054] ? show_regs.cold.9+0x22/0x2f [ 7956.278053] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7956.287352] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7956.300339] ? watchdog+0x40/0x40 [ 7956.306728] ? hrtimer_interrupt+0x128/0x2c0 [ 7956.309014] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7956.316808] ? apic_timer_interrupt+0xf/0x20 [ 7956.319696] [ 7956.323207] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7956.325970] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7956.329223] do_raw_spin_lock+0x177/0x190 [ 7956.331410] _raw_spin_lock+0x21/0x30 [ 7956.333492] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7956.336882] process_one_work+0x2c8/0x700 [ 7956.339270] worker_thread+0x296/0x6e0 [ 7956.341376] ? rescuer_thread+0x570/0x570 [ 7956.343925] kthread+0x1d1/0x200 [ 7956.346232] ? set_kthread_struct+0x70/0x70 [ 7956.349655] ret_from_fork+0x1f/0x30 [ 7972.059490] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 7972.063261] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7972.098145] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7972.110072] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7972.115728] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7972.123088] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7972.134215] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7972.144484] RAX: 0000000000007e9b RBX: ffff96848811cae0 RCX: 0000000000000001 [ 7972.149651] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 7972.154537] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7972.158374] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 7972.162439] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 7972.173898] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 7972.178247] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7972.181407] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 7972.185131] Call Trace: [ 7972.185941] [ 7972.186894] ? show_regs.cold.9+0x22/0x2f [ 7972.189777] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7972.193118] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7972.195652] ? watchdog+0x40/0x40 [ 7972.197915] ? hrtimer_interrupt+0x128/0x2c0 [ 7972.200467] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7972.204719] ? apic_timer_interrupt+0xf/0x20 [ 7972.208879] [ 7972.210002] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7972.216938] do_raw_spin_lock+0x177/0x190 [ 7972.224983] _raw_spin_lock+0x21/0x30 [ 7972.228106] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7972.232169] ? do_raw_spin_unlock+0x75/0x190 [ 7972.234516] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7972.241151] ? do_wait_intr+0xf0/0xf0 [ 7972.244270] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7972.249033] mdc_close+0x53b/0xd70 [mdc] [ 7972.250472] lmv_close+0x18d/0x490 [lmv] [ 7972.251633] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 7972.259127] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7972.263991] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7972.265439] ll_md_close+0x274/0x800 [lustre] [ 7972.269216] ll_file_release+0x128/0xc50 [lustre] [ 7972.273854] __fput+0x10d/0x440 [ 7972.276761] ____fput+0x12/0x20 [ 7972.277889] task_work_run+0xc6/0x110 [ 7972.280789] exit_to_usermode_loop+0x1dd/0x1f0 [ 7972.284258] do_syscall_64+0x3d6/0x440 [ 7972.287838] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7972.291853] RIP: 0033:0x7ff54af45cd8 [ 7972.294816] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7972.301825] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7972.304476] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 7972.306785] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 7972.309528] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 7972.312560] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 7972.319603] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7979.488578] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6020s! [ 7979.494681] Showing busy workqueues and worker pools: [ 7979.497078] workqueue events: flags=0x0 [ 7979.502034] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7979.513946] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 7979.519352] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7979.530226] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 7979.538608] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7979.546190] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 7979.558463] workqueue events_long: flags=0x0 [ 7979.568281] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7979.570666] pending: lu_objects_flush [obdclass] [ 7979.581802] workqueue events_unbound: flags=0x2 [ 7979.587834] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7979.590158] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7979.607251] workqueue events_power_efficient: flags=0x80 [ 7979.610013] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7979.612587] pending: neigh_periodic_work [ 7979.622342] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7979.632069] pending: do_cache_clean, check_lifetime [ 7979.633915] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7979.646094] pending: neigh_periodic_work [ 7979.649832] workqueue writeback: flags=0x4a [ 7979.659055] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7979.663374] pending: wb_workfn, wb_workfn [ 7979.668650] workqueue xprtiod: flags=0xa [ 7979.674492] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7979.681264] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 7979.696016] workqueue ipv6_addrconf: flags=0x40008 [ 7979.700304] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7979.707255] pending: addrconf_verify_work [ 7979.709534] workqueue knbd0-recv: flags=0x1a [ 7979.714647] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7979.721105] in-flight: 657:recv_work [ 7979.723579] workqueue ptlrpc_pinger: flags=0x2 [ 7979.725455] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7979.732917] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 7979.740497] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6020s workers=4 idle: 20831 12642 28859 [ 7979.748625] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7969s workers=2 idle: 659 [ 7980.060044] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [ 7980.063792] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7980.111527] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7980.129300] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7980.139471] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7980.148819] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7980.166260] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7980.175595] RAX: 0000000000005483 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 7980.178489] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 7980.181822] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7980.183841] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 7980.186639] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 7980.192563] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 7980.197117] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7980.198327] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 7980.201365] Call Trace: [ 7980.205682] [ 7980.208196] ? show_regs.cold.9+0x22/0x2f [ 7980.211808] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7980.217109] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7980.220785] ? watchdog+0x40/0x40 [ 7980.223623] ? hrtimer_interrupt+0x128/0x2c0 [ 7980.225450] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7980.227586] ? apic_timer_interrupt+0xf/0x20 [ 7980.229126] [ 7980.230103] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7980.232470] do_raw_spin_lock+0x177/0x190 [ 7980.235648] _raw_spin_lock+0x21/0x30 [ 7980.237355] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7980.240346] ? do_raw_spin_unlock+0x75/0x190 [ 7980.242656] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7980.245275] ? do_wait_intr+0xf0/0xf0 [ 7980.246222] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7980.247837] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7980.252435] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7980.256795] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7980.261776] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7980.266337] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7980.267615] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7980.270428] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7980.274160] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7980.276810] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7980.279719] lmv_intent_open+0x338/0x1150 [lmv] [ 7980.282146] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7980.286523] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7980.289229] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7980.291346] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7980.292954] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7980.296574] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7980.300332] ? do_raw_spin_unlock+0x75/0x190 [ 7980.302736] ? _raw_spin_unlock+0x12/0x30 [ 7980.305181] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7980.308488] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7980.312946] ll_atomic_open+0x36a/0xd70 [lustre] [ 7980.314622] lookup_open+0x356/0xb40 [ 7980.317413] do_last+0x313/0xfc0 [ 7980.318534] ? nd_jump_root+0xe5/0x160 [ 7980.319664] ? path_init+0x437/0x520 [ 7980.322839] path_openat+0xf7/0x500 [ 7980.326545] ? do_raw_spin_unlock+0x75/0x190 [ 7980.328752] do_filp_open+0x99/0x140 [ 7980.331039] ? getname_flags+0x6e/0x330 [ 7980.333008] ? __check_object_size+0xff/0x256 [ 7980.334811] ? do_raw_spin_unlock+0x75/0x190 [ 7980.337088] ? _raw_spin_unlock+0x12/0x30 [ 7980.338983] do_sys_openat2+0x2b4/0x410 [ 7980.340689] do_sys_open+0x73/0xa0 [ 7980.344135] __x64_sys_openat+0x24/0x30 [ 7980.346712] do_syscall_64+0xc1/0x440 [ 7980.348536] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7980.352302] RIP: 0033:0x7ff54af45332 [ 7980.354118] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7980.368721] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7980.375751] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 7980.379941] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 7980.387065] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 7980.397745] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 7980.402393] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 7984.060888] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 7984.073374] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7984.110074] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7984.117384] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7984.122370] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7984.126229] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7984.129883] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7984.141556] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7984.148486] RAX: 000000000000140f RBX: ffff96848811cae0 RCX: 0000000000000001 [ 7984.153944] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 7984.164452] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 7984.168747] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 7984.174429] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 7984.179676] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 7984.183413] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7984.186214] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 7984.190171] Call Trace: [ 7984.191551] [ 7984.196943] ? show_regs.cold.9+0x22/0x2f [ 7984.198218] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7984.201536] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7984.204240] ? watchdog+0x40/0x40 [ 7984.206221] ? hrtimer_interrupt+0x128/0x2c0 [ 7984.215233] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7984.223434] ? apic_timer_interrupt+0xf/0x20 [ 7984.225047] [ 7984.226210] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7984.229511] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7984.232776] do_raw_spin_lock+0x177/0x190 [ 7984.235272] _raw_spin_lock+0x21/0x30 [ 7984.238216] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7984.241460] process_one_work+0x2c8/0x700 [ 7984.243253] worker_thread+0x296/0x6e0 [ 7984.245025] ? rescuer_thread+0x570/0x570 [ 7984.246545] kthread+0x1d1/0x200 [ 7984.248036] ? set_kthread_struct+0x70/0x70 [ 7984.250079] ret_from_fork+0x1f/0x30 [ 8000.058789] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 8000.065586] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8000.104417] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8000.115418] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8000.131056] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8000.138003] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8000.162906] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 8000.171993] RAX: 0000000000007984 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 8000.181490] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 8000.188334] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8000.194538] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 8000.198364] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 8000.204119] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 8000.208305] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8000.218757] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 8000.228996] Call Trace: [ 8000.232794] [ 8000.235677] ? show_regs.cold.9+0x22/0x2f [ 8000.237893] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8000.241249] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8000.251500] ? watchdog+0x40/0x40 [ 8000.254403] ? hrtimer_interrupt+0x128/0x2c0 [ 8000.260590] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8000.273560] ? apic_timer_interrupt+0xf/0x20 [ 8000.278710] [ 8000.279659] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8000.281809] do_raw_spin_lock+0x177/0x190 [ 8000.287612] _raw_spin_lock+0x21/0x30 [ 8000.291543] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8000.295257] ? do_raw_spin_unlock+0x75/0x190 [ 8000.299608] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8000.306618] ? do_wait_intr+0xf0/0xf0 [ 8000.313805] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8000.327031] mdc_close+0x53b/0xd70 [mdc] [ 8000.331768] lmv_close+0x18d/0x490 [lmv] [ 8000.336023] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 8000.348003] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8000.361731] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8000.377391] ll_md_close+0x274/0x800 [lustre] [ 8000.383980] ll_file_release+0x128/0xc50 [lustre] [ 8000.393288] __fput+0x10d/0x440 [ 8000.401734] ____fput+0x12/0x20 [ 8000.405060] task_work_run+0xc6/0x110 [ 8000.408362] exit_to_usermode_loop+0x1dd/0x1f0 [ 8000.412734] do_syscall_64+0x3d6/0x440 [ 8000.414393] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8000.417333] RIP: 0033:0x7ff54af45cd8 [ 8000.419899] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8000.428383] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8000.432179] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 8000.438967] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 8000.444162] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 8000.447006] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 8000.450003] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8004.055023] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 8004.057162] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8004.107846] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8004.113595] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8004.120343] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8004.123530] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8004.131827] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8004.135294] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 8004.142058] RDX: 0000000000006f20 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 8004.148830] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 8004.157915] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 8004.163188] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8004.171692] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 8004.175340] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8004.177500] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 8004.185004] Call Trace: [ 8004.186531] [ 8004.189739] ? show_regs.cold.9+0x22/0x2f [ 8004.191341] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8004.193797] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8004.196178] ? watchdog+0x40/0x40 [ 8004.197578] ? hrtimer_interrupt+0x128/0x2c0 [ 8004.200087] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8004.202484] ? apic_timer_interrupt+0xf/0x20 [ 8004.204720] [ 8004.206144] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8004.208952] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8004.211752] do_raw_spin_lock+0x177/0x190 [ 8004.213470] _raw_spin_lock+0x21/0x30 [ 8004.218334] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8004.220497] obd_mod_free+0x46/0x140 [ptlrpc] [ 8004.227318] mdc_commit_open+0x91/0xb0 [mdc] [ 8004.233890] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8004.238267] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8004.240653] after_reply+0xe2c/0x1780 [ptlrpc] [ 8004.246239] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8004.249230] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8004.255126] ? do_wait_intr+0xf0/0xf0 [ 8004.258659] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8004.269270] kthread+0x1d1/0x200 [ 8004.270611] ? set_kthread_struct+0x70/0x70 [ 8004.278394] ret_from_fork+0x1f/0x30 [ 8008.060803] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [ 8008.071280] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8008.119546] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8008.132865] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8008.141495] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8008.145319] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8008.169423] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8008.180710] RAX: 0000000000007f10 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 8008.188873] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 8008.201514] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8008.209841] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 8008.222304] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 8008.231517] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 8008.237844] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8008.242526] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 8008.247656] Call Trace: [ 8008.249569] [ 8008.250755] ? show_regs.cold.9+0x22/0x2f [ 8008.252712] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8008.255476] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8008.258524] ? watchdog+0x40/0x40 [ 8008.260327] ? hrtimer_interrupt+0x128/0x2c0 [ 8008.262954] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8008.265341] ? apic_timer_interrupt+0xf/0x20 [ 8008.267500] [ 8008.268650] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8008.271702] do_raw_spin_lock+0x177/0x190 [ 8008.275540] _raw_spin_lock+0x21/0x30 [ 8008.281790] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8008.290630] ? do_raw_spin_unlock+0x75/0x190 [ 8008.300606] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8008.315821] ? do_wait_intr+0xf0/0xf0 [ 8008.317112] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8008.331368] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8008.339176] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8008.363554] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8008.385520] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8008.395444] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8008.402662] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8008.413093] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8008.421004] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8008.433041] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8008.442431] lmv_intent_open+0x338/0x1150 [lmv] [ 8008.452860] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8008.459120] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8008.461462] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8008.473706] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8008.482203] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8008.491020] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8008.500796] ? do_raw_spin_unlock+0x75/0x190 [ 8008.508922] ? _raw_spin_unlock+0x12/0x30 [ 8008.517858] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8008.524294] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8008.532322] ll_atomic_open+0x36a/0xd70 [lustre] [ 8008.533833] lookup_open+0x356/0xb40 [ 8008.540432] do_last+0x313/0xfc0 [ 8008.542717] ? nd_jump_root+0xe5/0x160 [ 8008.548503] ? path_init+0x437/0x520 [ 8008.550345] path_openat+0xf7/0x500 [ 8008.553194] ? do_raw_spin_unlock+0x75/0x190 [ 8008.559100] do_filp_open+0x99/0x140 [ 8008.560642] ? getname_flags+0x6e/0x330 [ 8008.563202] ? __check_object_size+0xff/0x256 [ 8008.566292] ? do_raw_spin_unlock+0x75/0x190 [ 8008.569340] ? _raw_spin_unlock+0x12/0x30 [ 8008.576387] do_sys_openat2+0x2b4/0x410 [ 8008.578391] do_sys_open+0x73/0xa0 [ 8008.580476] __x64_sys_openat+0x24/0x30 [ 8008.582686] do_syscall_64+0xc1/0x440 [ 8008.584274] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8008.590297] RIP: 0033:0x7ff54af45332 [ 8008.593865] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8008.609159] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8008.618730] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 8008.623478] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 8008.629464] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 8008.636231] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 8008.639520] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 8010.209451] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6051s! [ 8010.221481] Showing busy workqueues and worker pools: [ 8010.230074] workqueue events: flags=0x0 [ 8010.235485] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8010.242377] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 8010.254444] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8010.258926] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 8010.273472] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8010.286268] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 8010.298202] workqueue events_long: flags=0x0 [ 8010.308061] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8010.320073] pending: lu_objects_flush [obdclass] [ 8010.326968] workqueue events_unbound: flags=0x2 [ 8010.328182] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8010.340372] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8010.351071] workqueue events_power_efficient: flags=0x80 [ 8010.363954] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8010.378378] pending: neigh_periodic_work [ 8010.379953] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8010.399985] pending: do_cache_clean, check_lifetime [ 8010.401247] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8010.418483] pending: neigh_periodic_work [ 8010.425702] workqueue writeback: flags=0x4a [ 8010.430534] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8010.446414] pending: wb_workfn, wb_workfn [ 8010.456824] workqueue xprtiod: flags=0xa [ 8010.459593] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8010.475474] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 8010.492549] workqueue ipv6_addrconf: flags=0x40008 [ 8010.498762] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8010.508155] pending: addrconf_verify_work [ 8010.515534] workqueue knbd0-recv: flags=0x1a [ 8010.524227] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8010.537006] in-flight: 657:recv_work [ 8010.550255] workqueue ptlrpc_pinger: flags=0x2 [ 8010.551994] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8010.561689] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 8010.571464] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6051s workers=4 idle: 20831 12642 28859 [ 8010.577510] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8000s workers=2 idle: 659 [ 8012.059188] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 8012.074289] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8012.123760] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8012.133146] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8012.140945] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8012.146072] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8012.152412] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8012.170176] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8012.175242] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 8012.179920] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 8012.184860] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 8012.190457] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 8012.198338] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 8012.207069] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 8012.215238] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8012.222175] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 8012.228665] Call Trace: [ 8012.230131] [ 8012.232210] ? show_regs.cold.9+0x22/0x2f [ 8012.234845] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8012.239970] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8012.243169] ? watchdog+0x40/0x40 [ 8012.245784] ? hrtimer_interrupt+0x128/0x2c0 [ 8012.248106] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8012.251339] ? apic_timer_interrupt+0xf/0x20 [ 8012.254448] [ 8012.255895] ? native_safe_halt+0x2e/0x50 [ 8012.258785] kvm_wait+0xcd/0xe0 [ 8012.260172] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8012.262390] do_raw_spin_lock+0x177/0x190 [ 8012.264746] _raw_spin_lock+0x21/0x30 [ 8012.267184] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8012.270899] process_one_work+0x2c8/0x700 [ 8012.273077] worker_thread+0x296/0x6e0 [ 8012.275271] ? rescuer_thread+0x570/0x570 [ 8012.277802] kthread+0x1d1/0x200 [ 8012.279747] ? set_kthread_struct+0x70/0x70 [ 8012.282187] ret_from_fork+0x1f/0x30 [ 8028.058145] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 8028.063510] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8028.110138] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8028.116936] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8028.125061] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8028.128939] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8028.139731] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8028.143992] RAX: 0000000000006d8b RBX: ffff96848811cae0 RCX: 0000000000000001 [ 8028.148261] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 8028.155405] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8028.164368] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 8028.175237] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 8028.183564] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 8028.193642] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8028.204157] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 8028.212320] Call Trace: [ 8028.229652] [ 8028.231613] ? show_regs.cold.9+0x22/0x2f [ 8028.238156] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8028.249956] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8028.258611] ? watchdog+0x40/0x40 [ 8028.263325] ? hrtimer_interrupt+0x128/0x2c0 [ 8028.271739] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8028.278974] ? apic_timer_interrupt+0xf/0x20 [ 8028.283079] [ 8028.287119] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8028.291872] do_raw_spin_lock+0x177/0x190 [ 8028.295911] _raw_spin_lock+0x21/0x30 [ 8028.299928] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8028.306699] ? do_raw_spin_unlock+0x75/0x190 [ 8028.308207] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8028.312117] ? do_wait_intr+0xf0/0xf0 [ 8028.316280] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8028.321859] mdc_close+0x53b/0xd70 [mdc] [ 8028.324951] lmv_close+0x18d/0x490 [lmv] [ 8028.328254] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 8028.334370] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8028.339439] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8028.341816] ll_md_close+0x274/0x800 [lustre] [ 8028.347574] ll_file_release+0x128/0xc50 [lustre] [ 8028.353626] __fput+0x10d/0x440 [ 8028.356775] ____fput+0x12/0x20 [ 8028.358221] task_work_run+0xc6/0x110 [ 8028.360704] exit_to_usermode_loop+0x1dd/0x1f0 [ 8028.363513] do_syscall_64+0x3d6/0x440 [ 8028.364784] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8028.369992] RIP: 0033:0x7ff54af45cd8 [ 8028.375599] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8028.399602] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8028.407056] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 8028.412049] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 8028.416135] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 8028.422232] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 8028.423790] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8036.055026] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [ 8036.058083] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8036.060053] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [ 8036.071947] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8036.076733] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8036.076817] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8036.076825] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8036.076828] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8036.076841] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8036.076846] RSP: 0000:ffffaf9fc15234f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8036.076851] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 8036.076853] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 8036.076856] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8036.076858] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 8036.076861] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 8036.076864] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 8036.076867] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8036.076869] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 8036.076876] Call Trace: [ 8036.076881] [ 8036.076887] ? show_regs.cold.9+0x22/0x2f [ 8036.076956] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8036.076961] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8036.076964] ? watchdog+0x40/0x40 [ 8036.076967] ? hrtimer_interrupt+0x128/0x2c0 [ 8036.076969] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8036.076975] ? apic_timer_interrupt+0xf/0x20 [ 8036.076978] [ 8036.076980] ? native_safe_halt+0x2e/0x50 [ 8036.076983] kvm_wait+0xcd/0xe0 [ 8036.076989] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8036.076997] do_raw_spin_lock+0x177/0x190 [ 8036.077001] _raw_spin_lock+0x21/0x30 [ 8036.077006] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8036.077237] ? do_raw_spin_unlock+0x75/0x190 [ 8036.077243] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8036.077429] ? do_wait_intr+0xf0/0xf0 [ 8036.077438] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8036.077897] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8036.078133] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8036.078404] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8036.078537] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8036.078580] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8036.078607] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8036.078645] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8036.078742] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8036.078975] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8036.079008] lmv_intent_open+0x338/0x1150 [lmv] [ 8036.079041] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8036.079149] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8036.079239] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8036.079261] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8036.079345] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8036.079430] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8036.079524] ? do_raw_spin_unlock+0x75/0x190 [ 8036.079532] ? _raw_spin_unlock+0x12/0x30 [ 8036.079539] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8036.079566] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8036.079586] ll_atomic_open+0x36a/0xd70 [lustre] [ 8036.079675] lookup_open+0x356/0xb40 [ 8036.079685] do_last+0x313/0xfc0 [ 8036.079690] ? nd_jump_root+0xe5/0x160 [ 8036.079695] ? path_init+0x437/0x520 [ 8036.079700] path_openat+0xf7/0x500 [ 8036.079705] ? do_raw_spin_unlock+0x75/0x190 [ 8036.079710] do_filp_open+0x99/0x140 [ 8036.079717] ? getname_flags+0x6e/0x330 [ 8036.079721] ? __check_object_size+0xff/0x256 [ 8036.079730] ? do_raw_spin_unlock+0x75/0x190 [ 8036.079734] ? _raw_spin_unlock+0x12/0x30 [ 8036.079739] do_sys_openat2+0x2b4/0x410 [ 8036.079744] do_sys_open+0x73/0xa0 [ 8036.079749] __x64_sys_openat+0x24/0x30 [ 8036.079753] do_syscall_64+0xc1/0x440 [ 8036.080261] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8036.080270] RIP: 0033:0x7ff54af45332 [ 8036.080278] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8036.080283] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8036.080288] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 8036.080291] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 8036.080293] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 8036.080295] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 8036.080298] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 8036.310415] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8036.313518] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8036.315460] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8036.336667] RSP: 0000:ffffaf9fc1b97bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8036.345547] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 8036.354425] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 8036.356846] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 8036.360699] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 8036.363419] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8036.368157] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 8036.375503] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8036.379129] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 8036.383936] Call Trace: [ 8036.385309] [ 8036.386450] ? show_regs.cold.9+0x22/0x2f [ 8036.389905] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8036.392907] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8036.395441] ? watchdog+0x40/0x40 [ 8036.397535] ? hrtimer_interrupt+0x128/0x2c0 [ 8036.400396] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8036.405221] ? apic_timer_interrupt+0xf/0x20 [ 8036.411103] [ 8036.418154] ? native_safe_halt+0x2e/0x50 [ 8036.423656] kvm_wait+0xcd/0xe0 [ 8036.426094] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8036.430560] do_raw_spin_lock+0x177/0x190 [ 8036.431998] _raw_spin_lock+0x21/0x30 [ 8036.436523] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8036.446114] obd_mod_free+0x46/0x140 [ptlrpc] [ 8036.450735] mdc_commit_open+0x91/0xb0 [mdc] [ 8036.456867] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8036.463085] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8036.468080] after_reply+0xe2c/0x1780 [ptlrpc] [ 8036.472026] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8036.478089] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8036.481032] ? do_wait_intr+0xf0/0xf0 [ 8036.484738] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8036.489220] kthread+0x1d1/0x200 [ 8036.490703] ? set_kthread_struct+0x70/0x70 [ 8036.494034] ret_from_fork+0x1f/0x30 [ 8040.059401] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 8040.070001] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8040.111831] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8040.122037] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8040.127359] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8040.135021] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8040.139038] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8040.150224] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8040.160924] RAX: 0000000000007e9d RBX: ffff96848811cae0 RCX: 0000000000000001 [ 8040.173140] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 8040.177980] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 8040.184431] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 8040.191904] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 8040.203527] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 8040.213893] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8040.221219] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 8040.225720] Call Trace: [ 8040.227359] [ 8040.229111] ? show_regs.cold.9+0x22/0x2f [ 8040.244405] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8040.247202] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8040.253059] ? watchdog+0x40/0x40 [ 8040.255518] ? hrtimer_interrupt+0x128/0x2c0 [ 8040.258723] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8040.262427] ? apic_timer_interrupt+0xf/0x20 [ 8040.265620] [ 8040.266747] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8040.274409] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8040.282492] do_raw_spin_lock+0x177/0x190 [ 8040.287335] _raw_spin_lock+0x21/0x30 [ 8040.289303] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8040.298606] process_one_work+0x2c8/0x700 [ 8040.303211] worker_thread+0x296/0x6e0 [ 8040.306209] ? rescuer_thread+0x570/0x570 [ 8040.311458] kthread+0x1d1/0x200 [ 8040.317196] ? set_kthread_struct+0x70/0x70 [ 8040.323692] ret_from_fork+0x1f/0x30 [ 8040.928573] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6082s! [ 8040.939692] Showing busy workqueues and worker pools: [ 8040.952147] workqueue events: flags=0x0 [ 8040.955140] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8040.966882] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 8040.993677] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8041.015840] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 8041.030523] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8041.040512] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 8041.049076] workqueue events_long: flags=0x0 [ 8041.063769] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8041.079582] pending: lu_objects_flush [obdclass] [ 8041.085390] workqueue events_unbound: flags=0x2 [ 8041.095357] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8041.102637] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8041.111741] workqueue events_power_efficient: flags=0x80 [ 8041.115073] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8041.119022] pending: neigh_periodic_work [ 8041.121416] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8041.126590] pending: do_cache_clean, check_lifetime [ 8041.131720] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8041.136358] pending: neigh_periodic_work [ 8041.142048] workqueue writeback: flags=0x4a [ 8041.146520] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8041.152621] pending: wb_workfn, wb_workfn [ 8041.154732] workqueue xprtiod: flags=0xa [ 8041.157409] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8041.164478] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 8041.171684] workqueue ipv6_addrconf: flags=0x40008 [ 8041.177537] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8041.184204] pending: addrconf_verify_work [ 8041.192035] workqueue knbd0-recv: flags=0x1a [ 8041.198727] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8041.202255] in-flight: 657:recv_work [ 8041.211082] workqueue ptlrpc_pinger: flags=0x2 [ 8041.215046] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8041.220711] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 8041.226538] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6082s workers=4 idle: 20831 12642 28859 [ 8041.231538] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8031s workers=2 idle: 659 [ 8056.067581] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 8056.079712] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8056.114511] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8056.125304] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8056.133608] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8056.139934] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8056.155225] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 8056.157941] RAX: 0000000000004783 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 8056.171040] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 8056.176297] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8056.187431] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 8056.191978] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 8056.198543] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 8056.202491] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8056.207587] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 8056.213000] Call Trace: [ 8056.213880] [ 8056.215479] ? show_regs.cold.9+0x22/0x2f [ 8056.222144] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8056.227510] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8056.233486] ? watchdog+0x40/0x40 [ 8056.239353] ? hrtimer_interrupt+0x128/0x2c0 [ 8056.240669] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8056.249931] ? apic_timer_interrupt+0xf/0x20 [ 8056.257562] [ 8056.259519] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8056.272028] do_raw_spin_lock+0x177/0x190 [ 8056.278372] _raw_spin_lock+0x21/0x30 [ 8056.281521] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8056.286630] ? do_raw_spin_unlock+0x75/0x190 [ 8056.291569] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8056.300241] ? do_wait_intr+0xf0/0xf0 [ 8056.302534] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8056.305566] mdc_close+0x53b/0xd70 [mdc] [ 8056.309602] lmv_close+0x18d/0x490 [lmv] [ 8056.316612] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 8056.328369] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8056.337444] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8056.340678] ll_md_close+0x274/0x800 [lustre] [ 8056.344674] ll_file_release+0x128/0xc50 [lustre] [ 8056.351416] __fput+0x10d/0x440 [ 8056.353003] ____fput+0x12/0x20 [ 8056.354734] task_work_run+0xc6/0x110 [ 8056.360223] exit_to_usermode_loop+0x1dd/0x1f0 [ 8056.369300] do_syscall_64+0x3d6/0x440 [ 8056.371260] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8056.379452] RIP: 0033:0x7ff54af45cd8 [ 8056.384349] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8056.397400] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8056.406413] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 8056.408560] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 8056.414376] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 8056.417786] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 8056.421106] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8064.061239] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [ 8064.067624] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8064.120581] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8064.131325] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8064.139957] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8064.145791] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8064.166185] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8064.173895] RAX: 0000000000006e8f RBX: ffff96848811cae0 RCX: 0000000000000001 [ 8064.178191] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 8064.180725] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8064.184431] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 8064.188096] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 8064.191451] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 8064.195091] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8064.197536] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 8064.200484] Call Trace: [ 8064.201790] [ 8064.202517] ? show_regs.cold.9+0x22/0x2f [ 8064.203918] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8064.206298] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8064.214393] ? watchdog+0x40/0x40 [ 8064.216851] ? hrtimer_interrupt+0x128/0x2c0 [ 8064.221060] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8064.223247] ? apic_timer_interrupt+0xf/0x20 [ 8064.229616] [ 8064.232516] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8064.237517] do_raw_spin_lock+0x177/0x190 [ 8064.240560] _raw_spin_lock+0x21/0x30 [ 8064.242955] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8064.246830] ? do_raw_spin_unlock+0x75/0x190 [ 8064.252993] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8064.260880] ? do_wait_intr+0xf0/0xf0 [ 8064.265603] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8064.272369] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8064.277580] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8064.282800] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8064.285390] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8064.289564] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8064.293018] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8064.295044] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8064.297981] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8064.301979] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8064.305499] lmv_intent_open+0x338/0x1150 [lmv] [ 8064.308092] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8064.311948] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8064.317118] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8064.320300] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8064.322578] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8064.325458] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8064.328497] ? do_raw_spin_unlock+0x75/0x190 [ 8064.331407] ? _raw_spin_unlock+0x12/0x30 [ 8064.333972] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8064.338210] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8064.340959] ll_atomic_open+0x36a/0xd70 [lustre] [ 8064.343860] lookup_open+0x356/0xb40 [ 8064.346105] do_last+0x313/0xfc0 [ 8064.347984] ? nd_jump_root+0xe5/0x160 [ 8064.350117] ? path_init+0x437/0x520 [ 8064.354459] path_openat+0xf7/0x500 [ 8064.356497] ? do_raw_spin_unlock+0x75/0x190 [ 8064.358578] do_filp_open+0x99/0x140 [ 8064.360603] ? getname_flags+0x6e/0x330 [ 8064.362955] ? __check_object_size+0xff/0x256 [ 8064.365567] ? do_raw_spin_unlock+0x75/0x190 [ 8064.367876] ? _raw_spin_unlock+0x12/0x30 [ 8064.370249] do_sys_openat2+0x2b4/0x410 [ 8064.371957] do_sys_open+0x73/0xa0 [ 8064.374264] __x64_sys_openat+0x24/0x30 [ 8064.376433] do_syscall_64+0xc1/0x440 [ 8064.379794] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8064.381589] RIP: 0033:0x7ff54af45332 [ 8064.387820] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8064.401984] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8064.408312] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 8064.413276] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 8064.418183] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 8064.426880] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 8064.432723] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 8068.056076] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 8068.062763] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 8068.063377] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8068.063456] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8068.063464] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8068.063467] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8068.063554] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8068.063558] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8068.063563] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 8068.063566] RDX: 0000000000003fc2 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 8068.063568] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 8068.063570] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 8068.063573] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8068.063576] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 8068.063580] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8068.063582] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 8068.063590] Call Trace: [ 8068.063595] [ 8068.063601] ? show_regs.cold.9+0x22/0x2f [ 8068.063613] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8068.063620] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8068.063624] ? watchdog+0x40/0x40 [ 8068.063629] ? hrtimer_interrupt+0x128/0x2c0 [ 8068.063633] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8068.063642] ? apic_timer_interrupt+0xf/0x20 [ 8068.063647] [ 8068.063650] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8068.063655] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8068.063660] do_raw_spin_lock+0x177/0x190 [ 8068.063665] _raw_spin_lock+0x21/0x30 [ 8068.063671] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8068.063966] obd_mod_free+0x46/0x140 [ptlrpc] [ 8068.064421] mdc_commit_open+0x91/0xb0 [mdc] [ 8068.064461] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8068.064735] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8068.064944] after_reply+0xe2c/0x1780 [ptlrpc] [ 8068.065160] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8068.065376] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8068.072587] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8068.072676] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8068.072686] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8068.072696] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8068.131668] ? do_wait_intr+0xf0/0xf0 [ 8068.131693] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8068.131983] kthread+0x1d1/0x200 [ 8068.131990] ? set_kthread_struct+0x70/0x70 [ 8068.131995] ret_from_fork+0x1f/0x30 [ 8068.410456] [ 8068.411489] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8068.415782] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8068.429290] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8068.439017] RAX: 0000000000003a88 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 8068.441659] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 8068.453327] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 8068.461923] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 8068.465396] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 8068.469562] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 8068.476410] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8068.479326] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 8068.486409] Call Trace: [ 8068.489397] [ 8068.491871] ? show_regs.cold.9+0x22/0x2f [ 8068.494299] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8068.497932] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8068.499950] ? watchdog+0x40/0x40 [ 8068.501288] ? hrtimer_interrupt+0x128/0x2c0 [ 8068.503006] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8068.504896] ? apic_timer_interrupt+0xf/0x20 [ 8068.507337] [ 8068.508307] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8068.511198] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8068.523398] do_raw_spin_lock+0x177/0x190 [ 8068.527396] _raw_spin_lock+0x21/0x30 [ 8068.530431] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8068.537979] process_one_work+0x2c8/0x700 [ 8068.542258] worker_thread+0x296/0x6e0 [ 8068.544420] ? rescuer_thread+0x570/0x570 [ 8068.547933] kthread+0x1d1/0x200 [ 8068.551671] ? set_kthread_struct+0x70/0x70 [ 8068.554001] ret_from_fork+0x1f/0x30 [ 8071.650521] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6112s! [ 8071.658833] Showing busy workqueues and worker pools: [ 8071.664556] workqueue events: flags=0x0 [ 8071.669067] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8071.681364] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 8071.692048] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8071.701287] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 8071.723028] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8071.729671] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 8071.743312] workqueue events_long: flags=0x0 [ 8071.748845] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8071.764647] pending: lu_objects_flush [obdclass] [ 8071.774309] workqueue events_unbound: flags=0x2 [ 8071.777262] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8071.785670] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8071.795342] workqueue events_power_efficient: flags=0x80 [ 8071.799481] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8071.806892] pending: neigh_periodic_work [ 8071.811788] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8071.818282] pending: do_cache_clean, check_lifetime [ 8071.825324] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8071.833177] pending: neigh_periodic_work [ 8071.836052] workqueue writeback: flags=0x4a [ 8071.842879] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8071.851084] pending: wb_workfn, wb_workfn [ 8071.855538] workqueue xprtiod: flags=0xa [ 8071.857843] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8071.868662] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 8071.880614] workqueue ipv6_addrconf: flags=0x40008 [ 8071.890371] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8071.901197] pending: addrconf_verify_work [ 8071.909064] workqueue knbd0-recv: flags=0x1a [ 8071.911491] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8071.919453] in-flight: 657:recv_work [ 8071.926244] workqueue ptlrpc_pinger: flags=0x2 [ 8071.928108] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8071.935391] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 8071.942181] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6113s workers=4 idle: 20831 12642 28859 [ 8071.949934] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8062s workers=2 idle: 659 [ 8084.058383] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 8084.064656] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8084.089648] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8084.094685] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8084.099237] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8084.101675] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8084.110455] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8084.114070] RAX: 0000000000005b47 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 8084.117459] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 8084.120749] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8084.124341] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 8084.128307] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 8084.131685] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 8084.135569] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8084.138498] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 8084.141788] Call Trace: [ 8084.143234] [ 8084.144308] ? show_regs.cold.9+0x22/0x2f [ 8084.146118] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8084.147864] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8084.150102] ? watchdog+0x40/0x40 [ 8084.151550] ? hrtimer_interrupt+0x128/0x2c0 [ 8084.153619] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8084.155769] ? apic_timer_interrupt+0xf/0x20 [ 8084.157731] [ 8084.158833] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8084.161205] do_raw_spin_lock+0x177/0x190 [ 8084.162818] _raw_spin_lock+0x21/0x30 [ 8084.164460] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8084.167279] ? do_raw_spin_unlock+0x75/0x190 [ 8084.169372] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8084.172233] ? do_wait_intr+0xf0/0xf0 [ 8084.174558] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8084.176784] mdc_close+0x53b/0xd70 [mdc] [ 8084.178663] lmv_close+0x18d/0x490 [lmv] [ 8084.180373] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 8084.182677] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8084.185658] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8084.187299] ll_md_close+0x274/0x800 [lustre] [ 8084.189146] ll_file_release+0x128/0xc50 [lustre] [ 8084.191264] __fput+0x10d/0x440 [ 8084.192412] ____fput+0x12/0x20 [ 8084.193799] task_work_run+0xc6/0x110 [ 8084.195781] exit_to_usermode_loop+0x1dd/0x1f0 [ 8084.201043] do_syscall_64+0x3d6/0x440 [ 8084.202672] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8084.204511] RIP: 0033:0x7ff54af45cd8 [ 8084.206408] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8084.213413] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8084.215966] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 8084.219176] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 8084.222217] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 8084.224807] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 8084.227913] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8092.060083] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [ 8092.063298] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8092.108096] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8092.111631] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8092.118364] RIP: 0010:__pv_queued_spin_lock_slowpath+0x558/0x6d0 [ 8092.121230] Code: f5 d0 dc 02 01 48 83 05 7d d3 dc 02 01 e9 e7 fc ff ff 48 83 05 d0 d3 dc 02 01 45 31 ed e9 cf fb ff ff 48 83 05 00 d2 dc 02 01 <31> d2 e9 93 fe ff ff 48 83 05 09 d2 dc 02 01 65 48 ff 05 c1 86 e7 [ 8092.148859] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8092.159382] RAX: 0000000000000000 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 8092.162717] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 8092.172752] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8092.181149] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 8092.186336] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 8092.197106] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 8092.210439] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8092.217685] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 8092.227193] Call Trace: [ 8092.231660] [ 8092.233451] ? show_regs.cold.9+0x22/0x2f [ 8092.236578] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8092.243488] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8092.245710] ? watchdog+0x40/0x40 [ 8092.248135] ? hrtimer_interrupt+0x128/0x2c0 [ 8092.252305] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8092.254867] ? apic_timer_interrupt+0xf/0x20 [ 8092.256914] [ 8092.259861] ? __pv_queued_spin_lock_slowpath+0x558/0x6d0 [ 8092.265437] do_raw_spin_lock+0x177/0x190 [ 8092.266820] _raw_spin_lock+0x21/0x30 [ 8092.268914] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8092.275632] ? do_raw_spin_unlock+0x75/0x190 [ 8092.278816] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8092.283470] ? do_wait_intr+0xf0/0xf0 [ 8092.285554] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8092.288482] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8092.298502] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8092.307646] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8092.309067] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8092.314739] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8092.319683] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8092.328284] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8092.333664] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8092.343274] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8092.345153] lmv_intent_open+0x338/0x1150 [lmv] [ 8092.346708] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8092.361774] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8092.367506] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8092.377088] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8092.382517] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8092.390712] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8092.396183] ? do_raw_spin_unlock+0x75/0x190 [ 8092.399737] ? _raw_spin_unlock+0x12/0x30 [ 8092.401132] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8092.404690] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8092.413839] ll_atomic_open+0x36a/0xd70 [lustre] [ 8092.417073] lookup_open+0x356/0xb40 [ 8092.421439] do_last+0x313/0xfc0 [ 8092.424128] ? nd_jump_root+0xe5/0x160 [ 8092.428558] ? path_init+0x437/0x520 [ 8092.430997] path_openat+0xf7/0x500 [ 8092.436446] ? do_raw_spin_unlock+0x75/0x190 [ 8092.437835] do_filp_open+0x99/0x140 [ 8092.439053] ? getname_flags+0x6e/0x330 [ 8092.452819] ? __check_object_size+0xff/0x256 [ 8092.457142] ? do_raw_spin_unlock+0x75/0x190 [ 8092.463050] ? _raw_spin_unlock+0x12/0x30 [ 8092.466088] do_sys_openat2+0x2b4/0x410 [ 8092.470665] do_sys_open+0x73/0xa0 [ 8092.474231] __x64_sys_openat+0x24/0x30 [ 8092.479609] do_syscall_64+0xc1/0x440 [ 8092.481040] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8092.488210] RIP: 0033:0x7ff54af45332 [ 8092.492703] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8092.504380] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8092.510565] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 8092.514599] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 8092.527726] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 8092.532769] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 8092.536916] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 8096.055008] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [ 8096.063173] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 8096.063249] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8096.063524] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8096.063530] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8096.063533] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8096.063548] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8096.063552] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8096.063557] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 8096.063560] RDX: 000000000000361f RSI: 0000000000000003 RDI: ffff96848811cae0 [ 8096.063562] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 8096.063564] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 8096.063567] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8096.063570] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 8096.063573] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8096.063575] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 8096.063583] Call Trace: [ 8096.063589] [ 8096.063594] ? show_regs.cold.9+0x22/0x2f [ 8096.063604] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8096.063610] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8096.063614] ? watchdog+0x40/0x40 [ 8096.063619] ? hrtimer_interrupt+0x128/0x2c0 [ 8096.063623] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8096.063632] ? apic_timer_interrupt+0xf/0x20 [ 8096.063636] [ 8096.063639] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8096.063644] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8096.063648] do_raw_spin_lock+0x177/0x190 [ 8096.063652] _raw_spin_lock+0x21/0x30 [ 8096.063657] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8096.065865] obd_mod_free+0x46/0x140 [ptlrpc] [ 8096.066076] mdc_commit_open+0x91/0xb0 [mdc] [ 8096.066116] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8096.066881] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8096.077276] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8096.077370] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8096.077378] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8096.077384] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8096.077959] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8096.077971] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8096.077975] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8096.077979] RAX: 0000000000003304 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 8096.077981] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 8096.077983] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 8096.077985] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 8096.077988] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 8096.077991] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 8096.077994] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8096.077996] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 8096.078005] Call Trace: [ 8096.078010] [ 8096.078016] ? show_regs.cold.9+0x22/0x2f [ 8096.078027] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8096.078147] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8096.078152] ? watchdog+0x40/0x40 [ 8096.078156] ? hrtimer_interrupt+0x128/0x2c0 [ 8096.078427] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8096.078519] ? apic_timer_interrupt+0xf/0x20 [ 8096.078524] [ 8096.078527] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8096.078532] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8096.078537] do_raw_spin_lock+0x177/0x190 [ 8096.078542] _raw_spin_lock+0x21/0x30 [ 8096.078547] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8096.078849] process_one_work+0x2c8/0x700 [ 8096.078861] worker_thread+0x296/0x6e0 [ 8096.078866] ? rescuer_thread+0x570/0x570 [ 8096.078869] kthread+0x1d1/0x200 [ 8096.078874] ? set_kthread_struct+0x70/0x70 [ 8096.078877] ret_from_fork+0x1f/0x30 [ 8096.649283] after_reply+0xe2c/0x1780 [ptlrpc] [ 8096.653360] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8096.661095] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8096.665776] ? do_wait_intr+0xf0/0xf0 [ 8096.667654] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8096.672497] kthread+0x1d1/0x200 [ 8096.678643] ? set_kthread_struct+0x70/0x70 [ 8096.682943] ret_from_fork+0x1f/0x30 [ 8102.378017] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6143s! [ 8102.386403] Showing busy workqueues and worker pools: [ 8102.393699] workqueue events: flags=0x0 [ 8102.396580] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8102.405002] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 8102.413622] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8102.421626] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 8102.432982] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8102.439020] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 8102.444551] workqueue events_long: flags=0x0 [ 8102.451333] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8102.459550] pending: lu_objects_flush [obdclass] [ 8102.466949] workqueue events_unbound: flags=0x2 [ 8102.471347] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8102.480414] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8102.500294] workqueue events_power_efficient: flags=0x80 [ 8102.506358] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8102.511967] pending: neigh_periodic_work [ 8102.516646] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8102.524545] pending: do_cache_clean, check_lifetime [ 8102.531450] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8102.535169] pending: neigh_periodic_work [ 8102.543157] workqueue writeback: flags=0x4a [ 8102.546577] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8102.554407] pending: wb_workfn, wb_workfn [ 8102.558278] workqueue xprtiod: flags=0xa [ 8102.561993] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8102.567165] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 8102.571671] workqueue ipv6_addrconf: flags=0x40008 [ 8102.573415] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8102.580944] pending: addrconf_verify_work [ 8102.582198] workqueue knbd0-recv: flags=0x1a [ 8102.583422] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8102.593397] in-flight: 657:recv_work [ 8102.596168] workqueue ptlrpc_pinger: flags=0x2 [ 8102.598797] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8102.602626] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 8102.606616] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6143s workers=4 idle: 20831 12642 28859 [ 8102.610953] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8092s workers=2 idle: 659 [ 8112.063982] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 8112.072262] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8112.132881] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8112.137273] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8112.142194] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8112.145174] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8112.153325] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8112.165766] RAX: 0000000000007e09 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 8112.178316] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 8112.185332] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8112.202082] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 8112.215311] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 8112.221361] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 8112.238995] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8112.246968] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 8112.260106] Call Trace: [ 8112.262150] [ 8112.265152] ? show_regs.cold.9+0x22/0x2f [ 8112.270179] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8112.277965] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8112.288565] ? watchdog+0x40/0x40 [ 8112.296451] ? hrtimer_interrupt+0x128/0x2c0 [ 8112.297946] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8112.310860] ? apic_timer_interrupt+0xf/0x20 [ 8112.316432] [ 8112.319428] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8112.327511] do_raw_spin_lock+0x177/0x190 [ 8112.332892] _raw_spin_lock+0x21/0x30 [ 8112.337191] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8112.343487] ? do_raw_spin_unlock+0x75/0x190 [ 8112.348997] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8112.353270] ? do_wait_intr+0xf0/0xf0 [ 8112.356588] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8112.359431] mdc_close+0x53b/0xd70 [mdc] [ 8112.362545] lmv_close+0x18d/0x490 [lmv] [ 8112.366273] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 8112.375472] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8112.377550] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8112.386359] ll_md_close+0x274/0x800 [lustre] [ 8112.395124] ll_file_release+0x128/0xc50 [lustre] [ 8112.398075] __fput+0x10d/0x440 [ 8112.399542] ____fput+0x12/0x20 [ 8112.402851] task_work_run+0xc6/0x110 [ 8112.409373] exit_to_usermode_loop+0x1dd/0x1f0 [ 8112.415150] do_syscall_64+0x3d6/0x440 [ 8112.419348] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8112.425147] RIP: 0033:0x7ff54af45cd8 [ 8112.427426] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8112.447118] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8112.452396] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 8112.456318] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 8112.463887] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 8112.473127] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 8112.482759] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8120.060015] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 8120.063257] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8120.088011] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8120.095044] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8120.099389] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8120.102362] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8120.109979] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8120.115319] RAX: 0000000000006917 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 8120.118550] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 8120.121932] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8120.125564] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 8120.129354] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 8120.134243] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 8120.137742] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8120.140458] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 8120.144764] Call Trace: [ 8120.146135] [ 8120.147297] ? show_regs.cold.9+0x22/0x2f [ 8120.149258] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8120.151873] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8120.153965] ? watchdog+0x40/0x40 [ 8120.155477] ? hrtimer_interrupt+0x128/0x2c0 [ 8120.157797] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8120.160068] ? apic_timer_interrupt+0xf/0x20 [ 8120.162062] [ 8120.163159] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8120.165125] do_raw_spin_lock+0x177/0x190 [ 8120.167035] _raw_spin_lock+0x21/0x30 [ 8120.168390] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8120.170956] ? do_raw_spin_unlock+0x75/0x190 [ 8120.172314] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8120.174837] ? do_wait_intr+0xf0/0xf0 [ 8120.175984] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8120.178810] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8120.181158] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8120.184085] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8120.186692] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8120.188908] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8120.191623] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8120.193649] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8120.196011] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8120.198452] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8120.200680] lmv_intent_open+0x338/0x1150 [lmv] [ 8120.201894] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8120.203820] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8120.205904] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8120.207599] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8120.209396] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8120.211801] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8120.213825] ? do_raw_spin_unlock+0x75/0x190 [ 8120.215336] ? _raw_spin_unlock+0x12/0x30 [ 8120.217037] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8120.218950] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8120.221151] ll_atomic_open+0x36a/0xd70 [lustre] [ 8120.223362] lookup_open+0x356/0xb40 [ 8120.224956] do_last+0x313/0xfc0 [ 8120.226285] ? nd_jump_root+0xe5/0x160 [ 8120.227774] ? path_init+0x437/0x520 [ 8120.228936] path_openat+0xf7/0x500 [ 8120.230263] ? do_raw_spin_unlock+0x75/0x190 [ 8120.232115] do_filp_open+0x99/0x140 [ 8120.233795] ? getname_flags+0x6e/0x330 [ 8120.235974] ? __check_object_size+0xff/0x256 [ 8120.237278] ? do_raw_spin_unlock+0x75/0x190 [ 8120.239372] ? _raw_spin_unlock+0x12/0x30 [ 8120.240815] do_sys_openat2+0x2b4/0x410 [ 8120.242456] do_sys_open+0x73/0xa0 [ 8120.244130] __x64_sys_openat+0x24/0x30 [ 8120.245849] do_syscall_64+0xc1/0x440 [ 8120.247309] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8120.249681] RIP: 0033:0x7ff54af45332 [ 8120.251494] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8120.258654] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8120.261526] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 8120.264253] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 8120.266330] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 8120.268791] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 8120.275402] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 8124.059352] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 8124.066211] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8124.117224] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8124.130749] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8124.137355] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8124.146437] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8124.147955] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8124.183262] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8124.208092] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 8124.218171] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 8124.229920] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 8124.232732] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 8124.250136] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 8124.262576] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 8124.274168] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8124.288849] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 8124.297104] Call Trace: [ 8124.299945] [ 8124.300944] ? show_regs.cold.9+0x22/0x2f [ 8124.308521] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8124.316224] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8124.321712] ? watchdog+0x40/0x40 [ 8124.329848] ? hrtimer_interrupt+0x128/0x2c0 [ 8124.331501] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8124.347036] ? apic_timer_interrupt+0xf/0x20 [ 8124.351680] [ 8124.354414] ? native_safe_halt+0x2e/0x50 [ 8124.360389] kvm_wait+0xcd/0xe0 [ 8124.363873] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8124.371676] do_raw_spin_lock+0x177/0x190 [ 8124.377813] _raw_spin_lock+0x21/0x30 [ 8124.379013] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8124.398722] process_one_work+0x2c8/0x700 [ 8124.408739] worker_thread+0x296/0x6e0 [ 8124.417517] ? rescuer_thread+0x570/0x570 [ 8124.422532] kthread+0x1d1/0x200 [ 8124.427756] ? set_kthread_struct+0x70/0x70 [ 8124.430646] ret_from_fork+0x1f/0x30 [ 8128.058699] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 8128.078917] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8128.150306] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8128.175987] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8128.197893] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8128.202900] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8128.245241] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8128.256259] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 8128.262365] RDX: 00000000000014fc RSI: 0000000000000003 RDI: ffff96848811cae0 [ 8128.279279] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 8128.289211] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 8128.294666] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8128.303654] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 8128.308946] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8128.313275] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 8128.317966] Call Trace: [ 8128.319648] [ 8128.321646] ? show_regs.cold.9+0x22/0x2f [ 8128.325947] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8128.332101] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8128.335850] ? watchdog+0x40/0x40 [ 8128.338260] ? hrtimer_interrupt+0x128/0x2c0 [ 8128.342298] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8128.345470] ? apic_timer_interrupt+0xf/0x20 [ 8128.347432] [ 8128.349118] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8128.353287] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8128.357726] do_raw_spin_lock+0x177/0x190 [ 8128.360338] _raw_spin_lock+0x21/0x30 [ 8128.361895] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8128.364176] obd_mod_free+0x46/0x140 [ptlrpc] [ 8128.366252] mdc_commit_open+0x91/0xb0 [mdc] [ 8128.368047] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8128.370387] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8128.372815] after_reply+0xe2c/0x1780 [ptlrpc] [ 8128.375072] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8128.377250] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8128.379058] ? do_wait_intr+0xf0/0xf0 [ 8128.380525] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8128.382720] kthread+0x1d1/0x200 [ 8128.384094] ? set_kthread_struct+0x70/0x70 [ 8128.385811] ret_from_fork+0x1f/0x30 [ 8133.094083] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6174s! [ 8133.102129] Showing busy workqueues and worker pools: [ 8133.106206] workqueue events: flags=0x0 [ 8133.114096] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8133.122035] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 8133.131424] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8133.134886] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 8133.151662] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8133.162962] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 8133.171615] workqueue events_long: flags=0x0 [ 8133.175722] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8133.186129] pending: lu_objects_flush [obdclass] [ 8133.192682] workqueue events_unbound: flags=0x2 [ 8133.200815] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8133.203267] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8133.216184] workqueue events_power_efficient: flags=0x80 [ 8133.222693] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8133.233037] pending: neigh_periodic_work [ 8133.238416] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8133.253270] pending: do_cache_clean, check_lifetime [ 8133.261343] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8133.267846] pending: neigh_periodic_work [ 8133.272806] workqueue writeback: flags=0x4a [ 8133.278799] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8133.291290] pending: wb_workfn, wb_workfn [ 8133.297762] workqueue xprtiod: flags=0xa [ 8133.301911] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8133.308659] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 8133.319630] workqueue ipv6_addrconf: flags=0x40008 [ 8133.332251] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8133.346108] pending: addrconf_verify_work [ 8133.350223] workqueue knbd0-recv: flags=0x1a [ 8133.356489] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8133.361173] in-flight: 657:recv_work [ 8133.370931] workqueue ptlrpc_pinger: flags=0x2 [ 8133.373050] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8133.375289] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 8133.387322] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6174s workers=4 idle: 20831 12642 28859 [ 8133.400871] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8123s workers=2 idle: 659 [ 8134.018701] rcu: INFO: rcu_sched self-detected stall on CPU [ 8134.024970] rcu: 0-...!: (3187174 ticks this GP) idle=98a/1/0x4000000000000002 softirq=125154/125154 fqs=959 [ 8134.034798] (t=6180178 jiffies g=253585 q=4832) [ 8134.049590] rcu: rcu_sched kthread starved for 6175138 jiffies! g253585 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=3 [ 8134.058622] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 8134.072418] rcu: RCU grace-period kthread stack dump: [ 8134.080344] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 8134.092011] Call Trace: [ 8134.093168] __schedule+0x351/0xcb0 [ 8134.095482] schedule+0xc0/0x180 [ 8134.097109] schedule_timeout+0xb4/0x190 [ 8134.108241] ? do_raw_spin_unlock+0x75/0x190 [ 8134.114304] ? __next_timer_interrupt+0x160/0x160 [ 8134.118620] ? prepare_to_swait_event+0xf3/0x140 [ 8134.122936] rcu_gp_kthread+0x8dd/0x15b0 [ 8134.126154] ? force_qs_rnp+0x2c0/0x2c0 [ 8134.134016] kthread+0x1d1/0x200 [ 8134.135482] ? set_kthread_struct+0x70/0x70 [ 8134.144004] ret_from_fork+0x1f/0x30 [ 8134.146109] rcu: Stack dump where RCU GP kthread last ran: [ 8134.151986] Sending NMI from CPU 0 to CPUs 3: [ 8134.173240] INFO: NMI handler (nmi_cpu_backtrace_handler) took too long to run: 13.506 msecs [ 8134.175543] NMI backtrace for cpu 3 [ 8134.175553] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8134.175555] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8134.175556] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8134.175559] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8134.175561] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 [ 8134.175564] RAX: 0000000000006c20 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 8134.175565] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 8134.175566] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8134.175567] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 8134.175568] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 8134.175570] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 8134.175571] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8134.175572] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 8134.175572] Call Trace: [ 8134.175573] [ 8134.175574] ? show_regs.cold.9+0x22/0x2f [ 8134.175575] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 8134.175576] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 8134.175576] ? nmi_handle+0xbd/0x210 [ 8134.175577] ? default_do_nmi+0x4b/0x1b0 [ 8134.175578] ? do_nmi+0x280/0x380 [ 8134.175579] ? end_repeat_nmi+0x16/0x1f [ 8134.175580] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8134.175581] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8134.175582] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8134.175583] [ 8134.175584] do_raw_spin_lock+0x177/0x190 [ 8134.175586] _raw_spin_lock+0x21/0x30 [ 8134.175587] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8134.175588] ? do_raw_spin_unlock+0x75/0x190 [ 8134.175589] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8134.175590] ? do_wait_intr+0xf0/0xf0 [ 8134.175591] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8134.175592] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8134.175593] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8134.175594] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8134.175594] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8134.175595] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8134.175596] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8134.175597] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8134.175598] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8134.175599] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8134.175599] lmv_intent_open+0x338/0x1150 [lmv] [ 8134.175600] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8134.175601] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8134.175602] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8134.175603] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8134.175604] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8134.175605] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8134.175606] ? do_raw_spin_unlock+0x75/0x190 [ 8134.175607] ? _raw_spin_unlock+0x12/0x30 [ 8134.175608] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8134.175609] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8134.175610] ll_atomic_open+0x36a/0xd70 [lustre] [ 8134.175611] lookup_open+0x356/0xb40 [ 8134.175612] do_last+0x313/0xfc0 [ 8134.175613] ? nd_jump_root+0xe5/0x160 [ 8134.175615] ? path_init+0x437/0x520 [ 8134.175616] path_openat+0xf7/0x500 [ 8134.175617] ? do_raw_spin_unlock+0x75/0x190 [ 8134.175618] do_filp_open+0x99/0x140 [ 8134.175619] ? getname_flags+0x6e/0x330 [ 8134.175620] ? __check_object_size+0xff/0x256 [ 8134.175621] ? do_raw_spin_unlock+0x75/0x190 [ 8134.175623] ? _raw_spin_unlock+0x12/0x30 [ 8134.175623] do_sys_openat2+0x2b4/0x410 [ 8134.175624] do_sys_open+0x73/0xa0 [ 8134.175625] __x64_sys_openat+0x24/0x30 [ 8134.175626] do_syscall_64+0xc1/0x440 [ 8134.175627] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8134.175628] RIP: 0033:0x7ff54af45332 [ 8134.175629] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8134.175631] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8134.175633] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 8134.175634] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 8134.175635] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 8134.175637] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 8134.175638] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 8134.175643] NMI backtrace for cpu 0 [ 8134.878045] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8134.890006] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8134.920514] Call Trace: [ 8134.921690] [ 8134.923009] ? dump_stack+0xbb/0x10e [ 8134.940388] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 8134.942751] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 8134.951697] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 8134.971267] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 8134.973722] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 8134.982646] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 8134.991676] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 8134.995672] ? update_process_times+0x6d/0xc0 [ 8135.017564] ? tick_sched_handle.isra.18+0x27/0x90 [ 8135.019781] ? tick_sched_timer+0x53/0xb0 [ 8135.027551] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8135.034969] ? tick_sched_handle.isra.18+0x90/0x90 [ 8135.046929] ? hrtimer_interrupt+0x128/0x2c0 [ 8135.064003] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8135.081977] ? apic_timer_interrupt+0xf/0x20 [ 8135.095134] [ 8135.103009] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8135.115169] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8135.123158] ? do_raw_spin_lock+0x177/0x190 [ 8135.132761] ? _raw_spin_lock+0x21/0x30 [ 8135.135154] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8135.139816] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 8135.143895] ? mdc_commit_open+0x91/0xb0 [mdc] [ 8135.152321] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8135.162858] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8135.170763] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 8135.172425] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8135.186940] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8135.201838] ? do_wait_intr+0xf0/0xf0 [ 8135.206804] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8135.214473] ? kthread+0x1d1/0x200 [ 8135.215944] ? set_kthread_struct+0x70/0x70 [ 8135.232023] ? ret_from_fork+0x1f/0x30 [ 8140.058927] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [ 8140.068706] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8140.097021] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8140.115972] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8140.124906] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8140.138264] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8140.156712] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8140.162916] RAX: 00000000000041ef RBX: ffff96848811cae0 RCX: 0000000000000001 [ 8140.171756] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 8140.176306] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8140.182275] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 8140.186088] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 8140.194459] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 8140.203235] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8140.209626] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 8140.216628] Call Trace: [ 8140.218698] [ 8140.219704] ? show_regs.cold.9+0x22/0x2f [ 8140.223415] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8140.226610] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8140.229690] ? watchdog+0x40/0x40 [ 8140.231719] ? hrtimer_interrupt+0x128/0x2c0 [ 8140.240197] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8140.243609] ? apic_timer_interrupt+0xf/0x20 [ 8140.244970] [ 8140.250398] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8140.256734] do_raw_spin_lock+0x177/0x190 [ 8140.259475] _raw_spin_lock+0x21/0x30 [ 8140.260554] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8140.265530] ? do_raw_spin_unlock+0x75/0x190 [ 8140.268805] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8140.275436] ? do_wait_intr+0xf0/0xf0 [ 8140.276928] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8140.281208] mdc_close+0x53b/0xd70 [mdc] [ 8140.283238] lmv_close+0x18d/0x490 [lmv] [ 8140.286712] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 8140.295201] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8140.299200] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8140.307104] ll_md_close+0x274/0x800 [lustre] [ 8140.312322] ll_file_release+0x128/0xc50 [lustre] [ 8140.315141] __fput+0x10d/0x440 [ 8140.319101] ____fput+0x12/0x20 [ 8140.326389] task_work_run+0xc6/0x110 [ 8140.329181] exit_to_usermode_loop+0x1dd/0x1f0 [ 8140.333105] do_syscall_64+0x3d6/0x440 [ 8140.336277] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8140.338678] RIP: 0033:0x7ff54af45cd8 [ 8140.340661] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8140.354655] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8140.362171] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 8140.376741] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 8140.379049] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 8140.382912] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 8140.385072] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8152.061894] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 8152.065646] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8152.101491] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8152.110293] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8152.114727] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8152.118078] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8152.120679] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8152.128104] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8152.131602] RAX: 0000000000007d91 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 8152.134957] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 8152.137927] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 8152.141115] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 8152.143707] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 8152.146893] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 8152.150076] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8152.152577] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 8152.157064] Call Trace: [ 8152.158098] [ 8152.159030] ? show_regs.cold.9+0x22/0x2f [ 8152.160589] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8152.162561] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8152.164671] ? watchdog+0x40/0x40 [ 8152.166040] ? hrtimer_interrupt+0x128/0x2c0 [ 8152.167892] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8152.169999] ? apic_timer_interrupt+0xf/0x20 [ 8152.171900] [ 8152.173064] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8152.179233] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8152.184781] do_raw_spin_lock+0x177/0x190 [ 8152.186537] _raw_spin_lock+0x21/0x30 [ 8152.188959] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8152.198903] process_one_work+0x2c8/0x700 [ 8152.208601] worker_thread+0x296/0x6e0 [ 8152.210442] ? rescuer_thread+0x570/0x570 [ 8152.216266] kthread+0x1d1/0x200 [ 8152.218506] ? set_kthread_struct+0x70/0x70 [ 8152.224910] ret_from_fork+0x1f/0x30 [ 8160.055980] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 8160.064838] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8160.070557] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 8160.105088] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8160.105111] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8160.105115] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8160.105128] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8160.105133] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8160.105140] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 8160.105143] RDX: 00000000000062d1 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 8160.105145] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 8160.105148] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 8160.105150] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8160.105153] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 8160.105156] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8160.105159] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 8160.105167] Call Trace: [ 8160.105173] [ 8160.105178] ? show_regs.cold.9+0x22/0x2f [ 8160.105186] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8160.105192] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8160.105197] ? watchdog+0x40/0x40 [ 8160.105201] ? hrtimer_interrupt+0x128/0x2c0 [ 8160.105206] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8160.105225] ? apic_timer_interrupt+0xf/0x20 [ 8160.105231] [ 8160.105234] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8160.105238] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8160.105243] do_raw_spin_lock+0x177/0x190 [ 8160.105247] _raw_spin_lock+0x21/0x30 [ 8160.105254] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8160.106293] obd_mod_free+0x46/0x140 [ptlrpc] [ 8160.107198] mdc_commit_open+0x91/0xb0 [mdc] [ 8160.107590] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8160.107828] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8160.108057] after_reply+0xe2c/0x1780 [ptlrpc] [ 8160.108314] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8160.113870] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8160.114012] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8160.114020] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8160.114024] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8160.114038] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8160.114043] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8160.114048] RAX: 0000000000000c86 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 8160.114050] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 8160.114053] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8160.114055] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 8160.114058] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 8160.114062] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 8160.114065] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8160.114068] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 8160.114077] Call Trace: [ 8160.114083] [ 8160.114089] ? show_regs.cold.9+0x22/0x2f [ 8160.114100] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8160.114107] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8160.114111] ? watchdog+0x40/0x40 [ 8160.114115] ? hrtimer_interrupt+0x128/0x2c0 [ 8160.114119] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8160.114128] ? apic_timer_interrupt+0xf/0x20 [ 8160.114133] [ 8160.114137] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8160.114144] do_raw_spin_lock+0x177/0x190 [ 8160.114149] _raw_spin_lock+0x21/0x30 [ 8160.114154] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8160.114467] ? do_raw_spin_unlock+0x75/0x190 [ 8160.114473] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8160.123846] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8160.127171] ? do_wait_intr+0xf0/0xf0 [ 8160.127193] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8160.127485] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8160.127826] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8160.132374] ? do_wait_intr+0xf0/0xf0 [ 8160.140447] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8160.143836] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8160.148864] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8160.151976] kthread+0x1d1/0x200 [ 8160.151995] ? set_kthread_struct+0x70/0x70 [ 8160.152000] ret_from_fork+0x1f/0x30 [ 8160.607753] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8160.614658] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8160.618827] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8160.627401] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8160.638814] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8160.641808] lmv_intent_open+0x338/0x1150 [lmv] [ 8160.650952] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8160.660983] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8160.668820] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8160.672693] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8160.677248] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8160.679151] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8160.689520] ? do_raw_spin_unlock+0x75/0x190 [ 8160.690805] ? _raw_spin_unlock+0x12/0x30 [ 8160.698676] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8160.706720] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8160.713471] ll_atomic_open+0x36a/0xd70 [lustre] [ 8160.716732] lookup_open+0x356/0xb40 [ 8160.718467] do_last+0x313/0xfc0 [ 8160.719889] ? nd_jump_root+0xe5/0x160 [ 8160.722952] ? path_init+0x437/0x520 [ 8160.727502] path_openat+0xf7/0x500 [ 8160.730615] ? do_raw_spin_unlock+0x75/0x190 [ 8160.734939] do_filp_open+0x99/0x140 [ 8160.740037] ? getname_flags+0x6e/0x330 [ 8160.745251] ? __check_object_size+0xff/0x256 [ 8160.755616] ? do_raw_spin_unlock+0x75/0x190 [ 8160.762049] ? _raw_spin_unlock+0x12/0x30 [ 8160.771342] do_sys_openat2+0x2b4/0x410 [ 8160.777996] do_sys_open+0x73/0xa0 [ 8160.787353] __x64_sys_openat+0x24/0x30 [ 8160.793183] do_syscall_64+0xc1/0x440 [ 8160.805524] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8160.808663] RIP: 0033:0x7ff54af45332 [ 8160.811014] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8160.838617] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8160.854950] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 8160.864664] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 8160.870480] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 8160.886813] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 8160.899688] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 8163.813463] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6204s! [ 8163.831469] Showing busy workqueues and worker pools: [ 8163.838136] workqueue events: flags=0x0 [ 8163.843220] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8163.853853] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 8163.868602] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8163.873695] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 8163.886035] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8163.895276] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 8163.901772] workqueue events_long: flags=0x0 [ 8163.905976] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8163.910469] pending: lu_objects_flush [obdclass] [ 8163.923903] workqueue events_unbound: flags=0x2 [ 8163.925233] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8163.932464] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8163.935826] workqueue events_power_efficient: flags=0x80 [ 8163.947950] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8163.951036] pending: neigh_periodic_work [ 8163.955307] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8163.957927] pending: do_cache_clean, check_lifetime [ 8163.962208] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8163.964737] pending: neigh_periodic_work [ 8163.966533] workqueue writeback: flags=0x4a [ 8163.974016] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8163.985407] pending: wb_workfn, wb_workfn [ 8163.990884] workqueue xprtiod: flags=0xa [ 8163.997678] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8164.000847] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 8164.010560] workqueue ipv6_addrconf: flags=0x40008 [ 8164.011981] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8164.020853] pending: addrconf_verify_work [ 8164.023555] workqueue knbd0-recv: flags=0x1a [ 8164.032680] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8164.034924] in-flight: 657:recv_work [ 8164.040905] workqueue ptlrpc_pinger: flags=0x2 [ 8164.046645] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8164.054559] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 8164.059258] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6205s workers=4 idle: 20831 12642 28859 [ 8164.066393] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8154s workers=2 idle: 659 [ 8168.060255] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [ 8168.065654] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8168.114468] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8168.127106] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8168.132976] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8168.138288] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8168.155770] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8168.163393] RAX: 0000000000007623 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 8168.169774] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 8168.176676] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8168.184331] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 8168.189042] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 8168.197572] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 8168.206164] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8168.212784] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 8168.219979] Call Trace: [ 8168.225376] [ 8168.226968] ? show_regs.cold.9+0x22/0x2f [ 8168.232989] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8168.235916] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8168.239524] ? watchdog+0x40/0x40 [ 8168.241953] ? hrtimer_interrupt+0x128/0x2c0 [ 8168.244824] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8168.251491] ? apic_timer_interrupt+0xf/0x20 [ 8168.257003] [ 8168.258323] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8168.261755] do_raw_spin_lock+0x177/0x190 [ 8168.269967] _raw_spin_lock+0x21/0x30 [ 8168.274571] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8168.281146] ? do_raw_spin_unlock+0x75/0x190 [ 8168.284224] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8168.288726] ? do_wait_intr+0xf0/0xf0 [ 8168.290554] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8168.294570] mdc_close+0x53b/0xd70 [mdc] [ 8168.298239] lmv_close+0x18d/0x490 [lmv] [ 8168.300611] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 8168.303928] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8168.311188] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8168.315822] ll_md_close+0x274/0x800 [lustre] [ 8168.319437] ll_file_release+0x128/0xc50 [lustre] [ 8168.321554] __fput+0x10d/0x440 [ 8168.325568] ____fput+0x12/0x20 [ 8168.327467] task_work_run+0xc6/0x110 [ 8168.329904] exit_to_usermode_loop+0x1dd/0x1f0 [ 8168.334459] do_syscall_64+0x3d6/0x440 [ 8168.337525] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8168.342818] RIP: 0033:0x7ff54af45cd8 [ 8168.344494] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8168.354441] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8168.357766] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 8168.362647] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 8168.369393] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 8168.373391] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 8168.376514] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8180.065747] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 8180.077334] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8180.111671] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8180.117395] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8180.123019] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8180.126407] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8180.130682] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8180.138387] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8180.141677] RAX: 000000000000440b RBX: ffff96848811cae0 RCX: 0000000000000001 [ 8180.146706] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 8180.150047] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 8180.153317] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 8180.156686] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 8180.161509] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 8180.168621] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8180.173353] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 8180.177695] Call Trace: [ 8180.179251] [ 8180.181897] ? show_regs.cold.9+0x22/0x2f [ 8180.183978] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8180.187302] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8180.189907] ? watchdog+0x40/0x40 [ 8180.190926] ? hrtimer_interrupt+0x128/0x2c0 [ 8180.194345] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8180.202707] ? apic_timer_interrupt+0xf/0x20 [ 8180.204226] [ 8180.204964] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8180.214570] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8180.219595] do_raw_spin_lock+0x177/0x190 [ 8180.224585] _raw_spin_lock+0x21/0x30 [ 8180.230444] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8180.239471] process_one_work+0x2c8/0x700 [ 8180.240818] worker_thread+0x296/0x6e0 [ 8180.249538] ? rescuer_thread+0x570/0x570 [ 8180.257364] kthread+0x1d1/0x200 [ 8180.265153] ? set_kthread_struct+0x70/0x70 [ 8180.269650] ret_from_fork+0x1f/0x30 [ 8188.062152] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 8188.065273] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8188.108887] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8188.115420] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8188.121119] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8188.123608] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8188.134970] RSP: 0000:ffffaf9fc15234f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8188.140559] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 8188.146929] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 8188.149166] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8188.164910] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 8188.176798] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 8188.181736] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 8188.190385] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8188.196513] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 8188.200972] Call Trace: [ 8188.201806] [ 8188.202578] ? show_regs.cold.9+0x22/0x2f [ 8188.209165] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8188.210764] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8188.227228] ? watchdog+0x40/0x40 [ 8188.229481] ? hrtimer_interrupt+0x128/0x2c0 [ 8188.241500] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8188.251008] ? apic_timer_interrupt+0xf/0x20 [ 8188.257895] [ 8188.260458] ? native_safe_halt+0x2e/0x50 [ 8188.263123] kvm_wait+0xcd/0xe0 [ 8188.271476] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8188.280876] do_raw_spin_lock+0x177/0x190 [ 8188.288295] _raw_spin_lock+0x21/0x30 [ 8188.296126] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8188.299737] ? do_raw_spin_unlock+0x75/0x190 [ 8188.302709] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8188.305087] ? do_wait_intr+0xf0/0xf0 [ 8188.310753] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8188.318433] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8188.331878] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8188.339704] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8188.347605] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8188.353192] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8188.354878] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8188.359798] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8188.365305] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8188.373411] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8188.375154] lmv_intent_open+0x338/0x1150 [lmv] [ 8188.380411] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8188.384254] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8188.386332] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8188.395553] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8188.397226] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8188.406227] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8188.407936] ? do_raw_spin_unlock+0x75/0x190 [ 8188.415035] ? _raw_spin_unlock+0x12/0x30 [ 8188.420499] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8188.427789] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8188.432488] ll_atomic_open+0x36a/0xd70 [lustre] [ 8188.437689] lookup_open+0x356/0xb40 [ 8188.440148] do_last+0x313/0xfc0 [ 8188.441734] ? nd_jump_root+0xe5/0x160 [ 8188.445017] ? path_init+0x437/0x520 [ 8188.446114] path_openat+0xf7/0x500 [ 8188.453553] ? do_raw_spin_unlock+0x75/0x190 [ 8188.455862] do_filp_open+0x99/0x140 [ 8188.456686] ? getname_flags+0x6e/0x330 [ 8188.457558] ? __check_object_size+0xff/0x256 [ 8188.464634] ? do_raw_spin_unlock+0x75/0x190 [ 8188.469485] ? _raw_spin_unlock+0x12/0x30 [ 8188.470547] do_sys_openat2+0x2b4/0x410 [ 8188.472719] do_sys_open+0x73/0xa0 [ 8188.473969] __x64_sys_openat+0x24/0x30 [ 8188.483632] do_syscall_64+0xc1/0x440 [ 8188.484702] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8188.494329] RIP: 0033:0x7ff54af45332 [ 8188.496370] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8188.521683] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8188.529214] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 8188.537036] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 8188.546740] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 8188.555876] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 8188.563038] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 8192.062173] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 8192.071397] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8192.122708] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8192.144210] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8192.150381] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8192.153651] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8192.163453] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8192.176774] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 8192.184586] RDX: 0000000000007e29 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 8192.193502] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 8192.199598] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 8192.203998] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8192.206935] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 8192.212381] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8192.221085] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 8192.224925] Call Trace: [ 8192.225767] [ 8192.226391] ? show_regs.cold.9+0x22/0x2f [ 8192.229912] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8192.231681] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8192.233492] ? watchdog+0x40/0x40 [ 8192.236653] ? hrtimer_interrupt+0x128/0x2c0 [ 8192.239516] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8192.241180] ? apic_timer_interrupt+0xf/0x20 [ 8192.243915] [ 8192.245123] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8192.249910] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8192.251647] do_raw_spin_lock+0x177/0x190 [ 8192.255164] _raw_spin_lock+0x21/0x30 [ 8192.256325] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8192.267028] obd_mod_free+0x46/0x140 [ptlrpc] [ 8192.275705] mdc_commit_open+0x91/0xb0 [mdc] [ 8192.277640] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8192.283596] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8192.286540] after_reply+0xe2c/0x1780 [ptlrpc] [ 8192.289554] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8192.294264] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8192.297627] ? do_wait_intr+0xf0/0xf0 [ 8192.300556] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8192.305338] kthread+0x1d1/0x200 [ 8192.308169] ? set_kthread_struct+0x70/0x70 [ 8192.309760] ret_from_fork+0x1f/0x30 [ 8194.527081] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6235s! [ 8194.531885] Showing busy workqueues and worker pools: [ 8194.533985] workqueue events: flags=0x0 [ 8194.536451] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8194.550500] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 8194.560462] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8194.567107] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 8194.583406] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8194.590743] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 8194.592807] workqueue events_long: flags=0x0 [ 8194.608216] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8194.621463] pending: lu_objects_flush [obdclass] [ 8194.631351] workqueue events_unbound: flags=0x2 [ 8194.637679] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8194.654352] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8194.668465] workqueue events_power_efficient: flags=0x80 [ 8194.673491] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8194.683557] pending: neigh_periodic_work [ 8194.688119] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8194.695251] pending: do_cache_clean, check_lifetime [ 8194.703365] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8194.709645] pending: neigh_periodic_work [ 8194.716436] workqueue writeback: flags=0x4a [ 8194.724988] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8194.730150] pending: wb_workfn, wb_workfn [ 8194.734667] workqueue xprtiod: flags=0xa [ 8194.743318] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8194.753518] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 8194.763875] workqueue ipv6_addrconf: flags=0x40008 [ 8194.768709] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8194.775125] pending: addrconf_verify_work [ 8194.787397] workqueue knbd0-recv: flags=0x1a [ 8194.794753] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8194.800965] in-flight: 657:recv_work [ 8194.810559] workqueue ptlrpc_pinger: flags=0x2 [ 8194.814661] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8194.820510] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 8194.826697] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6235s workers=4 idle: 20831 12642 28859 [ 8194.845648] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8185s workers=2 idle: 659 [ 8196.062921] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [ 8196.080042] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8196.124708] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8196.136244] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8196.152473] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8196.165503] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8196.194883] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8196.197291] RAX: 0000000000007d0f RBX: ffff96848811cae0 RCX: 0000000000000001 [ 8196.205327] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 8196.216253] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8196.221103] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 8196.227775] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 8196.235280] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 8196.243410] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8196.251878] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 8196.258591] Call Trace: [ 8196.261543] [ 8196.264566] ? show_regs.cold.9+0x22/0x2f [ 8196.269222] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8196.274463] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8196.277530] ? watchdog+0x40/0x40 [ 8196.286225] ? hrtimer_interrupt+0x128/0x2c0 [ 8196.290478] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8196.298203] ? apic_timer_interrupt+0xf/0x20 [ 8196.300641] [ 8196.301465] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8196.308689] do_raw_spin_lock+0x177/0x190 [ 8196.312697] _raw_spin_lock+0x21/0x30 [ 8196.318720] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8196.323187] ? do_raw_spin_unlock+0x75/0x190 [ 8196.325489] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8196.333313] ? do_wait_intr+0xf0/0xf0 [ 8196.343716] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8196.346638] mdc_close+0x53b/0xd70 [mdc] [ 8196.358417] lmv_close+0x18d/0x490 [lmv] [ 8196.360096] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 8196.364069] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8196.371265] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8196.375601] ll_md_close+0x274/0x800 [lustre] [ 8196.379404] ll_file_release+0x128/0xc50 [lustre] [ 8196.383089] __fput+0x10d/0x440 [ 8196.385678] ____fput+0x12/0x20 [ 8196.390566] task_work_run+0xc6/0x110 [ 8196.395969] exit_to_usermode_loop+0x1dd/0x1f0 [ 8196.400935] do_syscall_64+0x3d6/0x440 [ 8196.409471] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8196.413106] RIP: 0033:0x7ff54af45cd8 [ 8196.418214] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8196.432332] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8196.438379] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 8196.445908] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 8196.450232] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 8196.461434] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 8196.469579] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8208.059126] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 8208.062984] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8208.084578] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8208.090456] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8208.095315] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8208.098395] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8208.100418] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8208.107936] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8208.111626] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 8208.115868] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 8208.119929] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 8208.123100] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 8208.126090] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 8208.129138] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 8208.132461] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8208.134999] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 8208.139161] Call Trace: [ 8208.140442] [ 8208.141512] ? show_regs.cold.9+0x22/0x2f [ 8208.143981] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8208.146153] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8208.148204] ? watchdog+0x40/0x40 [ 8208.149458] ? hrtimer_interrupt+0x128/0x2c0 [ 8208.151432] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8208.154167] ? apic_timer_interrupt+0xf/0x20 [ 8208.156158] [ 8208.157496] ? native_safe_halt+0x2e/0x50 [ 8208.159867] kvm_wait+0xcd/0xe0 [ 8208.161691] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8208.163904] do_raw_spin_lock+0x177/0x190 [ 8208.165578] _raw_spin_lock+0x21/0x30 [ 8208.167265] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8208.169556] process_one_work+0x2c8/0x700 [ 8208.171590] worker_thread+0x296/0x6e0 [ 8208.172824] ? rescuer_thread+0x570/0x570 [ 8208.174481] kthread+0x1d1/0x200 [ 8208.175943] ? set_kthread_struct+0x70/0x70 [ 8208.177759] ret_from_fork+0x1f/0x30 [ 8216.062913] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 8216.079221] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8216.111021] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8216.122899] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8216.130600] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8216.138658] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8216.155480] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8216.159874] RAX: 0000000000003084 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 8216.163362] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 8216.170531] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8216.180988] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 8216.186026] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 8216.190293] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 8216.194794] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8216.197588] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 8216.206830] Call Trace: [ 8216.208848] [ 8216.209784] ? show_regs.cold.9+0x22/0x2f [ 8216.211692] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8216.214721] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8216.220106] ? watchdog+0x40/0x40 [ 8216.224132] ? hrtimer_interrupt+0x128/0x2c0 [ 8216.226422] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8216.232128] ? apic_timer_interrupt+0xf/0x20 [ 8216.240041] [ 8216.241693] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8216.248132] do_raw_spin_lock+0x177/0x190 [ 8216.250459] _raw_spin_lock+0x21/0x30 [ 8216.252656] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8216.256146] ? do_raw_spin_unlock+0x75/0x190 [ 8216.265077] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8216.272862] ? do_wait_intr+0xf0/0xf0 [ 8216.276016] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8216.289968] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8216.300836] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8216.303938] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8216.308974] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8216.324382] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8216.332485] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8216.335734] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8216.339833] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8216.344534] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8216.352940] lmv_intent_open+0x338/0x1150 [lmv] [ 8216.360552] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8216.362323] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8216.380850] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8216.392417] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8216.395734] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8216.401962] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8216.407007] ? do_raw_spin_unlock+0x75/0x190 [ 8216.418713] ? _raw_spin_unlock+0x12/0x30 [ 8216.431568] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8216.433350] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8216.436609] ll_atomic_open+0x36a/0xd70 [lustre] [ 8216.447651] lookup_open+0x356/0xb40 [ 8216.452244] do_last+0x313/0xfc0 [ 8216.458157] ? nd_jump_root+0xe5/0x160 [ 8216.459345] ? path_init+0x437/0x520 [ 8216.460577] path_openat+0xf7/0x500 [ 8216.474132] ? do_raw_spin_unlock+0x75/0x190 [ 8216.478511] do_filp_open+0x99/0x140 [ 8216.482868] ? getname_flags+0x6e/0x330 [ 8216.488893] ? __check_object_size+0xff/0x256 [ 8216.494490] ? do_raw_spin_unlock+0x75/0x190 [ 8216.496004] ? _raw_spin_unlock+0x12/0x30 [ 8216.522789] do_sys_openat2+0x2b4/0x410 [ 8216.529855] do_sys_open+0x73/0xa0 [ 8216.535672] __x64_sys_openat+0x24/0x30 [ 8216.536833] do_syscall_64+0xc1/0x440 [ 8216.545270] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8216.550885] RIP: 0033:0x7ff54af45332 [ 8216.553997] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8216.568493] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8216.573541] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 8216.580524] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 8216.591350] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 8216.606809] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 8216.613853] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 8220.060796] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 8220.072609] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8220.171901] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8220.188262] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8220.204509] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8220.212028] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8220.235822] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8220.243651] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 8220.251162] RDX: 000000000000254f RSI: 0000000000000003 RDI: ffff96848811cae0 [ 8220.264895] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 8220.269111] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 8220.277445] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8220.280965] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 8220.304597] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8220.309793] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 8220.314446] Call Trace: [ 8220.315865] [ 8220.317149] ? show_regs.cold.9+0x22/0x2f [ 8220.320049] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8220.322265] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8220.325625] ? watchdog+0x40/0x40 [ 8220.327974] ? hrtimer_interrupt+0x128/0x2c0 [ 8220.330445] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8220.332972] ? apic_timer_interrupt+0xf/0x20 [ 8220.334777] [ 8220.335722] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8220.338098] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8220.341122] do_raw_spin_lock+0x177/0x190 [ 8220.343204] _raw_spin_lock+0x21/0x30 [ 8220.344637] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8220.348901] obd_mod_free+0x46/0x140 [ptlrpc] [ 8220.354257] mdc_commit_open+0x91/0xb0 [mdc] [ 8220.357436] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8220.361076] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8220.364466] after_reply+0xe2c/0x1780 [ptlrpc] [ 8220.367786] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8220.372539] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8220.377066] ? do_wait_intr+0xf0/0xf0 [ 8220.378988] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8220.381577] kthread+0x1d1/0x200 [ 8220.383230] ? set_kthread_struct+0x70/0x70 [ 8220.385437] ret_from_fork+0x1f/0x30 [ 8224.058725] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [ 8224.067011] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8224.137393] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8224.145827] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8224.158363] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8224.160125] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8224.172410] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8224.180380] RAX: 0000000000006785 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 8224.192513] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 8224.195025] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8224.205243] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 8224.213312] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 8224.221628] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 8224.229042] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8224.240976] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 8224.252589] Call Trace: [ 8224.257603] [ 8224.259519] ? show_regs.cold.9+0x22/0x2f [ 8224.269297] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8224.278898] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8224.288357] ? watchdog+0x40/0x40 [ 8224.294217] ? hrtimer_interrupt+0x128/0x2c0 [ 8224.301293] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8224.303141] ? apic_timer_interrupt+0xf/0x20 [ 8224.318150] [ 8224.318947] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8224.323233] do_raw_spin_lock+0x177/0x190 [ 8224.327943] _raw_spin_lock+0x21/0x30 [ 8224.341209] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8224.343411] ? do_raw_spin_unlock+0x75/0x190 [ 8224.348575] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8224.350040] ? do_wait_intr+0xf0/0xf0 [ 8224.363421] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8224.372940] mdc_close+0x53b/0xd70 [mdc] [ 8224.382637] lmv_close+0x18d/0x490 [lmv] [ 8224.392600] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 8224.405232] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8224.411421] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8224.424614] ll_md_close+0x274/0x800 [lustre] [ 8224.425852] ll_file_release+0x128/0xc50 [lustre] [ 8224.434643] __fput+0x10d/0x440 [ 8224.441875] ____fput+0x12/0x20 [ 8224.444247] task_work_run+0xc6/0x110 [ 8224.450477] exit_to_usermode_loop+0x1dd/0x1f0 [ 8224.455397] do_syscall_64+0x3d6/0x440 [ 8224.458963] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8224.464835] RIP: 0033:0x7ff54af45cd8 [ 8224.466447] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8224.474265] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8224.480418] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 8224.485715] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 8224.488438] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 8224.497186] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 8224.500585] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8225.247527] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6266s! [ 8225.256742] Showing busy workqueues and worker pools: [ 8225.261120] workqueue events: flags=0x0 [ 8225.263214] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8225.266824] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 8225.274254] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8225.278688] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 8225.290829] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8225.295125] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 8225.298489] workqueue events_long: flags=0x0 [ 8225.300316] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8225.308993] pending: lu_objects_flush [obdclass] [ 8225.313893] workqueue events_unbound: flags=0x2 [ 8225.316032] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8225.319117] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8225.323460] workqueue events_power_efficient: flags=0x80 [ 8225.327263] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8225.338095] pending: neigh_periodic_work [ 8225.342216] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8225.346982] pending: do_cache_clean, check_lifetime [ 8225.352820] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8225.356263] pending: neigh_periodic_work [ 8225.359009] workqueue writeback: flags=0x4a [ 8225.360959] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8225.362687] pending: wb_workfn, wb_workfn [ 8225.364257] workqueue xprtiod: flags=0xa [ 8225.365737] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8225.373420] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 8225.378715] workqueue ipv6_addrconf: flags=0x40008 [ 8225.381833] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8225.387418] pending: addrconf_verify_work [ 8225.394411] workqueue knbd0-recv: flags=0x1a [ 8225.397832] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8225.399384] in-flight: 657:recv_work [ 8225.404654] workqueue ptlrpc_pinger: flags=0x2 [ 8225.408363] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8225.415911] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 8225.419937] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6266s workers=4 idle: 20831 12642 28859 [ 8225.423177] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8215s workers=2 idle: 659 [ 8236.059101] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 8236.065158] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8236.129634] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8236.138182] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8236.146157] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8236.151415] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8236.153580] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8236.160904] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8236.163418] RAX: 0000000000005a13 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 8236.166014] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 8236.169209] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 8236.171618] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 8236.174421] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 8236.177253] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 8236.180953] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8236.183471] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 8236.186692] Call Trace: [ 8236.187690] [ 8236.188838] ? show_regs.cold.9+0x22/0x2f [ 8236.190311] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8236.192258] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8236.194175] ? watchdog+0x40/0x40 [ 8236.195526] ? hrtimer_interrupt+0x128/0x2c0 [ 8236.197219] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8236.199152] ? apic_timer_interrupt+0xf/0x20 [ 8236.200692] [ 8236.201603] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8236.203715] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8236.205893] do_raw_spin_lock+0x177/0x190 [ 8236.207467] _raw_spin_lock+0x21/0x30 [ 8236.208740] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8236.210686] process_one_work+0x2c8/0x700 [ 8236.212092] worker_thread+0x296/0x6e0 [ 8236.213623] ? rescuer_thread+0x570/0x570 [ 8236.215076] kthread+0x1d1/0x200 [ 8236.216318] ? set_kthread_struct+0x70/0x70 [ 8236.218125] ret_from_fork+0x1f/0x30 [ 8244.060498] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 8244.062555] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8244.108639] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8244.119770] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8244.134637] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8244.140939] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8244.160005] RSP: 0000:ffffaf9fc15234f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8244.166194] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 8244.177112] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 8244.190274] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8244.200518] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 8244.211323] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 8244.218173] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 8244.227998] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8244.236519] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 8244.245364] Call Trace: [ 8244.247496] [ 8244.250024] ? show_regs.cold.9+0x22/0x2f [ 8244.251537] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8244.264014] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8244.273156] ? watchdog+0x40/0x40 [ 8244.276857] ? hrtimer_interrupt+0x128/0x2c0 [ 8244.278335] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8244.284566] ? apic_timer_interrupt+0xf/0x20 [ 8244.288768] [ 8244.291367] ? native_safe_halt+0x2e/0x50 [ 8244.299763] kvm_wait+0xcd/0xe0 [ 8244.303633] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8244.307769] do_raw_spin_lock+0x177/0x190 [ 8244.311697] _raw_spin_lock+0x21/0x30 [ 8244.313780] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8244.319584] ? do_raw_spin_unlock+0x75/0x190 [ 8244.321049] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8244.327752] ? do_wait_intr+0xf0/0xf0 [ 8244.332646] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8244.336686] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8244.339254] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8244.346198] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8244.352242] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8244.354019] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8244.357614] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8244.366070] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8244.375903] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8244.388399] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8244.395614] lmv_intent_open+0x338/0x1150 [lmv] [ 8244.398817] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8244.407088] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8244.408894] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8244.415561] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8244.427044] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8244.429065] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8244.436515] ? do_raw_spin_unlock+0x75/0x190 [ 8244.443386] ? _raw_spin_unlock+0x12/0x30 [ 8244.447437] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8244.452728] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8244.457129] ll_atomic_open+0x36a/0xd70 [lustre] [ 8244.459635] lookup_open+0x356/0xb40 [ 8244.464099] do_last+0x313/0xfc0 [ 8244.465808] ? nd_jump_root+0xe5/0x160 [ 8244.468579] ? path_init+0x437/0x520 [ 8244.471760] path_openat+0xf7/0x500 [ 8244.475597] ? do_raw_spin_unlock+0x75/0x190 [ 8244.479809] do_filp_open+0x99/0x140 [ 8244.485625] ? getname_flags+0x6e/0x330 [ 8244.490850] ? __check_object_size+0xff/0x256 [ 8244.496795] ? do_raw_spin_unlock+0x75/0x190 [ 8244.503516] ? _raw_spin_unlock+0x12/0x30 [ 8244.505644] do_sys_openat2+0x2b4/0x410 [ 8244.507605] do_sys_open+0x73/0xa0 [ 8244.509196] __x64_sys_openat+0x24/0x30 [ 8244.511163] do_syscall_64+0xc1/0x440 [ 8244.513120] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8244.517754] RIP: 0033:0x7ff54af45332 [ 8244.521897] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8244.540411] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8244.552240] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 8244.561306] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 8244.565588] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 8244.574636] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 8244.585440] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 8252.058152] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [ 8252.059194] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [ 8252.077107] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8252.077186] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8252.077194] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8252.077198] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8252.077212] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8252.077217] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8252.077221] RAX: 0000000000005148 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 8252.077224] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 8252.077225] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8252.077228] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 8252.077230] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 8252.077233] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 8252.077235] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8252.077237] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 8252.077246] Call Trace: [ 8252.077251] [ 8252.077257] ? show_regs.cold.9+0x22/0x2f [ 8252.077266] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8252.077273] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8252.077278] ? watchdog+0x40/0x40 [ 8252.077282] ? hrtimer_interrupt+0x128/0x2c0 [ 8252.077286] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8252.077294] ? apic_timer_interrupt+0xf/0x20 [ 8252.077298] [ 8252.077300] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8252.077308] do_raw_spin_lock+0x177/0x190 [ 8252.077314] _raw_spin_lock+0x21/0x30 [ 8252.077318] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8252.077673] ? do_raw_spin_unlock+0x75/0x190 [ 8252.077679] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8252.082759] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) [ 8252.198563] ? do_wait_intr+0xf0/0xf0 [ 8252.198586] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8252.218535] lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8252.218602] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8252.218609] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8252.218612] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8252.218628] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8252.218632] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8252.218636] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 8252.218639] RDX: 00000000000073cb RSI: 0000000000000003 RDI: ffff96848811cae0 [ 8252.218641] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 8252.218643] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 8252.218646] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8252.218649] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 8252.218651] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8252.218654] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 8252.218662] Call Trace: [ 8252.218668] [ 8252.218674] ? show_regs.cold.9+0x22/0x2f [ 8252.218683] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8252.218689] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8252.218692] ? watchdog+0x40/0x40 [ 8252.218696] ? hrtimer_interrupt+0x128/0x2c0 [ 8252.218700] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8252.218709] ? apic_timer_interrupt+0xf/0x20 [ 8252.218714] [ 8252.218717] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8252.218721] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8252.218736] do_raw_spin_lock+0x177/0x190 [ 8252.218742] _raw_spin_lock+0x21/0x30 [ 8252.218757] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8252.219449] obd_mod_free+0x46/0x140 [ptlrpc] [ 8252.219680] mdc_commit_open+0x91/0xb0 [mdc] [ 8252.220463] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8252.220672] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8252.244138] mdc_close+0x53b/0xd70 [mdc] [ 8252.244262] lmv_close+0x18d/0x490 [lmv] [ 8252.244297] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 8252.252445] after_reply+0xe2c/0x1780 [ptlrpc] [ 8252.252755] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8252.253507] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8252.278467] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8252.278602] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8252.278683] ll_md_close+0x274/0x800 [lustre] [ 8252.278776] ll_file_release+0x128/0xc50 [lustre] [ 8252.279323] __fput+0x10d/0x440 [ 8252.279332] ____fput+0x12/0x20 [ 8252.279337] task_work_run+0xc6/0x110 [ 8252.279344] exit_to_usermode_loop+0x1dd/0x1f0 [ 8252.279350] do_syscall_64+0x3d6/0x440 [ 8252.279354] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8252.279363] RIP: 0033:0x7ff54af45cd8 [ 8252.279370] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8252.279374] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8252.279391] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 8252.279394] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 8252.279396] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 8252.279398] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 8252.279400] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8252.703578] ? do_wait_intr+0xf0/0xf0 [ 8252.705455] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8252.707917] kthread+0x1d1/0x200 [ 8252.709844] ? set_kthread_struct+0x70/0x70 [ 8252.712353] ret_from_fork+0x1f/0x30 [ 8255.456340] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6296s! [ 8255.460666] Showing busy workqueues and worker pools: [ 8255.467412] workqueue events: flags=0x0 [ 8255.468857] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8255.479677] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 8255.486039] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8255.496783] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 8255.504736] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8255.508235] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 8255.511942] workqueue events_long: flags=0x0 [ 8255.514341] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8255.519007] pending: lu_objects_flush [obdclass] [ 8255.522370] workqueue events_unbound: flags=0x2 [ 8255.528260] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8255.532741] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8255.540748] workqueue events_power_efficient: flags=0x80 [ 8255.546728] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8255.551321] pending: neigh_periodic_work [ 8255.556171] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8255.561002] pending: do_cache_clean, check_lifetime [ 8255.564791] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8255.567238] pending: neigh_periodic_work [ 8255.569114] workqueue writeback: flags=0x4a [ 8255.571014] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8255.573813] pending: wb_workfn, wb_workfn [ 8255.575787] workqueue xprtiod: flags=0xa [ 8255.577661] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8255.580120] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 8255.584048] workqueue ipv6_addrconf: flags=0x40008 [ 8255.586334] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8255.589092] pending: addrconf_verify_work [ 8255.591126] workqueue knbd0-recv: flags=0x1a [ 8255.592892] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8255.595704] in-flight: 657:recv_work [ 8255.597379] workqueue ptlrpc_pinger: flags=0x2 [ 8255.599295] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8255.601894] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 8255.604540] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6296s workers=4 idle: 20831 12642 28859 [ 8255.608088] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8245s workers=2 idle: 659 [ 8264.060930] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 8264.068557] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8264.155384] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8264.176403] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8264.201462] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8264.226514] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8264.242097] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8264.280409] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8264.292415] RAX: 0000000000007f0c RBX: ffff96848811cae0 RCX: 0000000000000001 [ 8264.305642] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 8264.320923] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 8264.335328] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 8264.356410] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 8264.363354] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 8264.373635] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8264.384352] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 8264.395234] Call Trace: [ 8264.396084] [ 8264.396778] ? show_regs.cold.9+0x22/0x2f [ 8264.398240] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8264.412601] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8264.420489] ? watchdog+0x40/0x40 [ 8264.423760] ? hrtimer_interrupt+0x128/0x2c0 [ 8264.431649] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8264.439759] ? apic_timer_interrupt+0xf/0x20 [ 8264.449020] [ 8264.451750] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8264.464549] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8264.469129] do_raw_spin_lock+0x177/0x190 [ 8264.470427] _raw_spin_lock+0x21/0x30 [ 8264.472970] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8264.475588] process_one_work+0x2c8/0x700 [ 8264.477771] worker_thread+0x296/0x6e0 [ 8264.483501] ? rescuer_thread+0x570/0x570 [ 8264.489724] kthread+0x1d1/0x200 [ 8264.495519] ? set_kthread_struct+0x70/0x70 [ 8264.500306] ret_from_fork+0x1f/0x30 [ 8272.062267] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 8272.068978] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8272.092024] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8272.101676] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8272.107599] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8272.114761] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8272.135271] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8272.152043] RAX: 0000000000003982 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 8272.160937] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 8272.169712] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8272.180892] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 8272.191675] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 8272.200759] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 8272.212802] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8272.218998] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 8272.223923] Call Trace: [ 8272.225759] [ 8272.226869] ? show_regs.cold.9+0x22/0x2f [ 8272.231620] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8272.236886] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8272.240773] ? watchdog+0x40/0x40 [ 8272.243630] ? hrtimer_interrupt+0x128/0x2c0 [ 8272.246152] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8272.249526] ? apic_timer_interrupt+0xf/0x20 [ 8272.258056] [ 8272.258965] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8272.260908] do_raw_spin_lock+0x177/0x190 [ 8272.270022] _raw_spin_lock+0x21/0x30 [ 8272.275001] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8272.280811] ? do_raw_spin_unlock+0x75/0x190 [ 8272.284282] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8272.289660] ? do_wait_intr+0xf0/0xf0 [ 8272.294902] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8272.303908] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8272.311848] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8272.318169] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8272.321843] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8272.326608] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8272.329777] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8272.333356] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8272.336870] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8272.341407] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8272.345861] lmv_intent_open+0x338/0x1150 [lmv] [ 8272.350391] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8272.356752] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8272.370325] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8272.373260] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8272.377029] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8272.383586] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8272.390569] ? do_raw_spin_unlock+0x75/0x190 [ 8272.394881] ? _raw_spin_unlock+0x12/0x30 [ 8272.400845] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8272.405740] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8272.408903] ll_atomic_open+0x36a/0xd70 [lustre] [ 8272.412946] lookup_open+0x356/0xb40 [ 8272.415115] do_last+0x313/0xfc0 [ 8272.416866] ? nd_jump_root+0xe5/0x160 [ 8272.422307] ? path_init+0x437/0x520 [ 8272.425276] path_openat+0xf7/0x500 [ 8272.427582] ? do_raw_spin_unlock+0x75/0x190 [ 8272.434070] do_filp_open+0x99/0x140 [ 8272.435285] ? getname_flags+0x6e/0x330 [ 8272.436547] ? __check_object_size+0xff/0x256 [ 8272.441678] ? do_raw_spin_unlock+0x75/0x190 [ 8272.446540] ? _raw_spin_unlock+0x12/0x30 [ 8272.450023] do_sys_openat2+0x2b4/0x410 [ 8272.463445] do_sys_open+0x73/0xa0 [ 8272.464785] __x64_sys_openat+0x24/0x30 [ 8272.475694] do_syscall_64+0xc1/0x440 [ 8272.481673] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8272.493278] RIP: 0033:0x7ff54af45332 [ 8272.498054] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8272.526039] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8272.534579] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 8272.547593] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 8272.552272] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 8272.563662] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 8272.575050] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 8280.055024] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 8280.057872] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8280.063599] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 8280.115470] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8280.115484] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8280.115488] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8280.115500] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8280.115504] RSP: 0000:ffffaf9fc1b97bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8280.115510] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 8280.115512] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 8280.115514] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 8280.115516] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 8280.115518] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8280.115522] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 8280.115525] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8280.115527] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 8280.115536] Call Trace: [ 8280.115541] [ 8280.115547] ? show_regs.cold.9+0x22/0x2f [ 8280.115558] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8280.115565] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8280.115570] ? watchdog+0x40/0x40 [ 8280.115574] ? hrtimer_interrupt+0x128/0x2c0 [ 8280.115579] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8280.115587] ? apic_timer_interrupt+0xf/0x20 [ 8280.115591] [ 8280.115594] ? native_safe_halt+0x2e/0x50 [ 8280.115598] kvm_wait+0xcd/0xe0 [ 8280.115606] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8280.115613] do_raw_spin_lock+0x177/0x190 [ 8280.115618] _raw_spin_lock+0x21/0x30 [ 8280.115624] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8280.123700] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8280.123779] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8280.123787] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8280.123790] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8280.123805] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8280.123809] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8280.123814] RAX: 0000000000007d07 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 8280.123816] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 8280.123819] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8280.123821] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 8280.123823] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 8280.123826] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 8280.123829] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8280.123832] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 8280.123841] Call Trace: [ 8280.123846] [ 8280.123852] ? show_regs.cold.9+0x22/0x2f [ 8280.123861] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8280.123867] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8280.123871] ? watchdog+0x40/0x40 [ 8280.123875] ? hrtimer_interrupt+0x128/0x2c0 [ 8280.123879] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8280.123886] ? apic_timer_interrupt+0xf/0x20 [ 8280.123891] [ 8280.123894] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8280.123901] do_raw_spin_lock+0x177/0x190 [ 8280.123906] _raw_spin_lock+0x21/0x30 [ 8280.123911] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8280.124731] ? do_raw_spin_unlock+0x75/0x190 [ 8280.124739] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8280.125122] ? do_wait_intr+0xf0/0xf0 [ 8280.125131] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8280.125613] mdc_close+0x53b/0xd70 [mdc] [ 8280.125653] lmv_close+0x18d/0x490 [lmv] [ 8280.125683] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 8280.125847] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8280.131660] obd_mod_free+0x46/0x140 [ptlrpc] [ 8280.132011] mdc_commit_open+0x91/0xb0 [mdc] [ 8280.132042] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8280.132198] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8280.132365] after_reply+0xe2c/0x1780 [ptlrpc] [ 8280.132492] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8280.132648] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8280.132852] ? do_wait_intr+0xf0/0xf0 [ 8280.133191] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8280.133337] kthread+0x1d1/0x200 [ 8280.133341] ? set_kthread_struct+0x70/0x70 [ 8280.133344] ret_from_fork+0x1f/0x30 [ 8280.940875] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8280.951359] ll_md_close+0x274/0x800 [lustre] [ 8280.957808] ll_file_release+0x128/0xc50 [lustre] [ 8280.963652] __fput+0x10d/0x440 [ 8280.968269] ____fput+0x12/0x20 [ 8280.972984] task_work_run+0xc6/0x110 [ 8280.975298] exit_to_usermode_loop+0x1dd/0x1f0 [ 8280.988127] do_syscall_64+0x3d6/0x440 [ 8280.991728] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8280.998735] RIP: 0033:0x7ff54af45cd8 [ 8281.001184] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8281.026398] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8281.035699] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 8281.050456] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 8281.059777] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 8281.076579] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 8281.087246] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8285.676361] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6326s! [ 8285.680270] Showing busy workqueues and worker pools: [ 8285.685415] workqueue events: flags=0x0 [ 8285.695010] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8285.697476] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 8285.711258] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8285.719043] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 8285.728418] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8285.734182] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 8285.744809] workqueue events_long: flags=0x0 [ 8285.749778] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8285.756285] pending: lu_objects_flush [obdclass] [ 8285.761427] workqueue events_unbound: flags=0x2 [ 8285.764389] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8285.770908] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8285.781670] workqueue events_power_efficient: flags=0x80 [ 8285.785367] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8285.793426] pending: neigh_periodic_work [ 8285.798674] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8285.804422] pending: do_cache_clean, check_lifetime [ 8285.813151] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8285.821587] pending: neigh_periodic_work [ 8285.823761] workqueue writeback: flags=0x4a [ 8285.826150] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8285.829576] pending: wb_workfn, wb_workfn [ 8285.831809] workqueue xprtiod: flags=0xa [ 8285.833868] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8285.839470] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 8285.849660] workqueue ipv6_addrconf: flags=0x40008 [ 8285.853270] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8285.859380] pending: addrconf_verify_work [ 8285.863760] workqueue knbd0-recv: flags=0x1a [ 8285.872216] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8285.882368] in-flight: 657:recv_work [ 8285.889771] workqueue ptlrpc_pinger: flags=0x2 [ 8285.897411] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8285.907887] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 8285.916065] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6327s workers=4 idle: 20831 12642 28859 [ 8285.923874] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8276s workers=2 idle: 659 [ 8292.059937] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 8292.067527] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8292.120076] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8292.140260] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8292.146950] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8292.151459] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8292.154450] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8292.183088] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8292.190846] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 8292.199610] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 8292.211520] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 8292.222124] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 8292.233899] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 8292.249495] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 8292.257798] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8292.261139] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 8292.270302] Call Trace: [ 8292.273922] [ 8292.279221] ? show_regs.cold.9+0x22/0x2f [ 8292.284357] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8292.291401] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8292.297087] ? watchdog+0x40/0x40 [ 8292.299837] ? hrtimer_interrupt+0x128/0x2c0 [ 8292.305179] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8292.310194] ? apic_timer_interrupt+0xf/0x20 [ 8292.316723] [ 8292.319078] ? native_safe_halt+0x2e/0x50 [ 8292.326173] kvm_wait+0xcd/0xe0 [ 8292.329598] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8292.333270] do_raw_spin_lock+0x177/0x190 [ 8292.341139] _raw_spin_lock+0x21/0x30 [ 8292.343846] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8292.345904] process_one_work+0x2c8/0x700 [ 8292.349638] worker_thread+0x296/0x6e0 [ 8292.355274] ? rescuer_thread+0x570/0x570 [ 8292.357439] kthread+0x1d1/0x200 [ 8292.358382] ? set_kthread_struct+0x70/0x70 [ 8292.372780] ret_from_fork+0x1f/0x30 [ 8300.061467] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [ 8300.066557] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8300.108696] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8300.116537] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8300.123211] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8300.128515] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8300.139883] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8300.145122] RAX: 0000000000004b17 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 8300.151147] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 8300.159358] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8300.166456] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 8300.170395] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 8300.180882] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 8300.188097] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8300.192321] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 8300.196730] Call Trace: [ 8300.198298] [ 8300.199620] ? show_regs.cold.9+0x22/0x2f [ 8300.207760] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8300.217156] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8300.222572] ? watchdog+0x40/0x40 [ 8300.225551] ? hrtimer_interrupt+0x128/0x2c0 [ 8300.229089] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8300.232723] ? apic_timer_interrupt+0xf/0x20 [ 8300.238335] [ 8300.239559] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8300.245412] do_raw_spin_lock+0x177/0x190 [ 8300.248081] _raw_spin_lock+0x21/0x30 [ 8300.250069] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8300.253389] ? do_raw_spin_unlock+0x75/0x190 [ 8300.256100] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8300.259580] ? do_wait_intr+0xf0/0xf0 [ 8300.261542] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8300.264634] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8300.269089] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8300.272589] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8300.276078] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8300.281174] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8300.284317] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8300.287567] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8300.294601] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8300.299347] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8300.301481] lmv_intent_open+0x338/0x1150 [lmv] [ 8300.303415] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8300.305153] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8300.306848] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8300.308287] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8300.309620] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8300.311468] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8300.313078] ? do_raw_spin_unlock+0x75/0x190 [ 8300.314183] ? _raw_spin_unlock+0x12/0x30 [ 8300.315216] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8300.316553] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8300.318074] ll_atomic_open+0x36a/0xd70 [lustre] [ 8300.320690] lookup_open+0x356/0xb40 [ 8300.325577] do_last+0x313/0xfc0 [ 8300.333040] ? nd_jump_root+0xe5/0x160 [ 8300.338057] ? path_init+0x437/0x520 [ 8300.341601] path_openat+0xf7/0x500 [ 8300.345689] ? do_raw_spin_unlock+0x75/0x190 [ 8300.359195] do_filp_open+0x99/0x140 [ 8300.368407] ? getname_flags+0x6e/0x330 [ 8300.374323] ? __check_object_size+0xff/0x256 [ 8300.384861] ? do_raw_spin_unlock+0x75/0x190 [ 8300.395392] ? _raw_spin_unlock+0x12/0x30 [ 8300.408197] do_sys_openat2+0x2b4/0x410 [ 8300.415498] do_sys_open+0x73/0xa0 [ 8300.428715] __x64_sys_openat+0x24/0x30 [ 8300.437624] do_syscall_64+0xc1/0x440 [ 8300.446689] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8300.456657] RIP: 0033:0x7ff54af45332 [ 8300.465068] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8300.479285] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8300.482905] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 8300.489522] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 8300.495517] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 8300.498423] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 8300.502535] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 8308.058042] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 8308.062947] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8308.113161] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8308.124048] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8308.130639] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8308.132226] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8308.141794] RSP: 0018:ffffaf9fc1543ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8308.151920] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 8308.161182] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 8308.172063] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8308.177089] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 8308.180895] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 8308.184173] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 8308.190099] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8308.194822] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 8308.199023] Call Trace: [ 8308.200957] [ 8308.204201] ? show_regs.cold.9+0x22/0x2f [ 8308.208368] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8308.216493] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8308.223929] ? watchdog+0x40/0x40 [ 8308.228974] ? hrtimer_interrupt+0x128/0x2c0 [ 8308.236523] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8308.243812] ? apic_timer_interrupt+0xf/0x20 [ 8308.246774] [ 8308.248022] ? native_safe_halt+0x2e/0x50 [ 8308.249580] kvm_wait+0xcd/0xe0 [ 8308.250745] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8308.254197] do_raw_spin_lock+0x177/0x190 [ 8308.256434] _raw_spin_lock+0x21/0x30 [ 8308.258531] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8308.261356] ? do_raw_spin_unlock+0x75/0x190 [ 8308.263075] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8308.266227] ? do_wait_intr+0xf0/0xf0 [ 8308.268016] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8308.275054] mdc_close+0x53b/0xd70 [mdc] [ 8308.277460] lmv_close+0x18d/0x490 [lmv] [ 8308.279580] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 8308.282444] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8308.287127] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8308.291998] ll_md_close+0x274/0x800 [lustre] [ 8308.295357] ll_file_release+0x128/0xc50 [lustre] [ 8308.298426] __fput+0x10d/0x440 [ 8308.299939] ____fput+0x12/0x20 [ 8308.301710] task_work_run+0xc6/0x110 [ 8308.303081] exit_to_usermode_loop+0x1dd/0x1f0 [ 8308.304640] do_syscall_64+0x3d6/0x440 [ 8308.309763] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8308.312066] RIP: 0033:0x7ff54af45cd8 [ 8308.314007] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8308.325597] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8308.329104] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 8308.332302] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 8308.335772] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 8308.342804] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 8308.346143] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8312.055963] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [ 8312.058870] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8312.073227] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8312.076533] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8312.079143] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8312.080845] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8312.085878] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8312.091421] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 8312.093459] RDX: 0000000000005725 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 8312.097547] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 8312.099455] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 8312.101398] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8312.103269] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 8312.105988] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8312.107417] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 8312.111577] Call Trace: [ 8312.112361] [ 8312.113334] ? show_regs.cold.9+0x22/0x2f [ 8312.114374] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8312.115533] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8312.116721] ? watchdog+0x40/0x40 [ 8312.117623] ? hrtimer_interrupt+0x128/0x2c0 [ 8312.118925] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8312.120088] ? apic_timer_interrupt+0xf/0x20 [ 8312.121105] [ 8312.122883] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8312.124431] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8312.125944] do_raw_spin_lock+0x177/0x190 [ 8312.127290] _raw_spin_lock+0x21/0x30 [ 8312.128479] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8312.130657] obd_mod_free+0x46/0x140 [ptlrpc] [ 8312.132290] mdc_commit_open+0x91/0xb0 [mdc] [ 8312.133419] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8312.135302] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8312.136964] after_reply+0xe2c/0x1780 [ptlrpc] [ 8312.138363] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8312.140038] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8312.141450] ? do_wait_intr+0xf0/0xf0 [ 8312.142544] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8312.144238] kthread+0x1d1/0x200 [ 8312.144978] ? set_kthread_struct+0x70/0x70 [ 8312.145941] ret_from_fork+0x1f/0x30 [ 8314.025172] rcu: INFO: rcu_sched self-detected stall on CPU [ 8314.027895] rcu: 0-...!: (3275449 ticks this GP) idle=98a/1/0x4000000000000002 softirq=125154/125154 fqs=959 [ 8314.035464] (t=6360185 jiffies g=253585 q=4910) [ 8314.043546] rcu: rcu_sched kthread starved for 6355145 jiffies! g253585 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [ 8314.052985] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 8314.063744] rcu: RCU grace-period kthread stack dump: [ 8314.073558] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 8314.081001] Call Trace: [ 8314.087001] __schedule+0x351/0xcb0 [ 8314.088475] schedule+0xc0/0x180 [ 8314.096496] schedule_timeout+0xb4/0x190 [ 8314.099482] ? do_raw_spin_unlock+0x75/0x190 [ 8314.104757] ? __next_timer_interrupt+0x160/0x160 [ 8314.107495] ? prepare_to_swait_event+0xf3/0x140 [ 8314.112344] rcu_gp_kthread+0x8dd/0x15b0 [ 8314.113553] ? force_qs_rnp+0x2c0/0x2c0 [ 8314.114749] kthread+0x1d1/0x200 [ 8314.117775] ? set_kthread_struct+0x70/0x70 [ 8314.119113] ret_from_fork+0x1f/0x30 [ 8314.124330] rcu: Stack dump where RCU GP kthread last ran: [ 8314.126311] Sending NMI from CPU 0 to CPUs 2: [ 8314.136328] NMI backtrace for cpu 2 [ 8314.136336] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8314.136337] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8314.136339] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8314.136342] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8314.136344] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8314.136346] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 [ 8314.136348] RAX: 0000000000004d8c RBX: ffff96848811cae0 RCX: 0000000000000001 [ 8314.136350] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 8314.136351] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 8314.136353] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 8314.136354] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 8314.136356] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 8314.136357] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8314.136359] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 8314.136360] Call Trace: [ 8314.136361] [ 8314.136362] ? show_regs.cold.9+0x22/0x2f [ 8314.136363] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 8314.136364] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 8314.136366] ? nmi_handle+0xbd/0x210 [ 8314.136367] ? default_do_nmi+0x4b/0x1b0 [ 8314.136368] ? do_nmi+0x280/0x380 [ 8314.136369] ? end_repeat_nmi+0x16/0x1f [ 8314.136370] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8314.136372] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8314.136373] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8314.136374] [ 8314.136375] do_raw_spin_lock+0x177/0x190 [ 8314.136376] _raw_spin_lock+0x21/0x30 [ 8314.136377] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8314.136379] process_one_work+0x2c8/0x700 [ 8314.136380] worker_thread+0x296/0x6e0 [ 8314.136381] ? rescuer_thread+0x570/0x570 [ 8314.136382] kthread+0x1d1/0x200 [ 8314.136384] ? set_kthread_struct+0x70/0x70 [ 8314.136385] ret_from_fork+0x1f/0x30 [ 8314.139691] NMI backtrace for cpu 0 [ 8314.263595] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8314.267938] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8314.271490] Call Trace: [ 8314.272991] [ 8314.273814] ? dump_stack+0xbb/0x10e [ 8314.275351] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 8314.278177] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 8314.280293] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 8314.282409] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 8314.284492] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 8314.286680] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 8314.288372] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 8314.290888] ? update_process_times+0x6d/0xc0 [ 8314.292397] ? tick_sched_handle.isra.18+0x27/0x90 [ 8314.294236] ? tick_sched_timer+0x53/0xb0 [ 8314.296244] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8314.297946] ? tick_sched_handle.isra.18+0x90/0x90 [ 8314.299793] ? hrtimer_interrupt+0x128/0x2c0 [ 8314.301427] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8314.302885] ? apic_timer_interrupt+0xf/0x20 [ 8314.304173] [ 8314.304801] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8314.306570] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8314.312525] ? do_raw_spin_lock+0x177/0x190 [ 8314.314014] ? _raw_spin_lock+0x21/0x30 [ 8314.317846] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8314.321334] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 8314.323197] ? mdc_commit_open+0x91/0xb0 [mdc] [ 8314.324527] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8314.328596] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8314.333545] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 8314.337704] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8314.340981] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8314.344499] ? do_wait_intr+0xf0/0xf0 [ 8314.345496] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8314.347291] ? kthread+0x1d1/0x200 [ 8314.348789] ? set_kthread_struct+0x70/0x70 [ 8314.349963] ? ret_from_fork+0x1f/0x30 [ 8316.383828] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6357s! [ 8316.395028] Showing busy workqueues and worker pools: [ 8316.400188] workqueue events: flags=0x0 [ 8316.403110] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8316.408036] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 8316.418197] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8316.420965] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 8316.426811] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8316.429148] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 8316.431893] workqueue events_long: flags=0x0 [ 8316.438244] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8316.445216] pending: lu_objects_flush [obdclass] [ 8316.450053] workqueue events_unbound: flags=0x2 [ 8316.451485] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8316.452869] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8316.462803] workqueue events_power_efficient: flags=0x80 [ 8316.464709] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8316.466162] pending: neigh_periodic_work [ 8316.467136] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8316.470153] pending: do_cache_clean, check_lifetime [ 8316.473356] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8316.475980] pending: neigh_periodic_work [ 8316.481030] workqueue writeback: flags=0x4a [ 8316.483078] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8316.485254] pending: wb_workfn, wb_workfn [ 8316.488185] workqueue xprtiod: flags=0xa [ 8316.492518] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8316.494864] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 8316.500031] workqueue ipv6_addrconf: flags=0x40008 [ 8316.503795] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8316.507123] pending: addrconf_verify_work [ 8316.508701] workqueue knbd0-recv: flags=0x1a [ 8316.510581] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8316.511893] in-flight: 657:recv_work [ 8316.512696] workqueue ptlrpc_pinger: flags=0x2 [ 8316.525230] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8316.532520] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 8316.540533] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6357s workers=4 idle: 20831 12642 28859 [ 8316.546589] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8306s workers=2 idle: 659 [ 8328.060662] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [ 8328.065945] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8328.101156] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8328.109604] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8328.117996] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8328.124164] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8328.135195] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8328.139418] RAX: 0000000000007512 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 8328.143910] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 8328.148112] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8328.155077] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 8328.159591] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 8328.167623] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 8328.176064] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8328.178414] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 8328.184461] Call Trace: [ 8328.187573] [ 8328.188924] ? show_regs.cold.9+0x22/0x2f [ 8328.194205] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8328.200312] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8328.207904] ? watchdog+0x40/0x40 [ 8328.212592] ? hrtimer_interrupt+0x128/0x2c0 [ 8328.215852] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8328.217800] ? apic_timer_interrupt+0xf/0x20 [ 8328.219442] [ 8328.221672] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8328.223354] do_raw_spin_lock+0x177/0x190 [ 8328.226133] _raw_spin_lock+0x21/0x30 [ 8328.229885] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8328.234190] ? do_raw_spin_unlock+0x75/0x190 [ 8328.237729] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8328.239260] ? do_wait_intr+0xf0/0xf0 [ 8328.240179] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8328.241303] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8328.244857] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8328.247447] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8328.250224] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8328.251795] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8328.253136] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8328.254526] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8328.256898] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8328.259238] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8328.261349] lmv_intent_open+0x338/0x1150 [lmv] [ 8328.263948] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8328.266081] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8328.268234] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8328.270127] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8328.272020] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8328.274906] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8328.277451] ? do_raw_spin_unlock+0x75/0x190 [ 8328.279724] ? _raw_spin_unlock+0x12/0x30 [ 8328.280942] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8328.288432] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8328.290399] ll_atomic_open+0x36a/0xd70 [lustre] [ 8328.292443] lookup_open+0x356/0xb40 [ 8328.298129] do_last+0x313/0xfc0 [ 8328.299299] ? nd_jump_root+0xe5/0x160 [ 8328.300633] ? path_init+0x437/0x520 [ 8328.302280] path_openat+0xf7/0x500 [ 8328.303891] ? do_raw_spin_unlock+0x75/0x190 [ 8328.306113] do_filp_open+0x99/0x140 [ 8328.308079] ? getname_flags+0x6e/0x330 [ 8328.309287] ? __check_object_size+0xff/0x256 [ 8328.310546] ? do_raw_spin_unlock+0x75/0x190 [ 8328.311710] ? _raw_spin_unlock+0x12/0x30 [ 8328.313250] do_sys_openat2+0x2b4/0x410 [ 8328.314836] do_sys_open+0x73/0xa0 [ 8328.316195] __x64_sys_openat+0x24/0x30 [ 8328.317992] do_syscall_64+0xc1/0x440 [ 8328.319233] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8328.320987] RIP: 0033:0x7ff54af45332 [ 8328.322070] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8328.338633] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8328.341589] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 8328.344324] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 8328.347189] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 8328.350225] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 8328.353668] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 8336.058170] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 8336.066496] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8336.099851] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8336.111383] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8336.116355] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8336.120923] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8336.136220] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8336.141225] RAX: 0000000000004092 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 8336.144982] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 8336.153153] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8336.161611] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 8336.174097] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 8336.184927] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 8336.193981] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8336.205517] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 8336.219807] Call Trace: [ 8336.222853] [ 8336.226307] ? show_regs.cold.9+0x22/0x2f [ 8336.228862] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8336.233979] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8336.235603] ? watchdog+0x40/0x40 [ 8336.236982] ? hrtimer_interrupt+0x128/0x2c0 [ 8336.241765] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8336.246125] ? apic_timer_interrupt+0xf/0x20 [ 8336.249637] [ 8336.253093] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8336.255703] do_raw_spin_lock+0x177/0x190 [ 8336.260729] _raw_spin_lock+0x21/0x30 [ 8336.262734] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8336.268709] ? do_raw_spin_unlock+0x75/0x190 [ 8336.272547] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8336.279811] ? do_wait_intr+0xf0/0xf0 [ 8336.283692] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8336.286804] mdc_close+0x53b/0xd70 [mdc] [ 8336.295812] lmv_close+0x18d/0x490 [lmv] [ 8336.297932] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 8336.301117] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8336.303062] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8336.307527] ll_md_close+0x274/0x800 [lustre] [ 8336.311537] ll_file_release+0x128/0xc50 [lustre] [ 8336.315993] __fput+0x10d/0x440 [ 8336.321228] ____fput+0x12/0x20 [ 8336.323126] task_work_run+0xc6/0x110 [ 8336.328855] exit_to_usermode_loop+0x1dd/0x1f0 [ 8336.332542] do_syscall_64+0x3d6/0x440 [ 8336.335551] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8336.339298] RIP: 0033:0x7ff54af45cd8 [ 8336.341594] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8336.352935] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8336.358482] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 8336.362778] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 8336.368755] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 8336.373055] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 8336.376970] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8340.059031] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 8340.062937] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8340.096064] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8340.101331] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8340.105868] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8340.109484] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8340.112068] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8340.120980] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8340.123697] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 8340.127089] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 8340.130756] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 8340.134476] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 8340.137422] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 8340.140064] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 8340.142868] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8340.145144] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 8340.147980] Call Trace: [ 8340.149037] [ 8340.149740] ? show_regs.cold.9+0x22/0x2f [ 8340.151533] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8340.153198] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8340.155361] ? watchdog+0x40/0x40 [ 8340.156629] ? hrtimer_interrupt+0x128/0x2c0 [ 8340.158441] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8340.160507] ? apic_timer_interrupt+0xf/0x20 [ 8340.162765] [ 8340.163613] ? native_safe_halt+0x2e/0x50 [ 8340.165048] kvm_wait+0xcd/0xe0 [ 8340.166608] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8340.168677] do_raw_spin_lock+0x177/0x190 [ 8340.170278] _raw_spin_lock+0x21/0x30 [ 8340.172054] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8340.174513] process_one_work+0x2c8/0x700 [ 8340.176116] worker_thread+0x296/0x6e0 [ 8340.177677] ? rescuer_thread+0x570/0x570 [ 8340.179479] kthread+0x1d1/0x200 [ 8340.181068] ? set_kthread_struct+0x70/0x70 [ 8340.183190] ret_from_fork+0x1f/0x30 [ 8344.055016] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [ 8344.059221] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8344.112629] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8344.118935] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8344.123257] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8344.126387] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8344.137802] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8344.141481] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 8344.152007] RDX: 0000000000003ff7 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 8344.160495] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 8344.164517] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 8344.168127] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8344.171937] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 8344.176984] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8344.182782] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 8344.186557] Call Trace: [ 8344.188427] [ 8344.190002] ? show_regs.cold.9+0x22/0x2f [ 8344.194091] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8344.198081] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8344.200495] ? watchdog+0x40/0x40 [ 8344.203208] ? hrtimer_interrupt+0x128/0x2c0 [ 8344.206454] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8344.212988] ? apic_timer_interrupt+0xf/0x20 [ 8344.219195] [ 8344.220542] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8344.224011] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8344.227097] do_raw_spin_lock+0x177/0x190 [ 8344.229519] _raw_spin_lock+0x21/0x30 [ 8344.232346] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8344.238076] obd_mod_free+0x46/0x140 [ptlrpc] [ 8344.242280] mdc_commit_open+0x91/0xb0 [mdc] [ 8344.246403] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8344.250222] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8344.255064] after_reply+0xe2c/0x1780 [ptlrpc] [ 8344.258975] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8344.270050] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8344.279023] ? do_wait_intr+0xf0/0xf0 [ 8344.281271] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8344.284468] kthread+0x1d1/0x200 [ 8344.288144] ? set_kthread_struct+0x70/0x70 [ 8344.295225] ret_from_fork+0x1f/0x30 [ 8346.591046] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6387s! [ 8346.595987] Showing busy workqueues and worker pools: [ 8346.597781] workqueue events: flags=0x0 [ 8346.600174] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8346.603543] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 8346.607854] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8346.610807] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 8346.615987] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8346.618818] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 8346.623266] workqueue events_long: flags=0x0 [ 8346.626107] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8346.629613] pending: lu_objects_flush [obdclass] [ 8346.633041] workqueue events_unbound: flags=0x2 [ 8346.634983] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8346.637812] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8346.642890] workqueue events_power_efficient: flags=0x80 [ 8346.648268] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8346.655788] pending: neigh_periodic_work [ 8346.662514] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8346.665395] pending: do_cache_clean, check_lifetime [ 8346.668495] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8346.672124] pending: neigh_periodic_work [ 8346.674927] workqueue writeback: flags=0x4a [ 8346.677114] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8346.680282] pending: wb_workfn, wb_workfn [ 8346.682753] workqueue xprtiod: flags=0xa [ 8346.684677] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8346.688497] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 8346.695772] workqueue ipv6_addrconf: flags=0x40008 [ 8346.699499] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8346.702543] pending: addrconf_verify_work [ 8346.705038] workqueue knbd0-recv: flags=0x1a [ 8346.707533] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8346.710837] in-flight: 657:recv_work [ 8346.712900] workqueue ptlrpc_pinger: flags=0x2 [ 8346.715595] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8346.718574] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 8346.721485] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6387s workers=4 idle: 20831 12642 28859 [ 8346.726204] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8336s workers=2 idle: 659 [ 8356.062092] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [ 8356.072086] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8356.106328] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8356.113263] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8356.118211] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8356.121888] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8356.133912] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8356.146829] RAX: 0000000000007215 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 8356.156866] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 8356.166718] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8356.177357] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 8356.186294] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 8356.189297] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 8356.191031] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8356.195806] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 8356.203821] Call Trace: [ 8356.207969] [ 8356.210332] ? show_regs.cold.9+0x22/0x2f [ 8356.216775] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8356.225796] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8356.232941] ? watchdog+0x40/0x40 [ 8356.238563] ? hrtimer_interrupt+0x128/0x2c0 [ 8356.242611] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8356.246437] ? apic_timer_interrupt+0xf/0x20 [ 8356.249792] [ 8356.251653] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8356.255419] do_raw_spin_lock+0x177/0x190 [ 8356.258696] _raw_spin_lock+0x21/0x30 [ 8356.261784] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8356.269908] ? do_raw_spin_unlock+0x75/0x190 [ 8356.273584] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8356.278809] ? do_wait_intr+0xf0/0xf0 [ 8356.287192] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8356.296976] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8356.307514] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8356.314678] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8356.327001] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8356.328912] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8356.339138] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8356.343377] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8356.347730] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8356.352063] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8356.359320] lmv_intent_open+0x338/0x1150 [lmv] [ 8356.367889] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8356.374916] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8356.378806] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8356.381845] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8356.385295] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8356.389878] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8356.394676] ? do_raw_spin_unlock+0x75/0x190 [ 8356.397717] ? _raw_spin_unlock+0x12/0x30 [ 8356.402116] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8356.405728] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8356.410610] ll_atomic_open+0x36a/0xd70 [lustre] [ 8356.415968] lookup_open+0x356/0xb40 [ 8356.418764] do_last+0x313/0xfc0 [ 8356.421362] ? nd_jump_root+0xe5/0x160 [ 8356.425886] ? path_init+0x437/0x520 [ 8356.428450] path_openat+0xf7/0x500 [ 8356.429998] ? do_raw_spin_unlock+0x75/0x190 [ 8356.434766] do_filp_open+0x99/0x140 [ 8356.440458] ? getname_flags+0x6e/0x330 [ 8356.442273] ? __check_object_size+0xff/0x256 [ 8356.443650] ? do_raw_spin_unlock+0x75/0x190 [ 8356.447867] ? _raw_spin_unlock+0x12/0x30 [ 8356.450278] do_sys_openat2+0x2b4/0x410 [ 8356.452679] do_sys_open+0x73/0xa0 [ 8356.453886] __x64_sys_openat+0x24/0x30 [ 8356.455101] do_syscall_64+0xc1/0x440 [ 8356.456692] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8356.458966] RIP: 0033:0x7ff54af45332 [ 8356.460489] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8356.468024] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8356.471365] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 8356.476331] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 8356.482764] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 8356.485191] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 8356.495748] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 8364.058027] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 8364.064320] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8364.103369] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8364.110491] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8364.116695] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8364.120013] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8364.132181] RSP: 0018:ffffaf9fc1543ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8364.136423] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 8364.140246] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 8364.144302] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8364.148669] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 8364.152944] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 8364.157731] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 8364.162912] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8364.165985] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 8364.170135] Call Trace: [ 8364.171721] [ 8364.172977] ? show_regs.cold.9+0x22/0x2f [ 8364.175204] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8364.178465] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8364.181018] ? watchdog+0x40/0x40 [ 8364.182743] ? hrtimer_interrupt+0x128/0x2c0 [ 8364.185117] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8364.187826] ? apic_timer_interrupt+0xf/0x20 [ 8364.190273] [ 8364.191347] ? native_safe_halt+0x2e/0x50 [ 8364.193460] kvm_wait+0xcd/0xe0 [ 8364.195184] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8364.198007] do_raw_spin_lock+0x177/0x190 [ 8364.200702] _raw_spin_lock+0x21/0x30 [ 8364.202756] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8364.206212] ? do_raw_spin_unlock+0x75/0x190 [ 8364.208726] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8364.212453] ? do_wait_intr+0xf0/0xf0 [ 8364.214858] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8364.218079] mdc_close+0x53b/0xd70 [mdc] [ 8364.220749] lmv_close+0x18d/0x490 [lmv] [ 8364.223055] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 8364.226817] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8364.230356] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8364.233167] ll_md_close+0x274/0x800 [lustre] [ 8364.235794] ll_file_release+0x128/0xc50 [lustre] [ 8364.238817] __fput+0x10d/0x440 [ 8364.240712] ____fput+0x12/0x20 [ 8364.242750] task_work_run+0xc6/0x110 [ 8364.245036] exit_to_usermode_loop+0x1dd/0x1f0 [ 8364.247660] do_syscall_64+0x3d6/0x440 [ 8364.249932] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8364.252913] RIP: 0033:0x7ff54af45cd8 [ 8364.255071] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8364.265826] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8364.269717] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 8364.273688] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 8364.277041] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 8364.280300] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 8364.283556] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8368.060226] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 8368.069344] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8368.095870] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8368.100689] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8368.104461] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8368.107743] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8368.110447] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8368.118403] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8368.121592] RAX: 0000000000005708 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 8368.125093] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 8368.132823] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 8368.137874] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 8368.141646] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 8368.148593] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 8368.158948] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8368.168793] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 8368.174712] Call Trace: [ 8368.177487] [ 8368.179509] ? show_regs.cold.9+0x22/0x2f [ 8368.183690] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8368.190663] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8368.200836] ? watchdog+0x40/0x40 [ 8368.207632] ? hrtimer_interrupt+0x128/0x2c0 [ 8368.211161] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8368.214517] ? apic_timer_interrupt+0xf/0x20 [ 8368.217732] [ 8368.220816] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8368.224560] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8368.231824] do_raw_spin_lock+0x177/0x190 [ 8368.234788] _raw_spin_lock+0x21/0x30 [ 8368.238804] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8368.244192] process_one_work+0x2c8/0x700 [ 8368.246341] worker_thread+0x296/0x6e0 [ 8368.252902] ? rescuer_thread+0x570/0x570 [ 8368.256953] kthread+0x1d1/0x200 [ 8368.258124] ? set_kthread_struct+0x70/0x70 [ 8368.260576] ret_from_fork+0x1f/0x30 [ 8372.055114] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [ 8372.058444] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8372.079154] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8372.083840] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8372.087969] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8372.090630] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8372.100106] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8372.103506] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 8372.106806] RDX: 00000000000007a8 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 8372.109846] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 8372.113133] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 8372.116888] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8372.119949] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 8372.123895] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8372.126765] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 8372.131013] Call Trace: [ 8372.132397] [ 8372.133379] ? show_regs.cold.9+0x22/0x2f [ 8372.135512] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8372.138359] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8372.141079] ? watchdog+0x40/0x40 [ 8372.142727] ? hrtimer_interrupt+0x128/0x2c0 [ 8372.145128] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8372.147730] ? apic_timer_interrupt+0xf/0x20 [ 8372.149957] [ 8372.150995] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8372.153774] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8372.156599] do_raw_spin_lock+0x177/0x190 [ 8372.159278] _raw_spin_lock+0x21/0x30 [ 8372.160690] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8372.163790] obd_mod_free+0x46/0x140 [ptlrpc] [ 8372.167342] mdc_commit_open+0x91/0xb0 [mdc] [ 8372.169353] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8372.172573] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8372.176127] after_reply+0xe2c/0x1780 [ptlrpc] [ 8372.178988] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8372.182322] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8372.184987] ? do_wait_intr+0xf0/0xf0 [ 8372.186269] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8372.188592] kthread+0x1d1/0x200 [ 8372.190388] ? set_kthread_struct+0x70/0x70 [ 8372.192779] ret_from_fork+0x1f/0x30 [ 8376.799046] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6417s! [ 8376.802606] Showing busy workqueues and worker pools: [ 8376.804850] workqueue events: flags=0x0 [ 8376.806665] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8376.809995] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 8376.814133] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8376.816870] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 8376.822526] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8376.825873] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 8376.829383] workqueue events_long: flags=0x0 [ 8376.831739] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8376.834925] pending: lu_objects_flush [obdclass] [ 8376.840687] workqueue events_unbound: flags=0x2 [ 8376.844520] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8376.855344] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8376.862976] workqueue events_power_efficient: flags=0x80 [ 8376.871260] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8376.880033] pending: neigh_periodic_work [ 8376.882931] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8376.890167] pending: do_cache_clean, check_lifetime [ 8376.897081] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8376.902813] pending: neigh_periodic_work [ 8376.905211] workqueue writeback: flags=0x4a [ 8376.908542] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8376.912128] pending: wb_workfn, wb_workfn [ 8376.915945] workqueue xprtiod: flags=0xa [ 8376.917821] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8376.929624] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 8376.937627] workqueue ipv6_addrconf: flags=0x40008 [ 8376.945052] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8376.954048] pending: addrconf_verify_work [ 8376.955462] workqueue knbd0-recv: flags=0x1a [ 8376.959149] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8376.972955] in-flight: 657:recv_work [ 8376.984588] workqueue ptlrpc_pinger: flags=0x2 [ 8376.991922] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8377.005059] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 8377.018617] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6418s workers=4 idle: 20831 12642 28859 [ 8377.025062] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8367s workers=2 idle: 659 [ 8384.060044] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [ 8384.062933] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8384.098106] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8384.102711] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8384.104859] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8384.106184] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8384.113346] RSP: 0000:ffffaf9fc15234f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8384.116221] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 8384.119180] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 8384.122149] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8384.125132] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 8384.128205] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 8384.131288] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 8384.134380] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8384.135922] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 8384.138369] Call Trace: [ 8384.139277] [ 8384.139828] ? show_regs.cold.9+0x22/0x2f [ 8384.140834] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8384.142416] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8384.144348] ? watchdog+0x40/0x40 [ 8384.145436] ? hrtimer_interrupt+0x128/0x2c0 [ 8384.146905] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8384.148651] ? apic_timer_interrupt+0xf/0x20 [ 8384.150148] [ 8384.150779] ? native_safe_halt+0x2e/0x50 [ 8384.152082] kvm_wait+0xcd/0xe0 [ 8384.153353] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8384.155467] do_raw_spin_lock+0x177/0x190 [ 8384.157062] _raw_spin_lock+0x21/0x30 [ 8384.158361] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8384.161027] ? do_raw_spin_unlock+0x75/0x190 [ 8384.162777] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8384.164896] ? do_wait_intr+0xf0/0xf0 [ 8384.166394] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8384.168507] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8384.170816] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8384.173410] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8384.175731] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8384.177994] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8384.179919] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8384.181688] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8384.183863] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8384.186467] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8384.188773] lmv_intent_open+0x338/0x1150 [lmv] [ 8384.190706] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8384.193122] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8384.195210] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8384.197040] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8384.199042] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8384.201153] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8384.203066] ? do_raw_spin_unlock+0x75/0x190 [ 8384.204770] ? _raw_spin_unlock+0x12/0x30 [ 8384.206342] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8384.208477] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8384.210755] ll_atomic_open+0x36a/0xd70 [lustre] [ 8384.212677] lookup_open+0x356/0xb40 [ 8384.214044] do_last+0x313/0xfc0 [ 8384.215356] ? nd_jump_root+0xe5/0x160 [ 8384.216987] ? path_init+0x437/0x520 [ 8384.218409] path_openat+0xf7/0x500 [ 8384.219852] ? do_raw_spin_unlock+0x75/0x190 [ 8384.221483] do_filp_open+0x99/0x140 [ 8384.222993] ? getname_flags+0x6e/0x330 [ 8384.224496] ? __check_object_size+0xff/0x256 [ 8384.226090] ? do_raw_spin_unlock+0x75/0x190 [ 8384.227819] ? _raw_spin_unlock+0x12/0x30 [ 8384.229354] do_sys_openat2+0x2b4/0x410 [ 8384.230860] do_sys_open+0x73/0xa0 [ 8384.232356] __x64_sys_openat+0x24/0x30 [ 8384.234186] do_syscall_64+0xc1/0x440 [ 8384.235633] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8384.237492] RIP: 0033:0x7ff54af45332 [ 8384.239030] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8384.247724] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8384.252423] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 8384.259195] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 8384.264355] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 8384.267241] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 8384.271680] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 8392.058042] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 8392.063345] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8392.145602] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8392.164827] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8392.176247] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8392.180819] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8392.199298] RSP: 0018:ffffaf9fc1543ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8392.209503] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 8392.218748] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 8392.220910] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8392.225832] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 8392.230703] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 8392.234714] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 8392.240698] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8392.244512] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 8392.257855] Call Trace: [ 8392.263711] [ 8392.268797] ? show_regs.cold.9+0x22/0x2f [ 8392.273100] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8392.276669] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8392.279493] ? watchdog+0x40/0x40 [ 8392.280972] ? hrtimer_interrupt+0x128/0x2c0 [ 8392.283692] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8392.297714] ? apic_timer_interrupt+0xf/0x20 [ 8392.301763] [ 8392.303737] ? native_safe_halt+0x2e/0x50 [ 8392.307034] kvm_wait+0xcd/0xe0 [ 8392.308832] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8392.313566] do_raw_spin_lock+0x177/0x190 [ 8392.315792] _raw_spin_lock+0x21/0x30 [ 8392.319633] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8392.323645] ? do_raw_spin_unlock+0x75/0x190 [ 8392.326804] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8392.330522] ? do_wait_intr+0xf0/0xf0 [ 8392.331839] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8392.339662] mdc_close+0x53b/0xd70 [mdc] [ 8392.349988] lmv_close+0x18d/0x490 [lmv] [ 8392.353473] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 8392.363075] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8392.376420] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8392.377884] ll_md_close+0x274/0x800 [lustre] [ 8392.387702] ll_file_release+0x128/0xc50 [lustre] [ 8392.398248] __fput+0x10d/0x440 [ 8392.400742] ____fput+0x12/0x20 [ 8392.405822] task_work_run+0xc6/0x110 [ 8392.408295] exit_to_usermode_loop+0x1dd/0x1f0 [ 8392.413408] do_syscall_64+0x3d6/0x440 [ 8392.417076] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8392.420509] RIP: 0033:0x7ff54af45cd8 [ 8392.422770] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8392.435435] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8392.444260] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 8392.454177] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 8392.465253] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 8392.470369] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 8392.477301] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8396.059032] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 8396.064244] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8396.134777] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8396.159403] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8396.167945] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8396.184586] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8396.193685] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8396.213801] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8396.217569] RAX: 0000000000005f2a RBX: ffff96848811cae0 RCX: 0000000000000001 [ 8396.222671] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 8396.225890] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 8396.229445] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 8396.233032] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 8396.236634] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 8396.241710] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8396.244677] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 8396.318148] Call Trace: [ 8396.319403] [ 8396.320491] ? show_regs.cold.9+0x22/0x2f [ 8396.322416] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8396.324900] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8396.327130] ? watchdog+0x40/0x40 [ 8396.328744] ? hrtimer_interrupt+0x128/0x2c0 [ 8396.330887] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8396.332569] ? apic_timer_interrupt+0xf/0x20 [ 8396.334463] [ 8396.335449] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8396.337932] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8396.340346] do_raw_spin_lock+0x177/0x190 [ 8396.342235] _raw_spin_lock+0x21/0x30 [ 8396.343975] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8396.346616] process_one_work+0x2c8/0x700 [ 8396.348677] worker_thread+0x296/0x6e0 [ 8396.350558] ? rescuer_thread+0x570/0x570 [ 8396.352141] kthread+0x1d1/0x200 [ 8396.353637] ? set_kthread_struct+0x70/0x70 [ 8396.355765] ret_from_fork+0x1f/0x30 [ 8404.055179] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 8404.064080] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8404.100916] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8404.107649] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8404.112099] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8404.114192] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8404.122728] RSP: 0000:ffffaf9fc1b97bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8404.131159] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 8404.139627] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 8404.148537] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 8404.154749] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 8404.166146] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8404.171685] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 8404.181128] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8404.186810] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 8404.193091] Call Trace: [ 8404.195226] [ 8404.197495] ? show_regs.cold.9+0x22/0x2f [ 8404.200986] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8404.204675] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8404.208965] ? watchdog+0x40/0x40 [ 8404.212515] ? hrtimer_interrupt+0x128/0x2c0 [ 8404.219213] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8404.221958] ? apic_timer_interrupt+0xf/0x20 [ 8404.226027] [ 8404.228994] ? native_safe_halt+0x2e/0x50 [ 8404.231782] kvm_wait+0xcd/0xe0 [ 8404.233238] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8404.239279] do_raw_spin_lock+0x177/0x190 [ 8404.246310] _raw_spin_lock+0x21/0x30 [ 8404.250533] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8404.254877] obd_mod_free+0x46/0x140 [ptlrpc] [ 8404.267098] mdc_commit_open+0x91/0xb0 [mdc] [ 8404.270842] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8404.276673] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8404.282522] after_reply+0xe2c/0x1780 [ptlrpc] [ 8404.288654] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8404.296917] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8404.300465] ? do_wait_intr+0xf0/0xf0 [ 8404.304817] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8404.307951] kthread+0x1d1/0x200 [ 8404.309843] ? set_kthread_struct+0x70/0x70 [ 8404.312657] ret_from_fork+0x1f/0x30 [ 8407.519081] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6448s! [ 8407.523514] Showing busy workqueues and worker pools: [ 8407.526149] workqueue events: flags=0x0 [ 8407.529082] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8407.537585] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 8407.549053] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8407.553635] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 8407.564348] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8407.571612] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 8407.577820] workqueue events_long: flags=0x0 [ 8407.580854] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8407.587674] pending: lu_objects_flush [obdclass] [ 8407.591971] workqueue events_unbound: flags=0x2 [ 8407.593646] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8407.601409] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8407.612647] workqueue events_power_efficient: flags=0x80 [ 8407.617281] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8407.624436] pending: neigh_periodic_work [ 8407.627639] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8407.630755] pending: do_cache_clean, check_lifetime [ 8407.633503] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8407.640195] pending: neigh_periodic_work [ 8407.642445] workqueue writeback: flags=0x4a [ 8407.644622] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8407.648268] pending: wb_workfn, wb_workfn [ 8407.652418] workqueue xprtiod: flags=0xa [ 8407.655178] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8407.660936] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 8407.665498] workqueue ipv6_addrconf: flags=0x40008 [ 8407.669596] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8407.675598] pending: addrconf_verify_work [ 8407.679555] workqueue knbd0-recv: flags=0x1a [ 8407.682016] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8407.685426] in-flight: 657:recv_work [ 8407.687217] workqueue ptlrpc_pinger: flags=0x2 [ 8407.690255] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8407.693230] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 8407.696046] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6448s workers=4 idle: 20831 12642 28859 [ 8407.700402] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8397s workers=2 idle: 659 [ 8412.060030] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [ 8412.063227] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8412.101815] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8412.112517] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8412.121499] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8412.125284] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8412.139009] RSP: 0000:ffffaf9fc15234f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8412.147877] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 8412.152267] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 8412.162606] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8412.167280] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 8412.171577] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 8412.179602] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 8412.185436] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8412.188392] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 8412.197408] Call Trace: [ 8412.201339] [ 8412.203782] ? show_regs.cold.9+0x22/0x2f [ 8412.207504] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8412.209515] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8412.211711] ? watchdog+0x40/0x40 [ 8412.213132] ? hrtimer_interrupt+0x128/0x2c0 [ 8412.214893] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8412.216461] ? apic_timer_interrupt+0xf/0x20 [ 8412.218341] [ 8412.219402] ? native_safe_halt+0x2e/0x50 [ 8412.223358] kvm_wait+0xcd/0xe0 [ 8412.224627] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8412.228230] do_raw_spin_lock+0x177/0x190 [ 8412.230501] _raw_spin_lock+0x21/0x30 [ 8412.232833] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8412.236482] ? do_raw_spin_unlock+0x75/0x190 [ 8412.239913] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8412.242956] ? do_wait_intr+0xf0/0xf0 [ 8412.248650] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8412.258637] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8412.262925] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8412.268119] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8412.269625] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8412.274384] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8412.279424] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8412.283742] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8412.287342] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8412.293080] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8412.297403] lmv_intent_open+0x338/0x1150 [lmv] [ 8412.304361] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8412.307836] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8412.314758] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8412.318172] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8412.319843] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8412.322094] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8412.323553] ? do_raw_spin_unlock+0x75/0x190 [ 8412.325505] ? _raw_spin_unlock+0x12/0x30 [ 8412.327349] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8412.329174] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8412.331081] ll_atomic_open+0x36a/0xd70 [lustre] [ 8412.333087] lookup_open+0x356/0xb40 [ 8412.334406] do_last+0x313/0xfc0 [ 8412.335987] ? nd_jump_root+0xe5/0x160 [ 8412.337402] ? path_init+0x437/0x520 [ 8412.338918] path_openat+0xf7/0x500 [ 8412.340501] ? do_raw_spin_unlock+0x75/0x190 [ 8412.342771] do_filp_open+0x99/0x140 [ 8412.344402] ? getname_flags+0x6e/0x330 [ 8412.346699] ? __check_object_size+0xff/0x256 [ 8412.347839] ? do_raw_spin_unlock+0x75/0x190 [ 8412.350270] ? _raw_spin_unlock+0x12/0x30 [ 8412.352202] do_sys_openat2+0x2b4/0x410 [ 8412.354456] do_sys_open+0x73/0xa0 [ 8412.359119] __x64_sys_openat+0x24/0x30 [ 8412.364022] do_syscall_64+0xc1/0x440 [ 8412.368008] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8412.371542] RIP: 0033:0x7ff54af45332 [ 8412.374424] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8412.390189] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8412.396591] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 8412.404499] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 8412.411895] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 8412.417487] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 8412.419472] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 8420.060726] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 8420.069268] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8420.110193] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8420.124862] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8420.133419] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8420.140980] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8420.159008] RSP: 0018:ffffaf9fc1543ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8420.165828] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 8420.169809] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 8420.177083] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8420.182645] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 8420.187658] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 8420.192883] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 8420.200240] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8420.203811] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 8420.208141] Call Trace: [ 8420.210006] [ 8420.212135] ? show_regs.cold.9+0x22/0x2f [ 8420.216171] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8420.220084] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8420.222873] ? watchdog+0x40/0x40 [ 8420.225896] ? hrtimer_interrupt+0x128/0x2c0 [ 8420.229810] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8420.232774] ? apic_timer_interrupt+0xf/0x20 [ 8420.235909] [ 8420.239782] ? native_safe_halt+0x2e/0x50 [ 8420.241067] kvm_wait+0xcd/0xe0 [ 8420.241874] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8420.243693] do_raw_spin_lock+0x177/0x190 [ 8420.244818] _raw_spin_lock+0x21/0x30 [ 8420.245850] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8420.250213] ? do_raw_spin_unlock+0x75/0x190 [ 8420.253251] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8420.256939] ? do_wait_intr+0xf0/0xf0 [ 8420.259168] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8420.263439] mdc_close+0x53b/0xd70 [mdc] [ 8420.265781] lmv_close+0x18d/0x490 [lmv] [ 8420.267941] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 8420.270724] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8420.276746] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8420.279224] ll_md_close+0x274/0x800 [lustre] [ 8420.286973] ll_file_release+0x128/0xc50 [lustre] [ 8420.296043] __fput+0x10d/0x440 [ 8420.304708] ____fput+0x12/0x20 [ 8420.308633] task_work_run+0xc6/0x110 [ 8420.314043] exit_to_usermode_loop+0x1dd/0x1f0 [ 8420.318853] do_syscall_64+0x3d6/0x440 [ 8420.323009] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8420.330367] RIP: 0033:0x7ff54af45cd8 [ 8420.333938] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8420.347557] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8420.350432] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 8420.361397] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 8420.368784] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 8420.374525] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 8420.386823] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8424.064686] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 8424.072302] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8424.097444] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8424.103910] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8424.110327] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8424.114335] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8424.116104] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8424.122803] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8424.126352] RAX: 0000000000007d85 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 8424.128287] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 8424.130309] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 8424.132674] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 8424.135418] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 8424.137836] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 8424.142396] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8424.145242] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 8424.147668] Call Trace: [ 8424.148487] [ 8424.149233] ? show_regs.cold.9+0x22/0x2f [ 8424.150365] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8424.152262] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8424.154281] ? watchdog+0x40/0x40 [ 8424.155283] ? hrtimer_interrupt+0x128/0x2c0 [ 8424.156371] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8424.157885] ? apic_timer_interrupt+0xf/0x20 [ 8424.160320] [ 8424.160879] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8424.162776] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8424.164389] do_raw_spin_lock+0x177/0x190 [ 8424.165464] _raw_spin_lock+0x21/0x30 [ 8424.167626] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8424.169747] process_one_work+0x2c8/0x700 [ 8424.171343] worker_thread+0x296/0x6e0 [ 8424.172245] ? rescuer_thread+0x570/0x570 [ 8424.173016] kthread+0x1d1/0x200 [ 8424.173707] ? set_kthread_struct+0x70/0x70 [ 8424.175826] ret_from_fork+0x1f/0x30 [ 8432.055031] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 8432.058189] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8432.076768] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8432.081573] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8432.084944] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8432.087039] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8432.095387] RSP: 0000:ffffaf9fc1b97bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8432.098886] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 8432.101990] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 8432.105270] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 8432.109418] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 8432.124911] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8432.134043] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 8432.144026] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8432.152828] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 8432.157220] Call Trace: [ 8432.160707] [ 8432.161440] ? show_regs.cold.9+0x22/0x2f [ 8432.162884] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8432.172250] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8432.184745] ? watchdog+0x40/0x40 [ 8432.189850] ? hrtimer_interrupt+0x128/0x2c0 [ 8432.200741] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8432.202158] ? apic_timer_interrupt+0xf/0x20 [ 8432.203458] [ 8432.212152] ? native_safe_halt+0x2e/0x50 [ 8432.217251] kvm_wait+0xcd/0xe0 [ 8432.228346] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8432.238342] do_raw_spin_lock+0x177/0x190 [ 8432.241674] _raw_spin_lock+0x21/0x30 [ 8432.244222] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8432.252858] obd_mod_free+0x46/0x140 [ptlrpc] [ 8432.260392] mdc_commit_open+0x91/0xb0 [mdc] [ 8432.264249] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8432.268736] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8432.271995] after_reply+0xe2c/0x1780 [ptlrpc] [ 8432.275873] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8432.278351] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8432.281402] ? do_wait_intr+0xf0/0xf0 [ 8432.284373] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8432.287429] kthread+0x1d1/0x200 [ 8432.288907] ? set_kthread_struct+0x70/0x70 [ 8432.291979] ret_from_fork+0x1f/0x30 [ 8437.727336] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6478s! [ 8437.734400] Showing busy workqueues and worker pools: [ 8437.736859] workqueue events: flags=0x0 [ 8437.738904] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8437.741731] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 8437.745871] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8437.748380] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 8437.760469] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8437.764450] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 8437.770288] workqueue events_long: flags=0x0 [ 8437.772352] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8437.776113] pending: lu_objects_flush [obdclass] [ 8437.778481] workqueue events_unbound: flags=0x2 [ 8437.780920] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8437.783829] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8437.788320] workqueue events_power_efficient: flags=0x80 [ 8437.790445] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8437.793728] pending: neigh_periodic_work [ 8437.795540] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8437.798550] pending: do_cache_clean, check_lifetime [ 8437.800746] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8437.804008] pending: neigh_periodic_work [ 8437.806027] workqueue writeback: flags=0x4a [ 8437.808173] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8437.811893] pending: wb_workfn, wb_workfn [ 8437.814178] workqueue xprtiod: flags=0xa [ 8437.816176] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8437.820225] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 8437.824367] workqueue ipv6_addrconf: flags=0x40008 [ 8437.826865] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8437.830396] pending: addrconf_verify_work [ 8437.832373] workqueue knbd0-recv: flags=0x1a [ 8437.834524] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8437.837650] in-flight: 657:recv_work [ 8437.839655] workqueue ptlrpc_pinger: flags=0x2 [ 8437.841484] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8437.844377] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 8437.847725] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6478s workers=4 idle: 20831 12642 28859 [ 8437.854800] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8428s workers=2 idle: 659 [ 8440.060070] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 8440.075314] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8440.126921] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8440.136304] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8440.143789] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8440.147273] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8440.163627] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8440.168986] RAX: 0000000000006430 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 8440.174553] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 8440.180034] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8440.183213] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 8440.188138] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 8440.192940] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 8440.203085] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8440.208988] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 8440.213683] Call Trace: [ 8440.215488] [ 8440.218116] ? show_regs.cold.9+0x22/0x2f [ 8440.220514] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8440.222204] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8440.226823] ? watchdog+0x40/0x40 [ 8440.229103] ? hrtimer_interrupt+0x128/0x2c0 [ 8440.233845] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8440.241515] ? apic_timer_interrupt+0xf/0x20 [ 8440.247010] [ 8440.248220] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8440.253357] do_raw_spin_lock+0x177/0x190 [ 8440.256706] _raw_spin_lock+0x21/0x30 [ 8440.259275] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8440.262694] ? do_raw_spin_unlock+0x75/0x190 [ 8440.265089] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8440.268030] ? do_wait_intr+0xf0/0xf0 [ 8440.270010] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8440.273828] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8440.275479] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8440.279500] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8440.282208] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8440.285278] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8440.289310] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8440.294835] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8440.302169] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8440.310131] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8440.314995] lmv_intent_open+0x338/0x1150 [lmv] [ 8440.318111] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8440.319822] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8440.331422] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8440.335172] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8440.338109] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8440.341861] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8440.346125] ? do_raw_spin_unlock+0x75/0x190 [ 8440.348642] ? _raw_spin_unlock+0x12/0x30 [ 8440.352217] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8440.353351] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8440.355671] ll_atomic_open+0x36a/0xd70 [lustre] [ 8440.358070] lookup_open+0x356/0xb40 [ 8440.361703] do_last+0x313/0xfc0 [ 8440.363172] ? nd_jump_root+0xe5/0x160 [ 8440.365205] ? path_init+0x437/0x520 [ 8440.366436] path_openat+0xf7/0x500 [ 8440.368249] ? do_raw_spin_unlock+0x75/0x190 [ 8440.372167] do_filp_open+0x99/0x140 [ 8440.377652] ? getname_flags+0x6e/0x330 [ 8440.383326] ? __check_object_size+0xff/0x256 [ 8440.387651] ? do_raw_spin_unlock+0x75/0x190 [ 8440.393041] ? _raw_spin_unlock+0x12/0x30 [ 8440.395759] do_sys_openat2+0x2b4/0x410 [ 8440.402369] do_sys_open+0x73/0xa0 [ 8440.407444] __x64_sys_openat+0x24/0x30 [ 8440.412228] do_syscall_64+0xc1/0x440 [ 8440.414567] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8440.417471] RIP: 0033:0x7ff54af45332 [ 8440.424145] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8440.458725] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8440.466720] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 8440.472185] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 8440.481816] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 8440.494758] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 8440.508212] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 8448.058026] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 8448.061074] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8448.103966] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8448.116048] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8448.121886] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8448.124269] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8448.146891] RSP: 0018:ffffaf9fc1543ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8448.152339] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 8448.162050] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 8448.168936] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8448.177727] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 8448.183913] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 8448.188955] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 8448.195242] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8448.197777] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 8448.200894] Call Trace: [ 8448.204711] [ 8448.207678] ? show_regs.cold.9+0x22/0x2f [ 8448.209829] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8448.212440] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8448.217984] ? watchdog+0x40/0x40 [ 8448.220630] ? hrtimer_interrupt+0x128/0x2c0 [ 8448.224720] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8448.228055] ? apic_timer_interrupt+0xf/0x20 [ 8448.230233] [ 8448.231220] ? native_safe_halt+0x2e/0x50 [ 8448.233217] kvm_wait+0xcd/0xe0 [ 8448.234720] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8448.238956] do_raw_spin_lock+0x177/0x190 [ 8448.240892] _raw_spin_lock+0x21/0x30 [ 8448.243976] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8448.247961] ? do_raw_spin_unlock+0x75/0x190 [ 8448.250132] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8448.253112] ? do_wait_intr+0xf0/0xf0 [ 8448.254462] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8448.257065] mdc_close+0x53b/0xd70 [mdc] [ 8448.258534] lmv_close+0x18d/0x490 [lmv] [ 8448.260177] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 8448.264815] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8448.269182] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8448.274322] ll_md_close+0x274/0x800 [lustre] [ 8448.276078] ll_file_release+0x128/0xc50 [lustre] [ 8448.277383] __fput+0x10d/0x440 [ 8448.279589] ____fput+0x12/0x20 [ 8448.280891] task_work_run+0xc6/0x110 [ 8448.283234] exit_to_usermode_loop+0x1dd/0x1f0 [ 8448.286552] do_syscall_64+0x3d6/0x440 [ 8448.291012] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8448.295822] RIP: 0033:0x7ff54af45cd8 [ 8448.298891] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8448.311216] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8448.318709] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 8448.327924] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 8448.336978] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 8448.342583] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 8448.346147] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8452.059161] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 8452.064363] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8452.108959] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8452.118203] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8452.124415] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8452.131003] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8452.137967] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8452.150512] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8452.155259] RAX: 00000000000022cd RBX: ffff96848811cae0 RCX: 0000000000000001 [ 8452.159827] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 8452.164141] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 8452.168363] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 8452.172703] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 8452.176806] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 8452.182955] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8452.187804] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 8452.192904] Call Trace: [ 8452.194757] [ 8452.195690] ? show_regs.cold.9+0x22/0x2f [ 8452.198354] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8452.202713] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8452.205240] ? watchdog+0x40/0x40 [ 8452.207169] ? hrtimer_interrupt+0x128/0x2c0 [ 8452.210770] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8452.213048] ? apic_timer_interrupt+0xf/0x20 [ 8452.216409] [ 8452.217820] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8452.220983] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8452.224503] do_raw_spin_lock+0x177/0x190 [ 8452.226868] _raw_spin_lock+0x21/0x30 [ 8452.228989] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8452.231904] process_one_work+0x2c8/0x700 [ 8452.234338] worker_thread+0x296/0x6e0 [ 8452.236727] ? rescuer_thread+0x570/0x570 [ 8452.239347] kthread+0x1d1/0x200 [ 8452.240778] ? set_kthread_struct+0x70/0x70 [ 8452.242535] ret_from_fork+0x1f/0x30 [ 8464.055240] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 8464.065606] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8464.109746] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8464.117830] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8464.127434] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8464.135308] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8464.142378] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8464.145133] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 8464.147896] RDX: 00000000000058c7 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 8464.151674] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 8464.155885] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 8464.158581] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8464.161273] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 8464.164566] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8464.166974] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 8464.170334] Call Trace: [ 8464.171901] [ 8464.172990] ? show_regs.cold.9+0x22/0x2f [ 8464.174782] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8464.177385] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8464.181360] ? watchdog+0x40/0x40 [ 8464.184009] ? hrtimer_interrupt+0x128/0x2c0 [ 8464.188028] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8464.191347] ? apic_timer_interrupt+0xf/0x20 [ 8464.193232] [ 8464.197140] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8464.205023] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8464.211973] do_raw_spin_lock+0x177/0x190 [ 8464.213789] _raw_spin_lock+0x21/0x30 [ 8464.216375] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8464.221691] obd_mod_free+0x46/0x140 [ptlrpc] [ 8464.228955] mdc_commit_open+0x91/0xb0 [mdc] [ 8464.230728] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8464.243608] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8464.251800] after_reply+0xe2c/0x1780 [ptlrpc] [ 8464.257016] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8464.263234] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8464.266521] ? do_wait_intr+0xf0/0xf0 [ 8464.268694] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8464.271464] kthread+0x1d1/0x200 [ 8464.273239] ? set_kthread_struct+0x70/0x70 [ 8464.276571] ret_from_fork+0x1f/0x30 [ 8467.935118] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6509s! [ 8467.938598] Showing busy workqueues and worker pools: [ 8467.942444] workqueue events: flags=0x0 [ 8467.944100] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8467.947327] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 8467.951969] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8467.954730] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 8467.959469] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8467.962172] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 8467.965355] workqueue events_long: flags=0x0 [ 8467.967710] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8467.970965] pending: lu_objects_flush [obdclass] [ 8467.973587] workqueue events_unbound: flags=0x2 [ 8467.978259] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8467.982276] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8467.991610] workqueue events_power_efficient: flags=0x80 [ 8467.997301] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8468.004792] pending: neigh_periodic_work [ 8468.006910] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8468.011903] pending: do_cache_clean, check_lifetime [ 8468.016662] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8468.020675] pending: neigh_periodic_work [ 8468.022859] workqueue writeback: flags=0x4a [ 8468.026740] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8468.031229] pending: wb_workfn, wb_workfn [ 8468.032995] workqueue xprtiod: flags=0xa [ 8468.037803] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8468.041254] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 8468.044586] workqueue ipv6_addrconf: flags=0x40008 [ 8468.046454] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8468.048931] pending: addrconf_verify_work [ 8468.052618] workqueue knbd0-recv: flags=0x1a [ 8468.055951] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8468.059693] in-flight: 657:recv_work [ 8468.060024] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 8468.061628] workqueue ptlrpc_pinger: flags=0x2 [ 8468.069557] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8468.069679] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8468.069685] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8468.069688] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8468.069700] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8468.069704] RSP: 0000:ffffaf9fc15234f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8468.069709] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 8468.069712] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 8468.069714] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8468.069717] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 8468.069719] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 8468.071162] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8468.071177] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 8468.071501] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6509s workers=4 idle: 20831 12642 28859 [ 8468.071515] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8458s workers=2 idle: 659 [ 8468.214876] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 8468.222096] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8468.227755] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 8468.231042] Call Trace: [ 8468.234082] [ 8468.237354] ? show_regs.cold.9+0x22/0x2f [ 8468.239992] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8468.244621] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8468.246572] ? watchdog+0x40/0x40 [ 8468.249815] ? hrtimer_interrupt+0x128/0x2c0 [ 8468.255616] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8468.258513] ? apic_timer_interrupt+0xf/0x20 [ 8468.261962] [ 8468.262711] ? native_safe_halt+0x2e/0x50 [ 8468.268366] kvm_wait+0xcd/0xe0 [ 8468.270631] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8468.275534] do_raw_spin_lock+0x177/0x190 [ 8468.279561] _raw_spin_lock+0x21/0x30 [ 8468.281812] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8468.286034] ? do_raw_spin_unlock+0x75/0x190 [ 8468.291084] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8468.297503] ? do_wait_intr+0xf0/0xf0 [ 8468.299379] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8468.304129] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8468.309594] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8468.312910] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8468.315095] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8468.318098] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8468.320842] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8468.323444] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8468.325484] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8468.327327] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8468.329765] lmv_intent_open+0x338/0x1150 [lmv] [ 8468.332088] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8468.334529] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8468.337048] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8468.339048] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8468.341455] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8468.344696] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8468.349991] ? do_raw_spin_unlock+0x75/0x190 [ 8468.352804] ? _raw_spin_unlock+0x12/0x30 [ 8468.357395] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8468.361895] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8468.367784] ll_atomic_open+0x36a/0xd70 [lustre] [ 8468.373361] lookup_open+0x356/0xb40 [ 8468.377355] do_last+0x313/0xfc0 [ 8468.380789] ? nd_jump_root+0xe5/0x160 [ 8468.387059] ? path_init+0x437/0x520 [ 8468.391434] path_openat+0xf7/0x500 [ 8468.395884] ? do_raw_spin_unlock+0x75/0x190 [ 8468.400737] do_filp_open+0x99/0x140 [ 8468.404626] ? getname_flags+0x6e/0x330 [ 8468.408111] ? __check_object_size+0xff/0x256 [ 8468.409519] ? do_raw_spin_unlock+0x75/0x190 [ 8468.418709] ? _raw_spin_unlock+0x12/0x30 [ 8468.421524] do_sys_openat2+0x2b4/0x410 [ 8468.425735] do_sys_open+0x73/0xa0 [ 8468.428688] __x64_sys_openat+0x24/0x30 [ 8468.432872] do_syscall_64+0xc1/0x440 [ 8468.436623] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8468.442600] RIP: 0033:0x7ff54af45332 [ 8468.447797] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8468.464849] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8468.469705] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 8468.474535] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 8468.484316] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 8468.488949] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 8468.492245] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 8476.059181] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 8476.074965] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8476.148118] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8476.155561] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8476.160465] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8476.172651] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8476.196424] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8476.207460] RAX: 0000000000007d85 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 8476.215673] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 8476.224322] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8476.230930] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 8476.236310] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 8476.244188] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 8476.259339] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8476.267678] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 8476.278630] Call Trace: [ 8476.280909] [ 8476.284316] ? show_regs.cold.9+0x22/0x2f [ 8476.291476] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8476.301455] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8476.309675] ? watchdog+0x40/0x40 [ 8476.312832] ? hrtimer_interrupt+0x128/0x2c0 [ 8476.316126] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8476.319958] ? apic_timer_interrupt+0xf/0x20 [ 8476.323445] [ 8476.326980] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8476.331351] do_raw_spin_lock+0x177/0x190 [ 8476.333770] _raw_spin_lock+0x21/0x30 [ 8476.336956] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8476.338990] ? do_raw_spin_unlock+0x75/0x190 [ 8476.347329] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8476.353273] ? do_wait_intr+0xf0/0xf0 [ 8476.356312] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8476.361493] mdc_close+0x53b/0xd70 [mdc] [ 8476.363873] lmv_close+0x18d/0x490 [lmv] [ 8476.365880] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 8476.370176] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8476.385926] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8476.390713] ll_md_close+0x274/0x800 [lustre] [ 8476.395267] ll_file_release+0x128/0xc50 [lustre] [ 8476.398606] __fput+0x10d/0x440 [ 8476.401014] ____fput+0x12/0x20 [ 8476.403454] task_work_run+0xc6/0x110 [ 8476.407246] exit_to_usermode_loop+0x1dd/0x1f0 [ 8476.413273] do_syscall_64+0x3d6/0x440 [ 8476.419556] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8476.423608] RIP: 0033:0x7ff54af45cd8 [ 8476.426507] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8476.449890] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8476.458652] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 8476.467589] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 8476.477674] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 8476.504988] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 8476.518651] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8480.059011] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 8480.069866] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8480.110443] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8480.122298] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8480.127220] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8480.134902] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8480.138594] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8480.151741] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8480.155415] RAX: 0000000000001ad6 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 8480.159458] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 8480.164368] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 8480.169780] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 8480.177202] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 8480.182292] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 8480.188444] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8480.191234] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 8480.195546] Call Trace: [ 8480.197116] [ 8480.197966] ? show_regs.cold.9+0x22/0x2f [ 8480.203552] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8480.208665] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8480.211509] ? watchdog+0x40/0x40 [ 8480.214257] ? hrtimer_interrupt+0x128/0x2c0 [ 8480.216391] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8480.230459] ? apic_timer_interrupt+0xf/0x20 [ 8480.234724] [ 8480.235458] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8480.237159] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8480.240450] do_raw_spin_lock+0x177/0x190 [ 8480.246042] _raw_spin_lock+0x21/0x30 [ 8480.251094] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8480.252929] process_one_work+0x2c8/0x700 [ 8480.255557] worker_thread+0x296/0x6e0 [ 8480.256880] ? rescuer_thread+0x570/0x570 [ 8480.264727] kthread+0x1d1/0x200 [ 8480.265945] ? set_kthread_struct+0x70/0x70 [ 8480.268418] ret_from_fork+0x1f/0x30 [ 8494.028030] rcu: INFO: rcu_sched self-detected stall on CPU [ 8494.030269] rcu: 0-...!: (3389242 ticks this GP) idle=98a/1/0x4000000000000002 softirq=125154/125154 fqs=959 [ 8494.033999] (t=6540188 jiffies g=253585 q=4943) [ 8494.035822] rcu: rcu_sched kthread starved for 6535148 jiffies! g253585 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [ 8494.038710] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 8494.041978] rcu: RCU grace-period kthread stack dump: [ 8494.043835] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 8494.047370] Call Trace: [ 8494.048282] __schedule+0x351/0xcb0 [ 8494.049643] schedule+0xc0/0x180 [ 8494.050945] schedule_timeout+0xb4/0x190 [ 8494.052399] ? do_raw_spin_unlock+0x75/0x190 [ 8494.053864] ? __next_timer_interrupt+0x160/0x160 [ 8494.054977] ? prepare_to_swait_event+0xf3/0x140 [ 8494.056011] rcu_gp_kthread+0x8dd/0x15b0 [ 8494.056926] ? force_qs_rnp+0x2c0/0x2c0 [ 8494.058189] kthread+0x1d1/0x200 [ 8494.059636] ? set_kthread_struct+0x70/0x70 [ 8494.067763] ret_from_fork+0x1f/0x30 [ 8494.069673] rcu: Stack dump where RCU GP kthread last ran: [ 8494.074430] NMI backtrace for cpu 0 [ 8494.079552] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8494.090823] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8494.100785] Call Trace: [ 8494.103258] [ 8494.104844] ? dump_stack+0xbb/0x10e [ 8494.107425] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 8494.109504] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 8494.119048] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 8494.125286] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 8494.133668] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 8494.141817] ? rcu_check_gp_kthread_starvation+0x16f/0x188 [ 8494.147610] ? rcu_sched_clock_irq.cold.51+0xce/0x307 [ 8494.151599] ? update_process_times+0x6d/0xc0 [ 8494.155401] ? tick_sched_handle.isra.18+0x27/0x90 [ 8494.161129] ? tick_sched_timer+0x53/0xb0 [ 8494.162956] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8494.166150] ? tick_sched_handle.isra.18+0x90/0x90 [ 8494.168919] ? hrtimer_interrupt+0x128/0x2c0 [ 8494.172841] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8494.176248] ? apic_timer_interrupt+0xf/0x20 [ 8494.179366] [ 8494.180418] ? native_safe_halt+0x2e/0x50 [ 8494.182576] ? kvm_wait+0xcd/0xe0 [ 8494.184821] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8494.190289] ? do_raw_spin_lock+0x177/0x190 [ 8494.195888] ? _raw_spin_lock+0x21/0x30 [ 8494.198668] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8494.202986] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 8494.208334] ? mdc_commit_open+0x91/0xb0 [mdc] [ 8494.213488] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8494.218726] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8494.226169] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 8494.236791] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8494.241304] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8494.248627] ? do_wait_intr+0xf0/0xf0 [ 8494.252403] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8494.258678] ? kthread+0x1d1/0x200 [ 8494.262370] ? set_kthread_struct+0x70/0x70 [ 8494.265798] ? ret_from_fork+0x1f/0x30 [ 8494.268118] NMI backtrace for cpu 0 [ 8494.272854] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8494.285636] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8494.291592] Call Trace: [ 8494.293335] [ 8494.294545] ? dump_stack+0xbb/0x10e [ 8494.296771] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 8494.303137] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 8494.305903] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 8494.312609] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 8494.318669] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 8494.322310] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 8494.326329] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 8494.329789] ? update_process_times+0x6d/0xc0 [ 8494.334205] ? tick_sched_handle.isra.18+0x27/0x90 [ 8494.338810] ? tick_sched_timer+0x53/0xb0 [ 8494.341195] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8494.344849] ? tick_sched_handle.isra.18+0x90/0x90 [ 8494.348332] ? hrtimer_interrupt+0x128/0x2c0 [ 8494.351929] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8494.354984] ? apic_timer_interrupt+0xf/0x20 [ 8494.358691] [ 8494.360507] ? native_safe_halt+0x2e/0x50 [ 8494.363498] ? kvm_wait+0xcd/0xe0 [ 8494.367105] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8494.375795] ? do_raw_spin_lock+0x177/0x190 [ 8494.377524] ? _raw_spin_lock+0x21/0x30 [ 8494.384590] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8494.397512] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 8494.406253] ? mdc_commit_open+0x91/0xb0 [mdc] [ 8494.409478] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8494.414771] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8494.434106] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 8494.442528] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8494.453437] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8494.460011] ? do_wait_intr+0xf0/0xf0 [ 8494.466193] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8494.472499] ? kthread+0x1d1/0x200 [ 8494.475137] ? set_kthread_struct+0x70/0x70 [ 8494.479750] ? ret_from_fork+0x1f/0x30 [ 8496.060710] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 8496.063093] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8496.102109] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8496.111553] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8496.130431] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8496.134282] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8496.142338] RSP: 0000:ffffaf9fc15234f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8496.158597] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 8496.165867] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 8496.170887] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8496.180327] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 8496.188390] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 8496.192774] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 8496.197770] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8496.206440] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 8496.210399] Call Trace: [ 8496.211715] [ 8496.212782] ? show_regs.cold.9+0x22/0x2f [ 8496.214610] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8496.216814] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8496.233606] ? watchdog+0x40/0x40 [ 8496.237406] ? hrtimer_interrupt+0x128/0x2c0 [ 8496.238928] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8496.243155] ? apic_timer_interrupt+0xf/0x20 [ 8496.244626] [ 8496.249632] ? native_safe_halt+0x2e/0x50 [ 8496.251715] kvm_wait+0xcd/0xe0 [ 8496.254799] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8496.265748] do_raw_spin_lock+0x177/0x190 [ 8496.270792] _raw_spin_lock+0x21/0x30 [ 8496.276926] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8496.283829] ? do_raw_spin_unlock+0x75/0x190 [ 8496.294783] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8496.305681] ? do_wait_intr+0xf0/0xf0 [ 8496.308074] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8496.310666] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8496.313276] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8496.315500] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8496.330624] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8496.342263] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8496.349324] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8496.359566] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8496.361882] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8496.365578] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8496.369152] lmv_intent_open+0x338/0x1150 [lmv] [ 8496.371076] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8496.373453] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8496.378302] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8496.380824] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8496.382342] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8496.392407] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8496.394085] ? do_raw_spin_unlock+0x75/0x190 [ 8496.399274] ? _raw_spin_unlock+0x12/0x30 [ 8496.400612] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8496.408446] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8496.410204] ll_atomic_open+0x36a/0xd70 [lustre] [ 8496.415223] lookup_open+0x356/0xb40 [ 8496.416756] do_last+0x313/0xfc0 [ 8496.420514] ? nd_jump_root+0xe5/0x160 [ 8496.425159] ? path_init+0x437/0x520 [ 8496.432274] path_openat+0xf7/0x500 [ 8496.444423] ? do_raw_spin_unlock+0x75/0x190 [ 8496.452225] do_filp_open+0x99/0x140 [ 8496.453372] ? getname_flags+0x6e/0x330 [ 8496.454592] ? __check_object_size+0xff/0x256 [ 8496.467500] ? do_raw_spin_unlock+0x75/0x190 [ 8496.468655] ? _raw_spin_unlock+0x12/0x30 [ 8496.469902] do_sys_openat2+0x2b4/0x410 [ 8496.472164] do_sys_open+0x73/0xa0 [ 8496.473348] __x64_sys_openat+0x24/0x30 [ 8496.477313] do_syscall_64+0xc1/0x440 [ 8496.482345] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8496.483900] RIP: 0033:0x7ff54af45332 [ 8496.486636] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8496.499236] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8496.502907] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 8496.514740] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 8496.529694] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 8496.546742] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 8496.548986] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 8498.143048] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6539s! [ 8498.147257] Showing busy workqueues and worker pools: [ 8498.149687] workqueue events: flags=0x0 [ 8498.151637] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8498.154587] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 8498.158286] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8498.166890] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 8498.193392] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8498.209331] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 8498.212597] workqueue events_long: flags=0x0 [ 8498.216393] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8498.226348] pending: lu_objects_flush [obdclass] [ 8498.232218] workqueue events_unbound: flags=0x2 [ 8498.233666] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8498.241819] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8498.253208] workqueue events_power_efficient: flags=0x80 [ 8498.256979] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8498.264155] pending: neigh_periodic_work [ 8498.268495] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8498.274759] pending: do_cache_clean, check_lifetime [ 8498.285374] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8498.293465] pending: neigh_periodic_work [ 8498.299671] workqueue writeback: flags=0x4a [ 8498.305901] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8498.311579] pending: wb_workfn, wb_workfn [ 8498.318446] workqueue xprtiod: flags=0xa [ 8498.322546] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8498.332214] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 8498.342338] workqueue ipv6_addrconf: flags=0x40008 [ 8498.347591] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8498.352584] pending: addrconf_verify_work [ 8498.356591] workqueue knbd0-recv: flags=0x1a [ 8498.361613] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8498.366708] in-flight: 657:recv_work [ 8498.371678] workqueue ptlrpc_pinger: flags=0x2 [ 8498.377671] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8498.385608] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 8498.394234] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6539s workers=4 idle: 20831 12642 28859 [ 8498.404267] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8488s workers=2 idle: 659 [ 8504.058127] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [ 8504.065469] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8504.115309] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8504.120470] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8504.132421] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8504.137427] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8504.150678] RSP: 0018:ffffaf9fc1543ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8504.154132] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 8504.156894] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 8504.159823] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8504.163106] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 8504.169252] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 8504.172851] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 8504.177219] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8504.180743] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 8504.185018] Call Trace: [ 8504.186276] [ 8504.187365] ? show_regs.cold.9+0x22/0x2f [ 8504.189750] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8504.192647] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8504.195616] ? watchdog+0x40/0x40 [ 8504.197987] ? hrtimer_interrupt+0x128/0x2c0 [ 8504.200856] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8504.204057] ? apic_timer_interrupt+0xf/0x20 [ 8504.205333] [ 8504.206119] ? native_safe_halt+0x2e/0x50 [ 8504.207276] kvm_wait+0xcd/0xe0 [ 8504.208301] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8504.210283] do_raw_spin_lock+0x177/0x190 [ 8504.211885] _raw_spin_lock+0x21/0x30 [ 8504.212992] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8504.215997] ? do_raw_spin_unlock+0x75/0x190 [ 8504.218211] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8504.219780] ? do_wait_intr+0xf0/0xf0 [ 8504.220806] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8504.222688] mdc_close+0x53b/0xd70 [mdc] [ 8504.224603] lmv_close+0x18d/0x490 [lmv] [ 8504.227209] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 8504.229185] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8504.233552] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8504.236680] ll_md_close+0x274/0x800 [lustre] [ 8504.239331] ll_file_release+0x128/0xc50 [lustre] [ 8504.242948] __fput+0x10d/0x440 [ 8504.253341] ____fput+0x12/0x20 [ 8504.255366] task_work_run+0xc6/0x110 [ 8504.256660] exit_to_usermode_loop+0x1dd/0x1f0 [ 8504.261734] do_syscall_64+0x3d6/0x440 [ 8504.269186] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8504.275680] RIP: 0033:0x7ff54af45cd8 [ 8504.284442] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8504.312257] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8504.317702] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 8504.325032] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 8504.335203] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 8504.343575] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 8504.360313] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8508.059059] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 8508.061704] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8508.109789] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8508.116179] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8508.127945] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8508.134303] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8508.139304] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8508.163983] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8508.178522] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 8508.191395] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 8508.207464] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 8508.212617] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 8508.219765] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 8508.228821] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 8508.242583] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8508.248014] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 8508.252022] Call Trace: [ 8508.253475] [ 8508.255049] ? show_regs.cold.9+0x22/0x2f [ 8508.257379] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8508.262299] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8508.264402] ? watchdog+0x40/0x40 [ 8508.267026] ? hrtimer_interrupt+0x128/0x2c0 [ 8508.276639] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8508.279176] ? apic_timer_interrupt+0xf/0x20 [ 8508.293317] [ 8508.297573] ? native_safe_halt+0x2e/0x50 [ 8508.300050] kvm_wait+0xcd/0xe0 [ 8508.303986] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8508.307640] do_raw_spin_lock+0x177/0x190 [ 8508.309011] _raw_spin_lock+0x21/0x30 [ 8508.322092] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8508.326402] process_one_work+0x2c8/0x700 [ 8508.328648] worker_thread+0x296/0x6e0 [ 8508.330702] ? rescuer_thread+0x570/0x570 [ 8508.333274] kthread+0x1d1/0x200 [ 8508.335238] ? set_kthread_struct+0x70/0x70 [ 8508.340267] ret_from_fork+0x1f/0x30 [ 8524.055029] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 8524.057889] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8524.060085] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 8524.078367] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8524.081926] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8524.082082] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8524.082089] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8524.082092] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8524.082108] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8524.082112] RSP: 0000:ffffaf9fc15234f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8524.082117] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 8524.082120] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 8524.082122] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8524.082124] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 8524.082127] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 8524.082130] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 8524.082134] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8524.082136] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 8524.082250] Call Trace: [ 8524.082258] [ 8524.082322] ? show_regs.cold.9+0x22/0x2f [ 8524.082335] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8524.082341] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8524.082346] ? watchdog+0x40/0x40 [ 8524.082350] ? hrtimer_interrupt+0x128/0x2c0 [ 8524.082354] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8524.082363] ? apic_timer_interrupt+0xf/0x20 [ 8524.082367] [ 8524.082370] ? native_safe_halt+0x2e/0x50 [ 8524.082375] kvm_wait+0xcd/0xe0 [ 8524.082383] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8524.082392] do_raw_spin_lock+0x177/0x190 [ 8524.082397] _raw_spin_lock+0x21/0x30 [ 8524.082402] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8524.082770] ? do_raw_spin_unlock+0x75/0x190 [ 8524.082776] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8524.082973] ? do_wait_intr+0xf0/0xf0 [ 8524.082981] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8524.083441] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8524.083955] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8524.084427] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8524.084552] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8524.084667] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8524.084698] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8524.084728] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8524.084897] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8524.093929] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8524.093941] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8524.093957] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8524.093961] RSP: 0000:ffffaf9fc1b97bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8524.093967] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 8524.093970] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 8524.093972] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 8524.093975] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 8524.093978] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8524.093982] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 8524.093986] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8524.093988] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 8524.094132] Call Trace: [ 8524.094140] [ 8524.094255] ? show_regs.cold.9+0x22/0x2f [ 8524.094334] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8524.094342] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8524.094346] ? watchdog+0x40/0x40 [ 8524.094350] ? hrtimer_interrupt+0x128/0x2c0 [ 8524.094354] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8524.094362] ? apic_timer_interrupt+0xf/0x20 [ 8524.094367] [ 8524.094370] ? native_safe_halt+0x2e/0x50 [ 8524.094374] kvm_wait+0xcd/0xe0 [ 8524.094384] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8524.094393] do_raw_spin_lock+0x177/0x190 [ 8524.094398] _raw_spin_lock+0x21/0x30 [ 8524.094404] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8524.094737] obd_mod_free+0x46/0x140 [ptlrpc] [ 8524.095080] mdc_commit_open+0x91/0xb0 [mdc] [ 8524.095122] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8524.095529] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8524.138520] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8524.138634] lmv_intent_open+0x338/0x1150 [lmv] [ 8524.138666] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8524.138797] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8524.138890] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8524.138915] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8524.139249] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8524.139400] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8524.139483] ? do_raw_spin_unlock+0x75/0x190 [ 8524.139492] ? _raw_spin_unlock+0x12/0x30 [ 8524.139500] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8524.139536] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8524.139557] ll_atomic_open+0x36a/0xd70 [lustre] [ 8524.139704] lookup_open+0x356/0xb40 [ 8524.139713] do_last+0x313/0xfc0 [ 8524.139718] ? nd_jump_root+0xe5/0x160 [ 8524.139723] ? path_init+0x437/0x520 [ 8524.139728] path_openat+0xf7/0x500 [ 8524.139733] ? do_raw_spin_unlock+0x75/0x190 [ 8524.139738] do_filp_open+0x99/0x140 [ 8524.139810] ? getname_flags+0x6e/0x330 [ 8524.139814] ? __check_object_size+0xff/0x256 [ 8524.139822] ? do_raw_spin_unlock+0x75/0x190 [ 8524.139826] ? _raw_spin_unlock+0x12/0x30 [ 8524.139830] do_sys_openat2+0x2b4/0x410 [ 8524.139834] do_sys_open+0x73/0xa0 [ 8524.139838] __x64_sys_openat+0x24/0x30 [ 8524.139842] do_syscall_64+0xc1/0x440 [ 8524.139848] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8524.139855] RIP: 0033:0x7ff54af45332 [ 8524.139862] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8524.139866] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8524.139872] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 8524.139874] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 8524.139877] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 8524.139879] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 8524.139881] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 8524.484444] after_reply+0xe2c/0x1780 [ptlrpc] [ 8524.486680] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8524.489897] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8524.492422] ? do_wait_intr+0xf0/0xf0 [ 8524.494022] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8524.496481] kthread+0x1d1/0x200 [ 8524.498358] ? set_kthread_struct+0x70/0x70 [ 8524.500232] ret_from_fork+0x1f/0x30 [ 8528.864837] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6569s! [ 8528.871466] Showing busy workqueues and worker pools: [ 8528.874244] workqueue events: flags=0x0 [ 8528.876356] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8528.880031] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 8528.884860] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8528.889212] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 8528.896352] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8528.900476] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 8528.908737] workqueue events_long: flags=0x0 [ 8528.913016] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8528.918206] pending: lu_objects_flush [obdclass] [ 8528.921540] workqueue events_unbound: flags=0x2 [ 8528.924392] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8528.927350] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8528.932315] workqueue events_power_efficient: flags=0x80 [ 8528.934791] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8528.938082] pending: neigh_periodic_work [ 8528.940234] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8528.943400] pending: do_cache_clean, check_lifetime [ 8528.946320] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8528.952296] pending: neigh_periodic_work [ 8528.954209] workqueue writeback: flags=0x4a [ 8528.956951] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8528.959197] pending: wb_workfn, wb_workfn [ 8528.961190] workqueue xprtiod: flags=0xa [ 8528.962541] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8528.964971] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 8528.967759] workqueue ipv6_addrconf: flags=0x40008 [ 8528.969261] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8528.971385] pending: addrconf_verify_work [ 8528.973242] workqueue knbd0-recv: flags=0x1a [ 8528.974666] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8528.977473] in-flight: 657:recv_work [ 8528.978919] workqueue ptlrpc_pinger: flags=0x2 [ 8528.980719] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8528.982443] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 8528.985069] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6570s workers=4 idle: 20831 12642 28859 [ 8528.987788] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8519s workers=2 idle: 659 [ 8532.058168] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [ 8532.062464] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8532.099632] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8532.105928] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8532.111649] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8532.118161] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8532.136556] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8532.142249] RAX: 0000000000000a3e RBX: ffff96848811cae0 RCX: 0000000000000001 [ 8532.152912] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 8532.156701] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8532.160899] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 8532.167356] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 8532.173681] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 8532.179977] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8532.183254] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 8532.189156] Call Trace: [ 8532.190196] [ 8532.191677] ? show_regs.cold.9+0x22/0x2f [ 8532.216608] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8532.232404] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8532.238435] ? watchdog+0x40/0x40 [ 8532.245161] ? hrtimer_interrupt+0x128/0x2c0 [ 8532.251738] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8532.266647] ? apic_timer_interrupt+0xf/0x20 [ 8532.267795] [ 8532.268460] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8532.278322] do_raw_spin_lock+0x177/0x190 [ 8532.281837] _raw_spin_lock+0x21/0x30 [ 8532.284392] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8532.288362] ? do_raw_spin_unlock+0x75/0x190 [ 8532.291579] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8532.293415] ? do_wait_intr+0xf0/0xf0 [ 8532.294527] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8532.301112] mdc_close+0x53b/0xd70 [mdc] [ 8532.307647] lmv_close+0x18d/0x490 [lmv] [ 8532.314502] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 8532.329340] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8532.337732] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8532.339170] ll_md_close+0x274/0x800 [lustre] [ 8532.340532] ll_file_release+0x128/0xc50 [lustre] [ 8532.343065] __fput+0x10d/0x440 [ 8532.344425] ____fput+0x12/0x20 [ 8532.345907] task_work_run+0xc6/0x110 [ 8532.348052] exit_to_usermode_loop+0x1dd/0x1f0 [ 8532.350579] do_syscall_64+0x3d6/0x440 [ 8532.352139] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8532.355301] RIP: 0033:0x7ff54af45cd8 [ 8532.358730] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8532.371714] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8532.379419] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 8532.382986] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 8532.388031] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 8532.397953] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 8532.405185] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8536.059021] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 8536.065189] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8536.106427] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8536.114350] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8536.121104] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8536.127553] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8536.131969] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8536.144892] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8536.153089] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 8536.155404] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 8536.168733] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 8536.174039] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 8536.181045] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 8536.189394] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 8536.201004] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8536.206334] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 8536.218230] Call Trace: [ 8536.221155] [ 8536.221935] ? show_regs.cold.9+0x22/0x2f [ 8536.223305] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8536.230356] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8536.237683] ? watchdog+0x40/0x40 [ 8536.238981] ? hrtimer_interrupt+0x128/0x2c0 [ 8536.244158] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8536.249243] ? apic_timer_interrupt+0xf/0x20 [ 8536.256524] [ 8536.257269] ? native_safe_halt+0x2e/0x50 [ 8536.259721] kvm_wait+0xcd/0xe0 [ 8536.265061] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8536.274696] do_raw_spin_lock+0x177/0x190 [ 8536.276318] _raw_spin_lock+0x21/0x30 [ 8536.282020] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8536.289287] process_one_work+0x2c8/0x700 [ 8536.295899] worker_thread+0x296/0x6e0 [ 8536.298134] ? rescuer_thread+0x570/0x570 [ 8536.301219] kthread+0x1d1/0x200 [ 8536.302925] ? set_kthread_struct+0x70/0x70 [ 8536.308506] ret_from_fork+0x1f/0x30 [ 8552.069497] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 8552.077187] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8552.148749] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8552.168964] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8552.187013] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8552.194779] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8552.215668] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8552.218086] RAX: 0000000000007cdf RBX: ffff96848811cae0 RCX: 0000000000000001 [ 8552.233575] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 8552.244963] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8552.255381] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 8552.260768] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 8552.268937] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 8552.285568] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8552.294988] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 8552.312857] Call Trace: [ 8552.316942] [ 8552.318747] ? show_regs.cold.9+0x22/0x2f [ 8552.322018] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8552.324606] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8552.326032] ? watchdog+0x40/0x40 [ 8552.332993] ? hrtimer_interrupt+0x128/0x2c0 [ 8552.336620] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8552.340807] ? apic_timer_interrupt+0xf/0x20 [ 8552.345917] [ 8552.347707] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8552.352998] do_raw_spin_lock+0x177/0x190 [ 8552.356553] _raw_spin_lock+0x21/0x30 [ 8552.358870] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8552.365759] ? do_raw_spin_unlock+0x75/0x190 [ 8552.370290] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8552.373411] ? do_wait_intr+0xf0/0xf0 [ 8552.380745] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8552.386593] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8552.392893] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8552.401357] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8552.404570] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8552.411131] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8552.422443] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8552.427911] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8552.430838] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8552.436029] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8552.443860] lmv_intent_open+0x338/0x1150 [lmv] [ 8552.446182] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8552.449902] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8552.457731] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8552.461946] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8552.466988] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8552.471007] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8552.476488] ? do_raw_spin_unlock+0x75/0x190 [ 8552.480815] ? _raw_spin_unlock+0x12/0x30 [ 8552.482785] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8552.486835] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8552.494917] ll_atomic_open+0x36a/0xd70 [lustre] [ 8552.502388] lookup_open+0x356/0xb40 [ 8552.503528] do_last+0x313/0xfc0 [ 8552.504750] ? nd_jump_root+0xe5/0x160 [ 8552.510533] ? path_init+0x437/0x520 [ 8552.512760] path_openat+0xf7/0x500 [ 8552.515600] ? do_raw_spin_unlock+0x75/0x190 [ 8552.521799] do_filp_open+0x99/0x140 [ 8552.525110] ? getname_flags+0x6e/0x330 [ 8552.526501] ? __check_object_size+0xff/0x256 [ 8552.530960] ? do_raw_spin_unlock+0x75/0x190 [ 8552.536634] ? _raw_spin_unlock+0x12/0x30 [ 8552.540449] do_sys_openat2+0x2b4/0x410 [ 8552.542606] do_sys_open+0x73/0xa0 [ 8552.545817] __x64_sys_openat+0x24/0x30 [ 8552.547073] do_syscall_64+0xc1/0x440 [ 8552.548143] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8552.556747] RIP: 0033:0x7ff54af45332 [ 8552.561952] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8552.600132] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8552.622276] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 8552.642816] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 8552.665011] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 8552.683319] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 8552.700897] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 8556.057547] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 8556.067217] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8556.133540] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8556.148335] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8556.158043] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8556.165647] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8556.190389] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8556.207010] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 8556.211682] RDX: 0000000000005aa9 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 8556.219590] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 8556.224483] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 8556.231170] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8556.240549] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 8556.246723] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8556.252368] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 8556.259649] Call Trace: [ 8556.265543] [ 8556.267485] ? show_regs.cold.9+0x22/0x2f [ 8556.269980] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8556.274146] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8556.280856] ? watchdog+0x40/0x40 [ 8556.283535] ? hrtimer_interrupt+0x128/0x2c0 [ 8556.291058] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8556.292734] ? apic_timer_interrupt+0xf/0x20 [ 8556.294224] [ 8556.299611] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8556.303788] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8556.307596] do_raw_spin_lock+0x177/0x190 [ 8556.311185] _raw_spin_lock+0x21/0x30 [ 8556.312358] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8556.314158] obd_mod_free+0x46/0x140 [ptlrpc] [ 8556.319083] mdc_commit_open+0x91/0xb0 [mdc] [ 8556.321638] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8556.326494] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8556.331181] after_reply+0xe2c/0x1780 [ptlrpc] [ 8556.336635] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8556.343374] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8556.346176] ? do_wait_intr+0xf0/0xf0 [ 8556.349380] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8556.354672] kthread+0x1d1/0x200 [ 8556.356299] ? set_kthread_struct+0x70/0x70 [ 8556.359230] ret_from_fork+0x1f/0x30 [ 8559.071041] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6600s! [ 8559.084208] Showing busy workqueues and worker pools: [ 8559.093121] workqueue events: flags=0x0 [ 8559.098568] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8559.107636] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 8559.117469] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8559.124304] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 8559.132563] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8559.135560] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 8559.139027] workqueue events_long: flags=0x0 [ 8559.143382] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8559.154677] pending: lu_objects_flush [obdclass] [ 8559.158381] workqueue events_unbound: flags=0x2 [ 8559.159996] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8559.173781] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8559.180358] workqueue events_power_efficient: flags=0x80 [ 8559.189163] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8559.193516] pending: neigh_periodic_work [ 8559.199155] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8559.209134] pending: do_cache_clean, check_lifetime [ 8559.216199] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8559.227680] pending: neigh_periodic_work [ 8559.238467] workqueue writeback: flags=0x4a [ 8559.243414] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8559.253134] pending: wb_workfn, wb_workfn [ 8559.261148] workqueue xprtiod: flags=0xa [ 8559.264027] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8559.271562] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 8559.284608] workqueue ipv6_addrconf: flags=0x40008 [ 8559.287975] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8559.300383] pending: addrconf_verify_work [ 8559.303752] workqueue knbd0-recv: flags=0x1a [ 8559.305881] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8559.316028] in-flight: 657:recv_work [ 8559.321593] workqueue ptlrpc_pinger: flags=0x2 [ 8559.326745] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8559.342738] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 8559.354673] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6600s workers=4 idle: 20831 12642 28859 [ 8559.370451] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8549s workers=2 idle: 659 [ 8560.068373] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [ 8560.086322] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8560.183366] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8560.196048] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8560.213043] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8560.218721] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8560.239067] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8560.241745] RAX: 000000000000630b RBX: ffff96848811cae0 RCX: 0000000000000001 [ 8560.243188] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 8560.247727] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8560.255816] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 8560.262927] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 8560.268164] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 8560.274014] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8560.279946] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 8560.285095] Call Trace: [ 8560.287820] [ 8560.290417] ? show_regs.cold.9+0x22/0x2f [ 8560.294139] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8560.303159] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8560.308686] ? watchdog+0x40/0x40 [ 8560.314576] ? hrtimer_interrupt+0x128/0x2c0 [ 8560.319713] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8560.321362] ? apic_timer_interrupt+0xf/0x20 [ 8560.332749] [ 8560.338467] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8560.344016] do_raw_spin_lock+0x177/0x190 [ 8560.346541] _raw_spin_lock+0x21/0x30 [ 8560.351030] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8560.361494] ? do_raw_spin_unlock+0x75/0x190 [ 8560.371744] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8560.380757] ? do_wait_intr+0xf0/0xf0 [ 8560.386681] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8560.389928] mdc_close+0x53b/0xd70 [mdc] [ 8560.406558] lmv_close+0x18d/0x490 [lmv] [ 8560.408606] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 8560.418349] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8560.429776] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8560.431624] ll_md_close+0x274/0x800 [lustre] [ 8560.441419] ll_file_release+0x128/0xc50 [lustre] [ 8560.447830] __fput+0x10d/0x440 [ 8560.450433] ____fput+0x12/0x20 [ 8560.452762] task_work_run+0xc6/0x110 [ 8560.457591] exit_to_usermode_loop+0x1dd/0x1f0 [ 8560.459159] do_syscall_64+0x3d6/0x440 [ 8560.474037] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8560.488008] RIP: 0033:0x7ff54af45cd8 [ 8560.498853] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8560.528801] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8560.533698] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 8560.537293] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 8560.542282] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 8560.549694] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 8560.557036] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8564.063989] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 8564.074579] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8564.107885] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8564.120911] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8564.126496] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8564.132276] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8564.134891] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8564.152171] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 8564.157975] RAX: 0000000000000b86 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 8564.165798] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 8564.174986] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 8564.178775] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 8564.183059] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 8564.188874] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 8564.194619] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8564.197510] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 8564.201781] Call Trace: [ 8564.204441] [ 8564.205626] ? show_regs.cold.9+0x22/0x2f [ 8564.212742] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8564.216721] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8564.222831] ? watchdog+0x40/0x40 [ 8564.226802] ? hrtimer_interrupt+0x128/0x2c0 [ 8564.231030] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8564.237981] ? apic_timer_interrupt+0xf/0x20 [ 8564.241827] [ 8564.243112] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8564.254788] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8564.260805] do_raw_spin_lock+0x177/0x190 [ 8564.262168] _raw_spin_lock+0x21/0x30 [ 8564.269496] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8564.281732] process_one_work+0x2c8/0x700 [ 8564.284560] worker_thread+0x296/0x6e0 [ 8564.286898] ? rescuer_thread+0x570/0x570 [ 8564.291926] kthread+0x1d1/0x200 [ 8564.295701] ? set_kthread_struct+0x70/0x70 [ 8564.301632] ret_from_fork+0x1f/0x30 [ 8580.067038] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 8580.077872] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8580.179626] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8580.196858] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8580.199686] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8580.211525] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8580.249847] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 8580.255883] RAX: 0000000000007c85 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 8580.262448] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 8580.287225] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8580.295358] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 8580.306490] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 8580.320106] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 8580.332393] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8580.348144] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 8580.350477] Call Trace: [ 8580.360949] [ 8580.365834] ? show_regs.cold.9+0x22/0x2f [ 8580.373609] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8580.379291] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8580.388125] ? watchdog+0x40/0x40 [ 8580.390745] ? hrtimer_interrupt+0x128/0x2c0 [ 8580.393713] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8580.397014] ? apic_timer_interrupt+0xf/0x20 [ 8580.399391] [ 8580.400377] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8580.403786] do_raw_spin_lock+0x177/0x190 [ 8580.407166] _raw_spin_lock+0x21/0x30 [ 8580.409058] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8580.414415] ? do_raw_spin_unlock+0x75/0x190 [ 8580.419359] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8580.426342] ? do_wait_intr+0xf0/0xf0 [ 8580.427754] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8580.432986] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8580.446437] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8580.449661] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8580.456403] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8580.460991] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8580.463725] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8580.469950] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8580.476797] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8580.483793] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8580.487170] lmv_intent_open+0x338/0x1150 [lmv] [ 8580.490642] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8580.494440] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8580.500802] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8580.509674] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8580.513770] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8580.518463] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8580.525509] ? do_raw_spin_unlock+0x75/0x190 [ 8580.529138] ? _raw_spin_unlock+0x12/0x30 [ 8580.530573] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8580.535317] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8580.537513] ll_atomic_open+0x36a/0xd70 [lustre] [ 8580.541759] lookup_open+0x356/0xb40 [ 8580.545447] do_last+0x313/0xfc0 [ 8580.551867] ? nd_jump_root+0xe5/0x160 [ 8580.555580] ? path_init+0x437/0x520 [ 8580.556709] path_openat+0xf7/0x500 [ 8580.560697] ? do_raw_spin_unlock+0x75/0x190 [ 8580.562174] do_filp_open+0x99/0x140 [ 8580.567616] ? getname_flags+0x6e/0x330 [ 8580.569606] ? __check_object_size+0xff/0x256 [ 8580.572848] ? do_raw_spin_unlock+0x75/0x190 [ 8580.575061] ? _raw_spin_unlock+0x12/0x30 [ 8580.581606] do_sys_openat2+0x2b4/0x410 [ 8580.583817] do_sys_open+0x73/0xa0 [ 8580.585767] __x64_sys_openat+0x24/0x30 [ 8580.589696] do_syscall_64+0xc1/0x440 [ 8580.593580] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8580.597967] RIP: 0033:0x7ff54af45332 [ 8580.601747] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8580.616929] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8580.622913] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 8580.630252] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 8580.632520] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 8580.640785] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 8580.646965] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 8584.055060] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 8584.059611] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8584.109168] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8584.128116] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8584.139052] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8584.146266] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8584.174002] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8584.184868] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 8584.200658] RDX: 0000000000006643 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 8584.213247] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 8584.223065] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 8584.230707] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8584.236928] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 8584.249825] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8584.253810] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 8584.268142] Call Trace: [ 8584.269602] [ 8584.272461] ? show_regs.cold.9+0x22/0x2f [ 8584.273937] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8584.278131] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8584.279700] ? watchdog+0x40/0x40 [ 8584.280855] ? hrtimer_interrupt+0x128/0x2c0 [ 8584.287442] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8584.293805] ? apic_timer_interrupt+0xf/0x20 [ 8584.295373] [ 8584.296134] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8584.303667] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8584.313467] do_raw_spin_lock+0x177/0x190 [ 8584.319822] _raw_spin_lock+0x21/0x30 [ 8584.324894] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8584.326919] obd_mod_free+0x46/0x140 [ptlrpc] [ 8584.335867] mdc_commit_open+0x91/0xb0 [mdc] [ 8584.339550] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8584.342796] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8584.351378] after_reply+0xe2c/0x1780 [ptlrpc] [ 8584.359085] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8584.364197] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8584.369540] ? do_wait_intr+0xf0/0xf0 [ 8584.373719] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8584.379363] kthread+0x1d1/0x200 [ 8584.380868] ? set_kthread_struct+0x70/0x70 [ 8584.386765] ret_from_fork+0x1f/0x30 [ 8588.061059] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [ 8588.073975] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8588.137078] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8588.146895] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8588.156934] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8588.159096] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8588.190511] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8588.200051] RAX: 000000000000491b RBX: ffff96848811cae0 RCX: 0000000000000001 [ 8588.209946] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 8588.219131] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8588.223280] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 8588.230839] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 8588.235766] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 8588.241470] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8588.245875] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 8588.250368] Call Trace: [ 8588.251915] [ 8588.255312] ? show_regs.cold.9+0x22/0x2f [ 8588.256558] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8588.259623] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8588.262005] ? watchdog+0x40/0x40 [ 8588.268282] ? hrtimer_interrupt+0x128/0x2c0 [ 8588.276468] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8588.283871] ? apic_timer_interrupt+0xf/0x20 [ 8588.286244] [ 8588.287616] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8588.293536] do_raw_spin_lock+0x177/0x190 [ 8588.296541] _raw_spin_lock+0x21/0x30 [ 8588.299218] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8588.302206] ? do_raw_spin_unlock+0x75/0x190 [ 8588.304422] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8588.310856] ? do_wait_intr+0xf0/0xf0 [ 8588.312769] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8588.314979] mdc_close+0x53b/0xd70 [mdc] [ 8588.320015] lmv_close+0x18d/0x490 [lmv] [ 8588.321562] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 8588.326637] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8588.328742] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8588.345716] ll_md_close+0x274/0x800 [lustre] [ 8588.350280] ll_file_release+0x128/0xc50 [lustre] [ 8588.353530] __fput+0x10d/0x440 [ 8588.355273] ____fput+0x12/0x20 [ 8588.357011] task_work_run+0xc6/0x110 [ 8588.359998] exit_to_usermode_loop+0x1dd/0x1f0 [ 8588.366532] do_syscall_64+0x3d6/0x440 [ 8588.373331] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8588.374941] RIP: 0033:0x7ff54af45cd8 [ 8588.379729] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8588.394765] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8588.401697] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 8588.406521] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 8588.415593] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 8588.419658] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 8588.429936] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8589.797986] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6630s! [ 8589.827463] Showing busy workqueues and worker pools: [ 8589.833698] workqueue events: flags=0x0 [ 8589.837912] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8589.847799] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 8589.862130] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8589.865792] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 8589.872181] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8589.876304] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 8589.880415] workqueue events_long: flags=0x0 [ 8589.883488] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8589.887994] pending: lu_objects_flush [obdclass] [ 8589.891518] workqueue events_unbound: flags=0x2 [ 8589.895835] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8589.899630] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8589.909432] workqueue events_power_efficient: flags=0x80 [ 8589.912572] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8589.917021] pending: neigh_periodic_work [ 8589.919702] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8589.923157] pending: do_cache_clean, check_lifetime [ 8589.926136] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8589.929218] pending: neigh_periodic_work [ 8589.931417] workqueue writeback: flags=0x4a [ 8589.933545] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8589.936401] pending: wb_workfn, wb_workfn [ 8589.939009] workqueue xprtiod: flags=0xa [ 8589.940876] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8589.943861] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 8589.947744] workqueue ipv6_addrconf: flags=0x40008 [ 8589.950296] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8589.953226] pending: addrconf_verify_work [ 8589.955797] workqueue knbd0-recv: flags=0x1a [ 8589.957639] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8589.960714] in-flight: 657:recv_work [ 8589.962407] workqueue ptlrpc_pinger: flags=0x2 [ 8589.964862] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8589.967516] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 8589.970071] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6631s workers=4 idle: 20831 12642 28859 [ 8589.974144] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8580s workers=2 idle: 659 [ 8592.059267] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 8592.064440] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8592.106032] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8592.120374] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8592.136793] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8592.151959] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8592.166755] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8592.180705] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8592.185921] RAX: 0000000000006d5d RBX: ffff96848811cae0 RCX: 0000000000000001 [ 8592.196297] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 8592.202867] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 8592.211692] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 8592.216927] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 8592.232603] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 8592.244384] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8592.248375] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 8592.255481] Call Trace: [ 8592.259269] [ 8592.260782] ? show_regs.cold.9+0x22/0x2f [ 8592.266111] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8592.273613] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8592.276656] ? watchdog+0x40/0x40 [ 8592.279259] ? hrtimer_interrupt+0x128/0x2c0 [ 8592.283896] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8592.288548] ? apic_timer_interrupt+0xf/0x20 [ 8592.296832] [ 8592.300693] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8592.303484] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8592.307374] do_raw_spin_lock+0x177/0x190 [ 8592.309738] _raw_spin_lock+0x21/0x30 [ 8592.312191] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8592.315460] process_one_work+0x2c8/0x700 [ 8592.317778] worker_thread+0x296/0x6e0 [ 8592.320052] ? rescuer_thread+0x570/0x570 [ 8592.322314] kthread+0x1d1/0x200 [ 8592.324050] ? set_kthread_struct+0x70/0x70 [ 8592.326287] ret_from_fork+0x1f/0x30 [ 8608.065094] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 8608.068644] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8608.095965] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8608.103738] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8608.107018] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8608.116158] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8608.129319] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8608.135493] RAX: 0000000000006c57 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 8608.140734] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 8608.148072] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8608.153710] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 8608.156547] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 8608.159411] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 8608.163553] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8608.165862] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 8608.173096] Call Trace: [ 8608.174017] [ 8608.174648] ? show_regs.cold.9+0x22/0x2f [ 8608.181195] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8608.183730] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8608.185884] ? watchdog+0x40/0x40 [ 8608.187733] ? hrtimer_interrupt+0x128/0x2c0 [ 8608.190040] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8608.195054] ? apic_timer_interrupt+0xf/0x20 [ 8608.198809] [ 8608.199665] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8608.203235] do_raw_spin_lock+0x177/0x190 [ 8608.211509] _raw_spin_lock+0x21/0x30 [ 8608.214914] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8608.217275] ? do_raw_spin_unlock+0x75/0x190 [ 8608.222934] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8608.228660] ? do_wait_intr+0xf0/0xf0 [ 8608.231544] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8608.234494] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8608.239423] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8608.241578] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8608.243205] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8608.244667] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8608.247356] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8608.252315] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8608.258587] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8608.267894] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8608.273503] lmv_intent_open+0x338/0x1150 [lmv] [ 8608.277888] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8608.287834] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8608.289681] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8608.294314] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8608.296708] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8608.302749] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8608.306140] ? do_raw_spin_unlock+0x75/0x190 [ 8608.307259] ? _raw_spin_unlock+0x12/0x30 [ 8608.308382] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8608.310492] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8608.313031] ll_atomic_open+0x36a/0xd70 [lustre] [ 8608.315196] lookup_open+0x356/0xb40 [ 8608.316156] do_last+0x313/0xfc0 [ 8608.317208] ? nd_jump_root+0xe5/0x160 [ 8608.318273] ? path_init+0x437/0x520 [ 8608.319357] path_openat+0xf7/0x500 [ 8608.320643] ? do_raw_spin_unlock+0x75/0x190 [ 8608.323660] do_filp_open+0x99/0x140 [ 8608.329884] ? getname_flags+0x6e/0x330 [ 8608.334628] ? __check_object_size+0xff/0x256 [ 8608.336722] ? do_raw_spin_unlock+0x75/0x190 [ 8608.343405] ? _raw_spin_unlock+0x12/0x30 [ 8608.344672] do_sys_openat2+0x2b4/0x410 [ 8608.349909] do_sys_open+0x73/0xa0 [ 8608.354872] __x64_sys_openat+0x24/0x30 [ 8608.356405] do_syscall_64+0xc1/0x440 [ 8608.358108] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8608.360096] RIP: 0033:0x7ff54af45332 [ 8608.361249] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8608.372573] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8608.376192] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 8608.379863] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 8608.382698] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 8608.388120] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 8608.391335] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 8616.055013] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [ 8616.059998] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [ 8616.063127] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8616.063214] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8616.063221] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8616.063224] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8616.063238] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8616.063242] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8616.063247] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 8616.063249] RDX: 00000000000038e4 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 8616.063251] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 8616.063253] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 8616.063255] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8616.063258] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 8616.063260] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8616.063263] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 8616.063272] Call Trace: [ 8616.063277] [ 8616.063283] ? show_regs.cold.9+0x22/0x2f [ 8616.063294] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8616.063301] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8616.063305] ? watchdog+0x40/0x40 [ 8616.063308] ? hrtimer_interrupt+0x128/0x2c0 [ 8616.063313] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8616.063321] ? apic_timer_interrupt+0xf/0x20 [ 8616.063326] [ 8616.063330] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8616.063335] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8616.063339] do_raw_spin_lock+0x177/0x190 [ 8616.063344] _raw_spin_lock+0x21/0x30 [ 8616.063350] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8616.063650] obd_mod_free+0x46/0x140 [ptlrpc] [ 8616.063927] mdc_commit_open+0x91/0xb0 [mdc] [ 8616.063972] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8616.064384] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8616.064612] after_reply+0xe2c/0x1780 [ptlrpc] [ 8616.065014] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8616.072071] Modules linked in: [ 8616.095291] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8616.095963] ? do_wait_intr+0xf0/0xf0 [ 8616.095975] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8616.096202] kthread+0x1d1/0x200 [ 8616.096209] ? set_kthread_struct+0x70/0x70 [ 8616.096213] ret_from_fork+0x1f/0x30 [ 8616.315279] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8616.381652] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8616.397319] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8616.408610] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8616.410540] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8616.423459] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8616.432151] RAX: 0000000000004b0b RBX: ffff96848811cae0 RCX: 0000000000000001 [ 8616.439678] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 8616.443305] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8616.447399] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 8616.454985] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 8616.458567] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 8616.466937] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8616.473219] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 8616.479678] Call Trace: [ 8616.481780] [ 8616.482465] ? show_regs.cold.9+0x22/0x2f [ 8616.487278] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8616.489551] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8616.491488] ? watchdog+0x40/0x40 [ 8616.493986] ? hrtimer_interrupt+0x128/0x2c0 [ 8616.496520] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8616.500940] ? apic_timer_interrupt+0xf/0x20 [ 8616.505171] [ 8616.508234] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8616.514114] do_raw_spin_lock+0x177/0x190 [ 8616.517496] _raw_spin_lock+0x21/0x30 [ 8616.518867] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8616.525607] ? do_raw_spin_unlock+0x75/0x190 [ 8616.530054] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8616.532002] ? do_wait_intr+0xf0/0xf0 [ 8616.536107] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8616.538950] mdc_close+0x53b/0xd70 [mdc] [ 8616.542430] lmv_close+0x18d/0x490 [lmv] [ 8616.547056] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 8616.550288] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8616.554336] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8616.558612] ll_md_close+0x274/0x800 [lustre] [ 8616.564576] ll_file_release+0x128/0xc50 [lustre] [ 8616.567650] __fput+0x10d/0x440 [ 8616.568961] ____fput+0x12/0x20 [ 8616.570172] task_work_run+0xc6/0x110 [ 8616.571453] exit_to_usermode_loop+0x1dd/0x1f0 [ 8616.573540] do_syscall_64+0x3d6/0x440 [ 8616.575174] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8616.577230] RIP: 0033:0x7ff54af45cd8 [ 8616.578543] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8616.586243] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8616.589323] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 8616.592247] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 8616.594683] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 8616.599398] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 8616.603643] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8620.000568] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6661s! [ 8620.006298] Showing busy workqueues and worker pools: [ 8620.009855] workqueue events: flags=0x0 [ 8620.016470] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8620.028985] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 8620.040507] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8620.046184] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 8620.055090] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8620.060203] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 8620.060487] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 8620.060495] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8620.060573] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8620.060580] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8620.060587] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8620.060974] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8620.060985] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8620.060989] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8620.060994] RAX: 0000000000006be9 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 8620.060997] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 8620.060999] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 8620.061002] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 8620.061005] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 8620.061009] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 8620.061012] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8620.061014] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 8620.061024] Call Trace: [ 8620.061030] [ 8620.061036] ? show_regs.cold.9+0x22/0x2f [ 8620.061046] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8620.061054] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8620.061059] ? watchdog+0x40/0x40 [ 8620.061063] ? hrtimer_interrupt+0x128/0x2c0 [ 8620.062320] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8620.062340] ? apic_timer_interrupt+0xf/0x20 [ 8620.062345] [ 8620.062349] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8620.062356] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8620.062362] do_raw_spin_lock+0x177/0x190 [ 8620.062368] _raw_spin_lock+0x21/0x30 [ 8620.062374] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8620.062633] process_one_work+0x2c8/0x700 [ 8620.062642] worker_thread+0x296/0x6e0 [ 8620.062646] ? rescuer_thread+0x570/0x570 [ 8620.062650] kthread+0x1d1/0x200 [ 8620.062655] ? set_kthread_struct+0x70/0x70 [ 8620.062659] ret_from_fork+0x1f/0x30 [ 8620.331375] [ 8620.332312] workqueue events_long: flags=0x0 [ 8620.334872] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8620.338415] pending: lu_objects_flush [obdclass] [ 8620.342150] workqueue events_unbound: flags=0x2 [ 8620.344275] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8620.348065] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8620.354068] workqueue events_power_efficient: flags=0x80 [ 8620.357202] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8620.360589] pending: neigh_periodic_work [ 8620.363033] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8620.366337] pending: do_cache_clean, check_lifetime [ 8620.369295] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8620.373199] pending: neigh_periodic_work [ 8620.375793] workqueue writeback: flags=0x4a [ 8620.378051] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8620.380402] pending: wb_workfn, wb_workfn [ 8620.381983] workqueue xprtiod: flags=0xa [ 8620.383534] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8620.386171] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 8620.389269] workqueue ipv6_addrconf: flags=0x40008 [ 8620.390792] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8620.392607] pending: addrconf_verify_work [ 8620.393821] workqueue knbd0-recv: flags=0x1a [ 8620.394982] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8620.397546] in-flight: 657:recv_work [ 8620.399152] workqueue ptlrpc_pinger: flags=0x2 [ 8620.401175] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8620.403964] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 8620.406933] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6661s workers=4 idle: 20831 12642 28859 [ 8620.410325] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8610s workers=2 idle: 659 [ 8636.060054] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 8636.066135] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8636.094762] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8636.100845] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8636.107813] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8636.109766] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8636.124915] RSP: 0000:ffffaf9fc15234f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8636.128963] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 8636.133155] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 8636.138324] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8636.144828] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 8636.149117] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 8636.152949] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 8636.156229] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8636.158736] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 8636.161623] Call Trace: [ 8636.162611] [ 8636.163668] ? show_regs.cold.9+0x22/0x2f [ 8636.165743] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8636.168221] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8636.170859] ? watchdog+0x40/0x40 [ 8636.172401] ? hrtimer_interrupt+0x128/0x2c0 [ 8636.174140] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8636.176712] ? apic_timer_interrupt+0xf/0x20 [ 8636.178368] [ 8636.179255] ? native_safe_halt+0x2e/0x50 [ 8636.180408] kvm_wait+0xcd/0xe0 [ 8636.181714] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8636.183131] do_raw_spin_lock+0x177/0x190 [ 8636.184984] _raw_spin_lock+0x21/0x30 [ 8636.186398] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8636.189048] ? do_raw_spin_unlock+0x75/0x190 [ 8636.190877] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8636.196179] ? do_wait_intr+0xf0/0xf0 [ 8636.199271] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8636.202101] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8636.205161] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8636.208472] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8636.211112] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8636.216836] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8636.221405] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8636.223799] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8636.231422] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8636.238287] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8636.242558] lmv_intent_open+0x338/0x1150 [lmv] [ 8636.248930] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8636.252732] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8636.258444] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8636.263447] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8636.266727] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8636.274768] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8636.277192] ? do_raw_spin_unlock+0x75/0x190 [ 8636.281925] ? _raw_spin_unlock+0x12/0x30 [ 8636.289789] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8636.294524] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8636.303571] ll_atomic_open+0x36a/0xd70 [lustre] [ 8636.307030] lookup_open+0x356/0xb40 [ 8636.309856] do_last+0x313/0xfc0 [ 8636.313880] ? nd_jump_root+0xe5/0x160 [ 8636.318318] ? path_init+0x437/0x520 [ 8636.319981] path_openat+0xf7/0x500 [ 8636.325096] ? do_raw_spin_unlock+0x75/0x190 [ 8636.326425] do_filp_open+0x99/0x140 [ 8636.327710] ? getname_flags+0x6e/0x330 [ 8636.329048] ? __check_object_size+0xff/0x256 [ 8636.330270] ? do_raw_spin_unlock+0x75/0x190 [ 8636.331305] ? _raw_spin_unlock+0x12/0x30 [ 8636.332360] do_sys_openat2+0x2b4/0x410 [ 8636.333589] do_sys_open+0x73/0xa0 [ 8636.334850] __x64_sys_openat+0x24/0x30 [ 8636.336141] do_syscall_64+0xc1/0x440 [ 8636.337621] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8636.339435] RIP: 0033:0x7ff54af45332 [ 8636.340572] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8636.346326] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8636.348826] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 8636.351149] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 8636.353287] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 8636.355406] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 8636.357464] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 8644.059647] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 8644.067904] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8644.111207] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8644.119204] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8644.123318] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8644.126338] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8644.137665] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8644.150053] RAX: 0000000000006c01 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 8644.153901] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 8644.156084] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8644.164842] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 8644.167464] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 8644.183080] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 8644.188496] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8644.193713] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 8644.202039] Call Trace: [ 8644.209294] [ 8644.210465] ? show_regs.cold.9+0x22/0x2f [ 8644.215054] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8644.217238] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8644.221317] ? watchdog+0x40/0x40 [ 8644.224225] ? hrtimer_interrupt+0x128/0x2c0 [ 8644.227430] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8644.230822] ? apic_timer_interrupt+0xf/0x20 [ 8644.233681] [ 8644.234792] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8644.237698] do_raw_spin_lock+0x177/0x190 [ 8644.242144] _raw_spin_lock+0x21/0x30 [ 8644.247340] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8644.257352] ? do_raw_spin_unlock+0x75/0x190 [ 8644.264351] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8644.269068] ? do_wait_intr+0xf0/0xf0 [ 8644.275618] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8644.280716] mdc_close+0x53b/0xd70 [mdc] [ 8644.284368] lmv_close+0x18d/0x490 [lmv] [ 8644.287842] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 8644.296001] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8644.297968] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8644.299369] ll_md_close+0x274/0x800 [lustre] [ 8644.302242] ll_file_release+0x128/0xc50 [lustre] [ 8644.304612] __fput+0x10d/0x440 [ 8644.308909] ____fput+0x12/0x20 [ 8644.310120] task_work_run+0xc6/0x110 [ 8644.312258] exit_to_usermode_loop+0x1dd/0x1f0 [ 8644.314053] do_syscall_64+0x3d6/0x440 [ 8644.319141] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8644.322569] RIP: 0033:0x7ff54af45cd8 [ 8644.324137] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8644.334910] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8644.341946] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 8644.346127] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 8644.349839] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 8644.353212] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 8644.356818] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8648.055059] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [ 8648.059577] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 8648.059775] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8648.059848] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8648.059854] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8648.059857] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8648.059871] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8648.059876] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8648.059880] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 8648.059883] RDX: 0000000000004c71 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 8648.059885] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 8648.059888] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 8648.059890] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8648.059893] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 8648.059897] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8648.059899] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 8648.059908] Call Trace: [ 8648.059914] [ 8648.059920] ? show_regs.cold.9+0x22/0x2f [ 8648.059930] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8648.059937] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8648.059941] ? watchdog+0x40/0x40 [ 8648.059946] ? hrtimer_interrupt+0x128/0x2c0 [ 8648.059951] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8648.059960] ? apic_timer_interrupt+0xf/0x20 [ 8648.059965] [ 8648.059968] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8648.059973] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8648.059978] do_raw_spin_lock+0x177/0x190 [ 8648.059983] _raw_spin_lock+0x21/0x30 [ 8648.059988] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8648.060301] obd_mod_free+0x46/0x140 [ptlrpc] [ 8648.060517] mdc_commit_open+0x91/0xb0 [mdc] [ 8648.060667] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8648.061136] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8648.061346] after_reply+0xe2c/0x1780 [ptlrpc] [ 8648.061818] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8648.062031] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8648.062287] ? do_wait_intr+0xf0/0xf0 [ 8648.062297] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8648.062492] kthread+0x1d1/0x200 [ 8648.062498] ? set_kthread_struct+0x70/0x70 [ 8648.062502] ret_from_fork+0x1f/0x30 [ 8648.344291] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8648.365437] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8648.371222] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8648.383203] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8648.388238] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8648.392591] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8648.406101] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8648.410568] RAX: 0000000000007e4e RBX: ffff96848811cae0 RCX: 0000000000000001 [ 8648.417920] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 8648.424086] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 8648.428138] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 8648.439149] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 8648.446329] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 8648.452207] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8648.456127] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 8648.460512] Call Trace: [ 8648.462172] [ 8648.463506] ? show_regs.cold.9+0x22/0x2f [ 8648.466287] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8648.473512] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8648.479241] ? watchdog+0x40/0x40 [ 8648.484551] ? hrtimer_interrupt+0x128/0x2c0 [ 8648.489147] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8648.493780] ? apic_timer_interrupt+0xf/0x20 [ 8648.498153] [ 8648.499951] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8648.504944] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8648.514082] do_raw_spin_lock+0x177/0x190 [ 8648.516604] _raw_spin_lock+0x21/0x30 [ 8648.521136] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8648.527928] process_one_work+0x2c8/0x700 [ 8648.533285] worker_thread+0x296/0x6e0 [ 8648.535624] ? rescuer_thread+0x570/0x570 [ 8648.537332] kthread+0x1d1/0x200 [ 8648.538811] ? set_kthread_struct+0x70/0x70 [ 8648.540641] ret_from_fork+0x1f/0x30 [ 8650.719031] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6691s! [ 8650.729119] Showing busy workqueues and worker pools: [ 8650.734581] workqueue events: flags=0x0 [ 8650.742017] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8650.750736] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 8650.764459] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8650.772765] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 8650.785093] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8650.796924] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 8650.810344] workqueue events_long: flags=0x0 [ 8650.813880] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8650.822696] pending: lu_objects_flush [obdclass] [ 8650.825838] workqueue events_unbound: flags=0x2 [ 8650.828503] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8650.831505] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8650.837315] workqueue events_power_efficient: flags=0x80 [ 8650.840173] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8650.843709] pending: neigh_periodic_work [ 8650.845869] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8650.851022] pending: do_cache_clean, check_lifetime [ 8650.853655] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8650.857734] pending: neigh_periodic_work [ 8650.860732] workqueue writeback: flags=0x4a [ 8650.865206] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8650.874998] pending: wb_workfn, wb_workfn [ 8650.877188] workqueue xprtiod: flags=0xa [ 8650.879058] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8650.891569] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 8650.897780] workqueue ipv6_addrconf: flags=0x40008 [ 8650.899669] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8650.908966] pending: addrconf_verify_work [ 8650.911653] workqueue knbd0-recv: flags=0x1a [ 8650.914352] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8650.920768] in-flight: 657:recv_work [ 8650.925312] workqueue ptlrpc_pinger: flags=0x2 [ 8650.928889] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8650.932463] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 8650.935488] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6692s workers=4 idle: 20831 12642 28859 [ 8650.939585] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8641s workers=2 idle: 659 [ 8664.063465] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [ 8664.067629] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8664.107279] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8664.119642] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8664.133493] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8664.144073] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8664.171875] RSP: 0000:ffffaf9fc15234f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8664.181747] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 8664.187762] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 8664.195272] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8664.199994] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 8664.217173] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 8664.230541] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 8664.244188] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8664.251557] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 8664.265411] Call Trace: [ 8664.268551] [ 8664.275513] ? show_regs.cold.9+0x22/0x2f [ 8664.284703] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8664.295363] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8664.302129] ? watchdog+0x40/0x40 [ 8664.306400] ? hrtimer_interrupt+0x128/0x2c0 [ 8664.313532] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8664.317008] ? apic_timer_interrupt+0xf/0x20 [ 8664.321882] [ 8664.329161] ? native_safe_halt+0x2e/0x50 [ 8664.331326] kvm_wait+0xcd/0xe0 [ 8664.332798] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8664.339399] do_raw_spin_lock+0x177/0x190 [ 8664.347334] _raw_spin_lock+0x21/0x30 [ 8664.349430] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8664.354816] ? do_raw_spin_unlock+0x75/0x190 [ 8664.362000] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8664.370853] ? do_wait_intr+0xf0/0xf0 [ 8664.373459] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8664.383807] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8664.393547] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8664.404819] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8664.410283] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8664.412837] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8664.418859] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8664.426315] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8664.427988] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8664.433802] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8664.440345] lmv_intent_open+0x338/0x1150 [lmv] [ 8664.441707] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8664.446702] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8664.450758] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8664.453641] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8664.456353] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8664.460288] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8664.462688] ? do_raw_spin_unlock+0x75/0x190 [ 8664.465014] ? _raw_spin_unlock+0x12/0x30 [ 8664.467677] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8664.470609] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8664.478511] ll_atomic_open+0x36a/0xd70 [lustre] [ 8664.486340] lookup_open+0x356/0xb40 [ 8664.492744] do_last+0x313/0xfc0 [ 8664.502687] ? nd_jump_root+0xe5/0x160 [ 8664.504613] ? path_init+0x437/0x520 [ 8664.510539] path_openat+0xf7/0x500 [ 8664.519090] ? do_raw_spin_unlock+0x75/0x190 [ 8664.520661] do_filp_open+0x99/0x140 [ 8664.529987] ? getname_flags+0x6e/0x330 [ 8664.531544] ? __check_object_size+0xff/0x256 [ 8664.535503] ? do_raw_spin_unlock+0x75/0x190 [ 8664.537260] ? _raw_spin_unlock+0x12/0x30 [ 8664.540492] do_sys_openat2+0x2b4/0x410 [ 8664.542568] do_sys_open+0x73/0xa0 [ 8664.558864] __x64_sys_openat+0x24/0x30 [ 8664.560203] do_syscall_64+0xc1/0x440 [ 8664.561548] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8664.577250] RIP: 0033:0x7ff54af45332 [ 8664.578654] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8664.595533] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8664.601780] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 8664.620270] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 8664.626369] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 8664.638069] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 8664.646700] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 8672.062925] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 8672.075765] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8672.134288] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8672.150825] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8672.166457] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8672.170605] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8672.191766] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8672.202541] RAX: 0000000000002810 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 8672.209269] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 8672.220437] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8672.228650] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 8672.234611] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 8672.251129] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 8672.265122] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8672.273663] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 8672.277803] Call Trace: [ 8672.281286] [ 8672.284487] ? show_regs.cold.9+0x22/0x2f [ 8672.294111] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8672.295903] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8672.299529] ? watchdog+0x40/0x40 [ 8672.301918] ? hrtimer_interrupt+0x128/0x2c0 [ 8672.307794] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8672.309940] ? apic_timer_interrupt+0xf/0x20 [ 8672.311913] [ 8672.317510] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8672.321464] do_raw_spin_lock+0x177/0x190 [ 8672.328906] _raw_spin_lock+0x21/0x30 [ 8672.336093] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8672.338326] ? do_raw_spin_unlock+0x75/0x190 [ 8672.346655] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8672.352781] ? do_wait_intr+0xf0/0xf0 [ 8672.358115] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8672.371853] mdc_close+0x53b/0xd70 [mdc] [ 8672.381877] lmv_close+0x18d/0x490 [lmv] [ 8672.391493] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 8672.417422] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8672.434610] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8672.448599] ll_md_close+0x274/0x800 [lustre] [ 8672.463365] ll_file_release+0x128/0xc50 [lustre] [ 8672.473384] __fput+0x10d/0x440 [ 8672.477460] ____fput+0x12/0x20 [ 8672.480905] task_work_run+0xc6/0x110 [ 8672.484795] exit_to_usermode_loop+0x1dd/0x1f0 [ 8672.489662] do_syscall_64+0x3d6/0x440 [ 8672.499258] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8672.509440] RIP: 0033:0x7ff54af45cd8 [ 8672.517541] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8672.554968] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8672.573079] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 8672.586078] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 8672.604923] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 8672.619138] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 8672.633885] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8674.031606] rcu: INFO: rcu_sched self-detected stall on CPU [ 8674.043294] rcu: 0-...!: (3488118 ticks this GP) idle=98a/1/0x4000000000000002 softirq=125154/125154 fqs=959 [ 8674.048890] (t=6720191 jiffies g=253585 q=5015) [ 8674.052193] rcu: rcu_sched kthread starved for 6715151 jiffies! g253585 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [ 8674.059276] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 8674.064532] rcu: RCU grace-period kthread stack dump: [ 8674.067250] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 8674.072184] Call Trace: [ 8674.073273] __schedule+0x351/0xcb0 [ 8674.075149] schedule+0xc0/0x180 [ 8674.076672] schedule_timeout+0xb4/0x190 [ 8674.080636] ? do_raw_spin_unlock+0x75/0x190 [ 8674.082954] ? __next_timer_interrupt+0x160/0x160 [ 8674.085608] ? prepare_to_swait_event+0xf3/0x140 [ 8674.088636] rcu_gp_kthread+0x8dd/0x15b0 [ 8674.090403] ? force_qs_rnp+0x2c0/0x2c0 [ 8674.091815] kthread+0x1d1/0x200 [ 8674.102473] ? set_kthread_struct+0x70/0x70 [ 8674.109548] ret_from_fork+0x1f/0x30 [ 8674.113273] rcu: Stack dump where RCU GP kthread last ran: [ 8674.118874] Sending NMI from CPU 0 to CPUs 2: [ 8674.129812] NMI backtrace for cpu 2 skipped: idling at native_safe_halt+0x2e/0x50 [ 8674.137621] NMI backtrace for cpu 0 [ 8674.148815] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8674.169225] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8674.198507] Call Trace: [ 8674.202514] [ 8674.203722] ? dump_stack+0xbb/0x10e [ 8674.209816] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 8674.215442] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 8674.223842] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 8674.238164] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 8674.245452] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 8674.251190] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 8674.257728] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 8674.261896] ? update_process_times+0x6d/0xc0 [ 8674.267809] ? tick_sched_handle.isra.18+0x27/0x90 [ 8674.269494] ? tick_sched_timer+0x53/0xb0 [ 8674.277948] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8674.279484] ? tick_sched_handle.isra.18+0x90/0x90 [ 8674.281240] ? hrtimer_interrupt+0x128/0x2c0 [ 8674.297054] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8674.308595] ? apic_timer_interrupt+0xf/0x20 [ 8674.315767] [ 8674.322539] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8674.326898] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8674.340647] ? do_raw_spin_lock+0x177/0x190 [ 8674.352600] ? _raw_spin_lock+0x21/0x30 [ 8674.356516] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8674.361921] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 8674.375793] ? mdc_commit_open+0x91/0xb0 [mdc] [ 8674.385726] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8674.394173] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8674.403790] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 8674.414163] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8674.427269] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8674.437687] ? do_wait_intr+0xf0/0xf0 [ 8674.441323] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8674.450907] ? kthread+0x1d1/0x200 [ 8674.452774] ? set_kthread_struct+0x70/0x70 [ 8674.463365] ? ret_from_fork+0x1f/0x30 [ 8681.439090] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6722s! [ 8681.443981] Showing busy workqueues and worker pools: [ 8681.459289] workqueue events: flags=0x0 [ 8681.461976] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8681.470621] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 8681.478649] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8681.484338] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 8681.502571] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8681.510581] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 8681.515026] workqueue events_long: flags=0x0 [ 8681.517558] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8681.521807] pending: lu_objects_flush [obdclass] [ 8681.525250] workqueue events_unbound: flags=0x2 [ 8681.527544] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8681.530498] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8681.535593] workqueue events_power_efficient: flags=0x80 [ 8681.538697] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8681.542752] pending: neigh_periodic_work [ 8681.544806] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8681.548607] pending: do_cache_clean, check_lifetime [ 8681.551802] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8681.555072] pending: neigh_periodic_work [ 8681.556711] workqueue writeback: flags=0x4a [ 8681.558627] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8681.561257] pending: wb_workfn, wb_workfn [ 8681.563537] workqueue xprtiod: flags=0xa [ 8681.565738] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8681.569207] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 8681.572907] workqueue ipv6_addrconf: flags=0x40008 [ 8681.575257] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8681.577539] pending: addrconf_verify_work [ 8681.579692] workqueue knbd0-recv: flags=0x1a [ 8681.582331] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8681.585701] in-flight: 657:recv_work [ 8681.587684] workqueue ptlrpc_pinger: flags=0x2 [ 8681.590359] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8681.593467] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 8681.596480] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6722s workers=4 idle: 20831 12642 28859 [ 8681.600688] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8671s workers=2 idle: 659 [ 8692.061842] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [ 8692.077545] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8692.125569] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8692.135443] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8692.140658] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8692.147556] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8692.161539] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8692.164405] RAX: 0000000000003007 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 8692.172586] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 8692.178706] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8692.193088] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 8692.204977] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 8692.212847] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 8692.222952] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8692.228197] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 8692.238529] Call Trace: [ 8692.241500] [ 8692.244537] ? show_regs.cold.9+0x22/0x2f [ 8692.250903] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8692.256478] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8692.258731] ? watchdog+0x40/0x40 [ 8692.263268] ? hrtimer_interrupt+0x128/0x2c0 [ 8692.272412] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8692.278261] ? apic_timer_interrupt+0xf/0x20 [ 8692.286664] [ 8692.287425] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8692.300893] do_raw_spin_lock+0x177/0x190 [ 8692.303427] _raw_spin_lock+0x21/0x30 [ 8692.305674] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8692.310947] ? do_raw_spin_unlock+0x75/0x190 [ 8692.315789] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8692.322559] ? do_wait_intr+0xf0/0xf0 [ 8692.326472] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8692.330256] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8692.335771] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8692.344144] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8692.349973] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8692.353534] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8692.356887] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8692.364612] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8692.370546] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8692.384333] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8692.391265] lmv_intent_open+0x338/0x1150 [lmv] [ 8692.398800] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8692.402172] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8692.406451] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8692.416860] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8692.421495] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8692.426277] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8692.429488] ? do_raw_spin_unlock+0x75/0x190 [ 8692.431724] ? _raw_spin_unlock+0x12/0x30 [ 8692.433784] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8692.436321] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8692.439196] ll_atomic_open+0x36a/0xd70 [lustre] [ 8692.442590] lookup_open+0x356/0xb40 [ 8692.444495] do_last+0x313/0xfc0 [ 8692.446722] ? nd_jump_root+0xe5/0x160 [ 8692.451575] ? path_init+0x437/0x520 [ 8692.454383] path_openat+0xf7/0x500 [ 8692.455361] ? do_raw_spin_unlock+0x75/0x190 [ 8692.459447] do_filp_open+0x99/0x140 [ 8692.461331] ? getname_flags+0x6e/0x330 [ 8692.463532] ? __check_object_size+0xff/0x256 [ 8692.473876] ? do_raw_spin_unlock+0x75/0x190 [ 8692.480888] ? _raw_spin_unlock+0x12/0x30 [ 8692.484381] do_sys_openat2+0x2b4/0x410 [ 8692.492527] do_sys_open+0x73/0xa0 [ 8692.495246] __x64_sys_openat+0x24/0x30 [ 8692.498671] do_syscall_64+0xc1/0x440 [ 8692.502829] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8692.511014] RIP: 0033:0x7ff54af45332 [ 8692.519682] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8692.530508] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8692.534448] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 8692.538887] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 8692.553451] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 8692.559796] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 8692.563843] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 8700.058850] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 8700.062107] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 8700.064575] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8700.064718] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8700.064726] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8700.064729] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8700.064853] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8700.064858] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8700.064862] RAX: 00000000000046a3 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 8700.064864] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 8700.064866] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8700.064868] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 8700.064871] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 8700.064874] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 8700.064876] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8700.064878] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 8700.064887] Call Trace: [ 8700.064893] [ 8700.064899] ? show_regs.cold.9+0x22/0x2f [ 8700.064909] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8700.064915] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8700.064919] ? watchdog+0x40/0x40 [ 8700.064924] ? hrtimer_interrupt+0x128/0x2c0 [ 8700.064927] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8700.064936] ? apic_timer_interrupt+0xf/0x20 [ 8700.064941] [ 8700.064944] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8700.064951] do_raw_spin_lock+0x177/0x190 [ 8700.064956] _raw_spin_lock+0x21/0x30 [ 8700.064960] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8700.065256] ? do_raw_spin_unlock+0x75/0x190 [ 8700.065262] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8700.065457] ? do_wait_intr+0xf0/0xf0 [ 8700.065465] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8700.065870] mdc_close+0x53b/0xd70 [mdc] [ 8700.065914] lmv_close+0x18d/0x490 [lmv] [ 8700.065943] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 8700.066211] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8700.072151] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8700.072283] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8700.072291] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8700.072297] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8700.072604] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8700.072616] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8700.072619] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8700.072624] RAX: 00000000000066d3 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 8700.072626] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 8700.072629] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 8700.072631] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 8700.072633] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 8700.072636] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 8700.072639] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8700.072641] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 8700.072650] Call Trace: [ 8700.072656] [ 8700.072662] ? show_regs.cold.9+0x22/0x2f [ 8700.072671] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8700.072678] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8700.072682] ? watchdog+0x40/0x40 [ 8700.072686] ? hrtimer_interrupt+0x128/0x2c0 [ 8700.072690] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8700.072850] ? apic_timer_interrupt+0xf/0x20 [ 8700.072857] [ 8700.072860] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8700.072865] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8700.072870] do_raw_spin_lock+0x177/0x190 [ 8700.072875] _raw_spin_lock+0x21/0x30 [ 8700.072880] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8700.073470] process_one_work+0x2c8/0x700 [ 8700.073478] worker_thread+0x296/0x6e0 [ 8700.073483] ? rescuer_thread+0x570/0x570 [ 8700.073486] kthread+0x1d1/0x200 [ 8700.073491] ? set_kthread_struct+0x70/0x70 [ 8700.073494] ret_from_fork+0x1f/0x30 [ 8700.464992] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8700.467969] ll_md_close+0x274/0x800 [lustre] [ 8700.475235] ll_file_release+0x128/0xc50 [lustre] [ 8700.483284] __fput+0x10d/0x440 [ 8700.484598] ____fput+0x12/0x20 [ 8700.485606] task_work_run+0xc6/0x110 [ 8700.487674] exit_to_usermode_loop+0x1dd/0x1f0 [ 8700.490115] do_syscall_64+0x3d6/0x440 [ 8700.491296] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8700.493673] RIP: 0033:0x7ff54af45cd8 [ 8700.498952] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8700.521905] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8700.535247] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 8700.547364] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 8700.554019] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 8700.560373] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 8700.567553] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8708.056741] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [ 8708.063468] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8708.101214] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8708.106064] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8708.110284] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8708.113118] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8708.121341] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8708.124355] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 8708.127627] RDX: 0000000000000c73 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 8708.138452] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 8708.142144] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 8708.151945] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8708.159113] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 8708.168543] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8708.171489] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 8708.183372] Call Trace: [ 8708.185620] [ 8708.186644] ? show_regs.cold.9+0x22/0x2f [ 8708.190628] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8708.198728] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8708.201096] ? watchdog+0x40/0x40 [ 8708.203182] ? hrtimer_interrupt+0x128/0x2c0 [ 8708.205030] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8708.209907] ? apic_timer_interrupt+0xf/0x20 [ 8708.211930] [ 8708.212775] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8708.215312] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8708.217496] do_raw_spin_lock+0x177/0x190 [ 8708.219291] _raw_spin_lock+0x21/0x30 [ 8708.220713] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8708.222686] obd_mod_free+0x46/0x140 [ptlrpc] [ 8708.225037] mdc_commit_open+0x91/0xb0 [mdc] [ 8708.227061] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8708.231412] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8708.233708] after_reply+0xe2c/0x1780 [ptlrpc] [ 8708.237692] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8708.240895] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8708.243619] ? do_wait_intr+0xf0/0xf0 [ 8708.245112] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8708.248432] kthread+0x1d1/0x200 [ 8708.249577] ? set_kthread_struct+0x70/0x70 [ 8708.251593] ret_from_fork+0x1f/0x30 [ 8711.649601] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6752s! [ 8711.653863] Showing busy workqueues and worker pools: [ 8711.659755] workqueue events: flags=0x0 [ 8711.663601] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8711.667596] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 8711.685095] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8711.694856] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 8711.710316] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8711.713877] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 8711.721772] workqueue events_long: flags=0x0 [ 8711.724826] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8711.730115] pending: lu_objects_flush [obdclass] [ 8711.734381] workqueue events_unbound: flags=0x2 [ 8711.737518] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8711.741728] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8711.753089] workqueue events_power_efficient: flags=0x80 [ 8711.759572] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8711.768994] pending: neigh_periodic_work [ 8711.772623] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8711.775722] pending: do_cache_clean, check_lifetime [ 8711.778335] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8711.783531] pending: neigh_periodic_work [ 8711.787298] workqueue writeback: flags=0x4a [ 8711.788989] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8711.792559] pending: wb_workfn, wb_workfn [ 8711.795294] workqueue xprtiod: flags=0xa [ 8711.797329] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8711.803872] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 8711.810991] workqueue ipv6_addrconf: flags=0x40008 [ 8711.814141] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8711.816490] pending: addrconf_verify_work [ 8711.818462] workqueue knbd0-recv: flags=0x1a [ 8711.820742] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8711.824494] in-flight: 657:recv_work [ 8711.826366] workqueue ptlrpc_pinger: flags=0x2 [ 8711.828905] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8711.832509] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 8711.835220] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6752s workers=4 idle: 20831 12642 28859 [ 8711.838340] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8702s workers=2 idle: 659 [ 8720.060030] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [ 8720.066515] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8720.097543] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8720.105931] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8720.114538] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8720.116828] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8720.128751] RSP: 0000:ffffaf9fc15234f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8720.134208] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 8720.137351] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 8720.140616] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8720.143777] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 8720.146638] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 8720.149403] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 8720.152833] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8720.154832] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 8720.163997] Call Trace: [ 8720.168520] [ 8720.171615] ? show_regs.cold.9+0x22/0x2f [ 8720.172787] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8720.174432] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8720.175633] ? watchdog+0x40/0x40 [ 8720.176512] ? hrtimer_interrupt+0x128/0x2c0 [ 8720.181566] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8720.188582] ? apic_timer_interrupt+0xf/0x20 [ 8720.193287] [ 8720.193818] ? native_safe_halt+0x2e/0x50 [ 8720.194786] kvm_wait+0xcd/0xe0 [ 8720.195564] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8720.204101] do_raw_spin_lock+0x177/0x190 [ 8720.205153] _raw_spin_lock+0x21/0x30 [ 8720.211546] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8720.216007] ? do_raw_spin_unlock+0x75/0x190 [ 8720.221358] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8720.222919] ? do_wait_intr+0xf0/0xf0 [ 8720.223792] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8720.234861] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8720.240948] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8720.244776] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8720.248307] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8720.252250] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8720.253423] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8720.254624] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8720.257571] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8720.259816] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8720.262535] lmv_intent_open+0x338/0x1150 [lmv] [ 8720.264217] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8720.266871] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8720.269538] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8720.271197] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8720.272827] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8720.275433] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8720.277302] ? do_raw_spin_unlock+0x75/0x190 [ 8720.278920] ? _raw_spin_unlock+0x12/0x30 [ 8720.280364] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8720.282410] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8720.285561] ll_atomic_open+0x36a/0xd70 [lustre] [ 8720.287930] lookup_open+0x356/0xb40 [ 8720.289524] do_last+0x313/0xfc0 [ 8720.293062] ? nd_jump_root+0xe5/0x160 [ 8720.294449] ? path_init+0x437/0x520 [ 8720.296228] path_openat+0xf7/0x500 [ 8720.297550] ? do_raw_spin_unlock+0x75/0x190 [ 8720.299586] do_filp_open+0x99/0x140 [ 8720.300986] ? getname_flags+0x6e/0x330 [ 8720.302001] ? __check_object_size+0xff/0x256 [ 8720.303414] ? do_raw_spin_unlock+0x75/0x190 [ 8720.304820] ? _raw_spin_unlock+0x12/0x30 [ 8720.306994] do_sys_openat2+0x2b4/0x410 [ 8720.318192] do_sys_open+0x73/0xa0 [ 8720.321950] __x64_sys_openat+0x24/0x30 [ 8720.326576] do_syscall_64+0xc1/0x440 [ 8720.328502] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8720.333000] RIP: 0033:0x7ff54af45332 [ 8720.335660] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8720.346658] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8720.349798] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 8720.352457] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 8720.354780] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 8720.357383] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 8720.359991] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 8728.058397] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 8728.059067] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 8728.067558] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8728.067695] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8728.067704] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8728.067707] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8728.067722] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8728.067727] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8728.067731] RAX: 0000000000007275 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 8728.067734] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 8728.067737] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8728.067739] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 8728.067741] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 8728.067745] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 8728.067748] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8728.067750] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 8728.067759] Call Trace: [ 8728.067831] [ 8728.067839] ? show_regs.cold.9+0x22/0x2f [ 8728.067850] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8728.067857] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8728.067862] ? watchdog+0x40/0x40 [ 8728.067867] ? hrtimer_interrupt+0x128/0x2c0 [ 8728.067871] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8728.067879] ? apic_timer_interrupt+0xf/0x20 [ 8728.067883] [ 8728.067887] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8728.067894] do_raw_spin_lock+0x177/0x190 [ 8728.067899] _raw_spin_lock+0x21/0x30 [ 8728.067905] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8728.069854] ? do_raw_spin_unlock+0x75/0x190 [ 8728.069864] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8728.077151] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8728.077908] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8728.077917] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8728.078314] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8728.079421] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8728.079434] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8728.079439] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8728.079443] RAX: 0000000000004574 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 8728.079446] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 8728.079448] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 8728.079450] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 8728.079453] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 8728.079456] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 8728.079459] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8728.079462] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 8728.079471] Call Trace: [ 8728.079477] [ 8728.079483] ? show_regs.cold.9+0x22/0x2f [ 8728.079494] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8728.079501] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8728.079505] ? watchdog+0x40/0x40 [ 8728.079509] ? hrtimer_interrupt+0x128/0x2c0 [ 8728.079514] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8728.079523] ? apic_timer_interrupt+0xf/0x20 [ 8728.079528] [ 8728.079531] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8728.079536] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8728.079542] do_raw_spin_lock+0x177/0x190 [ 8728.079547] _raw_spin_lock+0x21/0x30 [ 8728.079553] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8728.080004] process_one_work+0x2c8/0x700 [ 8728.080012] worker_thread+0x296/0x6e0 [ 8728.080017] ? rescuer_thread+0x570/0x570 [ 8728.080022] kthread+0x1d1/0x200 [ 8728.080026] ? set_kthread_struct+0x70/0x70 [ 8728.080030] ret_from_fork+0x1f/0x30 [ 8728.284115] ? do_wait_intr+0xf0/0xf0 [ 8728.285177] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8728.286893] mdc_close+0x53b/0xd70 [mdc] [ 8728.288061] lmv_close+0x18d/0x490 [lmv] [ 8728.289198] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 8728.290876] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8728.292505] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8728.294019] ll_md_close+0x274/0x800 [lustre] [ 8728.295394] ll_file_release+0x128/0xc50 [lustre] [ 8728.296914] __fput+0x10d/0x440 [ 8728.297859] ____fput+0x12/0x20 [ 8728.298718] task_work_run+0xc6/0x110 [ 8728.300135] exit_to_usermode_loop+0x1dd/0x1f0 [ 8728.301741] do_syscall_64+0x3d6/0x440 [ 8728.302667] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8728.304212] RIP: 0033:0x7ff54af45cd8 [ 8728.305223] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8728.313466] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8728.316068] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 8728.318165] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 8728.320524] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 8728.322467] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 8728.324474] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8736.056128] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [ 8736.082820] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8736.294043] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8736.299272] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8736.302782] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8736.304399] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8736.311480] RSP: 0000:ffffaf9fc1b97bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8736.314501] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 8736.317259] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 8736.319940] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 8736.322334] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 8736.324912] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8736.327158] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 8736.330158] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8736.332311] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 8736.335039] Call Trace: [ 8736.336223] [ 8736.337018] ? show_regs.cold.9+0x22/0x2f [ 8736.338750] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8736.340968] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8736.342740] ? watchdog+0x40/0x40 [ 8736.344040] ? hrtimer_interrupt+0x128/0x2c0 [ 8736.345785] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8736.347627] ? apic_timer_interrupt+0xf/0x20 [ 8736.349250] [ 8736.350093] ? native_safe_halt+0x2e/0x50 [ 8736.351608] kvm_wait+0xcd/0xe0 [ 8736.352779] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8736.354650] do_raw_spin_lock+0x177/0x190 [ 8736.356063] _raw_spin_lock+0x21/0x30 [ 8736.357445] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8736.359497] obd_mod_free+0x46/0x140 [ptlrpc] [ 8736.361439] mdc_commit_open+0x91/0xb0 [mdc] [ 8736.363112] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8736.365085] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8736.367112] after_reply+0xe2c/0x1780 [ptlrpc] [ 8736.368978] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8736.370983] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8736.372755] ? do_wait_intr+0xf0/0xf0 [ 8736.374062] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8736.376192] kthread+0x1d1/0x200 [ 8736.377385] ? set_kthread_struct+0x70/0x70 [ 8736.379016] ret_from_fork+0x1f/0x30 [ 8741.863440] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6782s! [ 8741.887316] Showing busy workqueues and worker pools: [ 8741.896663] workqueue events: flags=0x0 [ 8741.900856] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8741.906736] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 8741.925165] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8741.932109] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 8741.943328] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8741.958658] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 8741.970693] workqueue events_long: flags=0x0 [ 8741.972197] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8741.992316] pending: lu_objects_flush [obdclass] [ 8742.003604] workqueue events_unbound: flags=0x2 [ 8742.013609] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8742.026198] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8742.049177] workqueue events_power_efficient: flags=0x80 [ 8742.056772] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8742.064345] pending: neigh_periodic_work [ 8742.070100] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8742.074399] pending: do_cache_clean, check_lifetime [ 8742.078061] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8742.085442] pending: neigh_periodic_work [ 8742.092621] workqueue writeback: flags=0x4a [ 8742.095308] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8742.099207] pending: wb_workfn, wb_workfn [ 8742.101770] workqueue xprtiod: flags=0xa [ 8742.104188] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8742.110151] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 8742.123339] workqueue ipv6_addrconf: flags=0x40008 [ 8742.129821] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8742.144263] pending: addrconf_verify_work [ 8742.158405] workqueue knbd0-recv: flags=0x1a [ 8742.162736] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8742.169967] in-flight: 657:recv_work [ 8742.177599] workqueue ptlrpc_pinger: flags=0x2 [ 8742.182366] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8742.187111] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 8742.191449] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6783s workers=4 idle: 20831 12642 28859 [ 8742.205764] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8732s workers=2 idle: 659 [ 8748.064562] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [ 8748.067814] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8748.104388] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8748.118886] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8748.130161] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8748.137947] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8748.152027] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8748.155564] RAX: 0000000000007c84 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 8748.162642] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 8748.165520] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8748.172953] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 8748.182314] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 8748.186448] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 8748.190549] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8748.195134] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 8748.202293] Call Trace: [ 8748.203461] [ 8748.204491] ? show_regs.cold.9+0x22/0x2f [ 8748.207934] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8748.210539] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8748.213701] ? watchdog+0x40/0x40 [ 8748.215777] ? hrtimer_interrupt+0x128/0x2c0 [ 8748.218055] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8748.220049] ? apic_timer_interrupt+0xf/0x20 [ 8748.222371] [ 8748.223997] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8748.226383] do_raw_spin_lock+0x177/0x190 [ 8748.228342] _raw_spin_lock+0x21/0x30 [ 8748.229920] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8748.232478] ? do_raw_spin_unlock+0x75/0x190 [ 8748.233864] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8748.236353] ? do_wait_intr+0xf0/0xf0 [ 8748.238345] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8748.242120] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8748.247095] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8748.250014] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8748.252502] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8748.254873] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8748.256645] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8748.258589] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8748.260680] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8748.262928] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8748.265342] lmv_intent_open+0x338/0x1150 [lmv] [ 8748.267350] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8748.270548] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8748.275842] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8748.277811] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8748.279685] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8748.281884] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8748.283882] ? do_raw_spin_unlock+0x75/0x190 [ 8748.285661] ? _raw_spin_unlock+0x12/0x30 [ 8748.287818] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8748.289913] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8748.292270] ll_atomic_open+0x36a/0xd70 [lustre] [ 8748.294251] lookup_open+0x356/0xb40 [ 8748.295959] do_last+0x313/0xfc0 [ 8748.297242] ? nd_jump_root+0xe5/0x160 [ 8748.298969] ? path_init+0x437/0x520 [ 8748.300308] path_openat+0xf7/0x500 [ 8748.301784] ? do_raw_spin_unlock+0x75/0x190 [ 8748.302943] do_filp_open+0x99/0x140 [ 8748.304050] ? getname_flags+0x6e/0x330 [ 8748.305012] ? __check_object_size+0xff/0x256 [ 8748.306721] ? do_raw_spin_unlock+0x75/0x190 [ 8748.308665] ? _raw_spin_unlock+0x12/0x30 [ 8748.310619] do_sys_openat2+0x2b4/0x410 [ 8748.311913] do_sys_open+0x73/0xa0 [ 8748.313222] __x64_sys_openat+0x24/0x30 [ 8748.314940] do_syscall_64+0xc1/0x440 [ 8748.316599] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8748.319066] RIP: 0033:0x7ff54af45332 [ 8748.320548] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8748.326925] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8748.329026] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 8748.331605] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 8748.334006] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 8748.336896] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 8748.339860] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 8756.058011] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 8756.061841] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8756.062975] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 8756.106875] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8756.106901] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8756.106905] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8756.106919] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8756.106923] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8756.106928] RAX: 000000000000020b RBX: ffff96848811cae0 RCX: 0000000000000001 [ 8756.106931] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 8756.106933] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8756.106936] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 8756.106939] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 8756.106942] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 8756.106944] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8756.106947] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 8756.106954] Call Trace: [ 8756.106960] [ 8756.106965] ? show_regs.cold.9+0x22/0x2f [ 8756.106976] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8756.106982] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8756.106987] ? watchdog+0x40/0x40 [ 8756.106992] ? hrtimer_interrupt+0x128/0x2c0 [ 8756.106995] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8756.107005] ? apic_timer_interrupt+0xf/0x20 [ 8756.107009] [ 8756.107012] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8756.107020] do_raw_spin_lock+0x177/0x190 [ 8756.107025] _raw_spin_lock+0x21/0x30 [ 8756.107030] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8756.116694] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8756.116781] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8756.116790] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8756.116799] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8756.119453] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8756.119471] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8756.119475] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8756.119479] RAX: 0000000000002b10 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 8756.119481] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 8756.119483] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 8756.119485] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 8756.119488] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 8756.119501] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 8756.119504] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8756.119506] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 8756.119514] Call Trace: [ 8756.119519] [ 8756.119525] ? show_regs.cold.9+0x22/0x2f [ 8756.119536] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8756.119542] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8756.119546] ? watchdog+0x40/0x40 [ 8756.119551] ? hrtimer_interrupt+0x128/0x2c0 [ 8756.119554] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8756.119563] ? apic_timer_interrupt+0xf/0x20 [ 8756.119568] [ 8756.119571] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8756.119575] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8756.119581] do_raw_spin_lock+0x177/0x190 [ 8756.119586] _raw_spin_lock+0x21/0x30 [ 8756.119590] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8756.131521] ? do_raw_spin_unlock+0x75/0x190 [ 8756.131544] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8756.131831] ? do_wait_intr+0xf0/0xf0 [ 8756.131839] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8756.135793] mdc_close+0x53b/0xd70 [mdc] [ 8756.135847] lmv_close+0x18d/0x490 [lmv] [ 8756.135884] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 8756.138619] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8756.138739] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8756.138819] ll_md_close+0x274/0x800 [lustre] [ 8756.138913] ll_file_release+0x128/0xc50 [lustre] [ 8756.139001] __fput+0x10d/0x440 [ 8756.139012] ____fput+0x12/0x20 [ 8756.139018] task_work_run+0xc6/0x110 [ 8756.139029] exit_to_usermode_loop+0x1dd/0x1f0 [ 8756.139038] do_syscall_64+0x3d6/0x440 [ 8756.139043] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8756.139054] RIP: 0033:0x7ff54af45cd8 [ 8756.139062] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8756.139066] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8756.139073] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 8756.139075] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 8756.139077] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 8756.139080] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 8756.139082] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8756.529420] process_one_work+0x2c8/0x700 [ 8756.531070] worker_thread+0x296/0x6e0 [ 8756.534509] ? rescuer_thread+0x570/0x570 [ 8756.537431] kthread+0x1d1/0x200 [ 8756.538556] ? set_kthread_struct+0x70/0x70 [ 8756.544659] ret_from_fork+0x1f/0x30 [ 8768.055023] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 8768.060364] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8768.087501] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8768.093668] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8768.098654] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8768.101532] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8768.117764] RSP: 0000:ffffaf9fc1b97bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8768.125952] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 8768.129731] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 8768.133501] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 8768.136957] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 8768.140657] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8768.143933] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 8768.147184] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8768.149551] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 8768.152209] Call Trace: [ 8768.153185] [ 8768.153902] ? show_regs.cold.9+0x22/0x2f [ 8768.155115] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8768.156589] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8768.158148] ? watchdog+0x40/0x40 [ 8768.159405] ? hrtimer_interrupt+0x128/0x2c0 [ 8768.160989] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8768.162844] ? apic_timer_interrupt+0xf/0x20 [ 8768.164800] [ 8768.165643] ? native_safe_halt+0x2e/0x50 [ 8768.167057] kvm_wait+0xcd/0xe0 [ 8768.168467] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8768.170450] do_raw_spin_lock+0x177/0x190 [ 8768.171915] _raw_spin_lock+0x21/0x30 [ 8768.173337] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8768.175603] obd_mod_free+0x46/0x140 [ptlrpc] [ 8768.177661] mdc_commit_open+0x91/0xb0 [mdc] [ 8768.183649] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8768.185970] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8768.188498] after_reply+0xe2c/0x1780 [ptlrpc] [ 8768.190584] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8768.192966] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8768.194925] ? do_wait_intr+0xf0/0xf0 [ 8768.196478] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8768.198795] kthread+0x1d1/0x200 [ 8768.200085] ? set_kthread_struct+0x70/0x70 [ 8768.201855] ret_from_fork+0x1f/0x30 [ 8772.575665] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6813s! [ 8772.592298] Showing busy workqueues and worker pools: [ 8772.601797] workqueue events: flags=0x0 [ 8772.610920] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8772.621463] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 8772.630488] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8772.638314] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 8772.651118] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8772.658321] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 8772.665455] workqueue events_long: flags=0x0 [ 8772.667788] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8772.677836] pending: lu_objects_flush [obdclass] [ 8772.687789] workqueue events_unbound: flags=0x2 [ 8772.692821] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8772.699260] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8772.714754] workqueue events_power_efficient: flags=0x80 [ 8772.721744] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8772.726626] pending: neigh_periodic_work [ 8772.731739] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8772.738281] pending: do_cache_clean, check_lifetime [ 8772.744088] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8772.752503] pending: neigh_periodic_work [ 8772.759701] workqueue writeback: flags=0x4a [ 8772.765001] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8772.770553] pending: wb_workfn, wb_workfn [ 8772.774593] workqueue xprtiod: flags=0xa [ 8772.776997] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8772.791599] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 8772.802253] workqueue ipv6_addrconf: flags=0x40008 [ 8772.808494] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8772.817592] pending: addrconf_verify_work [ 8772.824499] workqueue knbd0-recv: flags=0x1a [ 8772.828656] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8772.833891] in-flight: 657:recv_work [ 8772.836617] workqueue ptlrpc_pinger: flags=0x2 [ 8772.840518] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8772.846293] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 8772.848620] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6813s workers=4 idle: 20831 12642 28859 [ 8772.858441] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8763s workers=2 idle: 659 [ 8776.060946] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [ 8776.067897] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8776.100056] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8776.109588] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8776.117919] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8776.124081] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8776.145017] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8776.152973] RAX: 0000000000003e86 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 8776.158710] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 8776.166845] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8776.170721] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 8776.174554] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 8776.177119] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 8776.180839] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8776.183529] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 8776.186704] Call Trace: [ 8776.187612] [ 8776.188469] ? show_regs.cold.9+0x22/0x2f [ 8776.189868] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8776.191640] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8776.193062] ? watchdog+0x40/0x40 [ 8776.194474] ? hrtimer_interrupt+0x128/0x2c0 [ 8776.196043] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8776.197886] ? apic_timer_interrupt+0xf/0x20 [ 8776.202639] [ 8776.203718] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8776.205991] do_raw_spin_lock+0x177/0x190 [ 8776.207968] _raw_spin_lock+0x21/0x30 [ 8776.209470] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8776.211690] ? do_raw_spin_unlock+0x75/0x190 [ 8776.213377] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8776.215571] ? do_wait_intr+0xf0/0xf0 [ 8776.216992] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8776.219366] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8776.221780] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8776.224509] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8776.226757] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8776.228913] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8776.230784] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8776.232584] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8776.234779] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8776.237318] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8776.239605] lmv_intent_open+0x338/0x1150 [lmv] [ 8776.241462] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8776.243641] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8776.246572] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8776.248151] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8776.252006] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8776.256972] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8776.259091] ? do_raw_spin_unlock+0x75/0x190 [ 8776.260989] ? _raw_spin_unlock+0x12/0x30 [ 8776.262689] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8776.265363] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8776.267794] ll_atomic_open+0x36a/0xd70 [lustre] [ 8776.270247] lookup_open+0x356/0xb40 [ 8776.272967] do_last+0x313/0xfc0 [ 8776.276197] ? nd_jump_root+0xe5/0x160 [ 8776.277555] ? path_init+0x437/0x520 [ 8776.281984] path_openat+0xf7/0x500 [ 8776.285520] ? do_raw_spin_unlock+0x75/0x190 [ 8776.287656] do_filp_open+0x99/0x140 [ 8776.289082] ? getname_flags+0x6e/0x330 [ 8776.290776] ? __check_object_size+0xff/0x256 [ 8776.292483] ? do_raw_spin_unlock+0x75/0x190 [ 8776.294274] ? _raw_spin_unlock+0x12/0x30 [ 8776.295758] do_sys_openat2+0x2b4/0x410 [ 8776.297504] do_sys_open+0x73/0xa0 [ 8776.299075] __x64_sys_openat+0x24/0x30 [ 8776.300868] do_syscall_64+0xc1/0x440 [ 8776.302755] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8776.305036] RIP: 0033:0x7ff54af45332 [ 8776.306828] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8776.315653] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8776.318787] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 8776.321741] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 8776.324843] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 8776.329095] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 8776.335766] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 8784.060465] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 8784.069476] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8784.070478] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 8784.112727] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8784.112742] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8784.112745] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8784.112759] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8784.112764] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8784.112769] RAX: 0000000000007209 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 8784.112772] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 8784.112774] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8784.112776] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 8784.112779] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 8784.112782] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 8784.112785] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8784.112787] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 8784.112795] Call Trace: [ 8784.112801] [ 8784.112807] ? show_regs.cold.9+0x22/0x2f [ 8784.112816] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8784.112824] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8784.112828] ? watchdog+0x40/0x40 [ 8784.112832] ? hrtimer_interrupt+0x128/0x2c0 [ 8784.112836] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8784.112845] ? apic_timer_interrupt+0xf/0x20 [ 8784.112850] [ 8784.112852] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8784.112860] do_raw_spin_lock+0x177/0x190 [ 8784.112865] _raw_spin_lock+0x21/0x30 [ 8784.112870] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8784.113137] ? do_raw_spin_unlock+0x75/0x190 [ 8784.113143] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8784.115152] ? do_wait_intr+0xf0/0xf0 [ 8784.115162] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8784.115369] mdc_close+0x53b/0xd70 [mdc] [ 8784.115409] lmv_close+0x18d/0x490 [lmv] [ 8784.115437] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 8784.115784] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8784.115900] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8784.130687] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8784.130795] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8784.130803] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8784.130809] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8784.131757] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8784.131769] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8784.131773] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8784.131778] RAX: 0000000000006890 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 8784.131780] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 8784.131782] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 8784.131784] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 8784.131786] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 8784.131789] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 8784.131792] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8784.131794] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 8784.131803] Call Trace: [ 8784.131808] [ 8784.131814] ? show_regs.cold.9+0x22/0x2f [ 8784.131823] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8784.131829] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8784.132474] ? watchdog+0x40/0x40 [ 8784.132482] ? hrtimer_interrupt+0x128/0x2c0 [ 8784.132485] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8784.132492] ? apic_timer_interrupt+0xf/0x20 [ 8784.132497] [ 8784.132500] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8784.132505] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8784.132511] do_raw_spin_lock+0x177/0x190 [ 8784.132515] _raw_spin_lock+0x21/0x30 [ 8784.132520] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8784.132841] process_one_work+0x2c8/0x700 [ 8784.132848] worker_thread+0x296/0x6e0 [ 8784.132853] ? rescuer_thread+0x570/0x570 [ 8784.132857] kthread+0x1d1/0x200 [ 8784.132861] ? set_kthread_struct+0x70/0x70 [ 8784.132864] ret_from_fork+0x1f/0x30 [ 8784.719873] ll_md_close+0x274/0x800 [lustre] [ 8784.723592] ll_file_release+0x128/0xc50 [lustre] [ 8784.731015] __fput+0x10d/0x440 [ 8784.738502] ____fput+0x12/0x20 [ 8784.739753] task_work_run+0xc6/0x110 [ 8784.746547] exit_to_usermode_loop+0x1dd/0x1f0 [ 8784.749756] do_syscall_64+0x3d6/0x440 [ 8784.751825] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8784.754466] RIP: 0033:0x7ff54af45cd8 [ 8784.765348] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8784.783294] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8784.789722] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 8784.805758] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 8784.812619] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 8784.816736] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 8784.820479] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8800.055016] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 8800.062444] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8800.100180] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8800.111996] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8800.119522] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8800.125643] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8800.140731] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8800.145559] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 8800.152124] RDX: 000000000000488c RSI: 0000000000000003 RDI: ffff96848811cae0 [ 8800.154734] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 8800.161810] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 8800.167659] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8800.173233] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 8800.179193] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8800.182235] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 8800.203735] Call Trace: [ 8800.207156] [ 8800.208354] ? show_regs.cold.9+0x22/0x2f [ 8800.211252] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8800.216543] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8800.221586] ? watchdog+0x40/0x40 [ 8800.225434] ? hrtimer_interrupt+0x128/0x2c0 [ 8800.229785] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8800.235429] ? apic_timer_interrupt+0xf/0x20 [ 8800.237711] [ 8800.238845] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8800.242661] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8800.245685] do_raw_spin_lock+0x177/0x190 [ 8800.248149] _raw_spin_lock+0x21/0x30 [ 8800.251369] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8800.255517] obd_mod_free+0x46/0x140 [ptlrpc] [ 8800.258950] mdc_commit_open+0x91/0xb0 [mdc] [ 8800.261640] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8800.266284] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8800.272237] after_reply+0xe2c/0x1780 [ptlrpc] [ 8800.275810] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8800.278770] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8800.281824] ? do_wait_intr+0xf0/0xf0 [ 8800.284730] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8800.287579] kthread+0x1d1/0x200 [ 8800.290697] ? set_kthread_struct+0x70/0x70 [ 8800.292766] ret_from_fork+0x1f/0x30 [ 8803.295470] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6844s! [ 8803.315303] Showing busy workqueues and worker pools: [ 8803.319555] workqueue events: flags=0x0 [ 8803.323743] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8803.326836] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 8803.335804] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8803.341857] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 8803.351248] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8803.356705] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 8803.366434] workqueue events_long: flags=0x0 [ 8803.367838] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8803.376344] pending: lu_objects_flush [obdclass] [ 8803.380780] workqueue events_unbound: flags=0x2 [ 8803.385783] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8803.396420] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8803.416266] workqueue events_power_efficient: flags=0x80 [ 8803.422750] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8803.428347] pending: neigh_periodic_work [ 8803.430669] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8803.434224] pending: do_cache_clean, check_lifetime [ 8803.437024] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8803.440286] pending: neigh_periodic_work [ 8803.443134] workqueue writeback: flags=0x4a [ 8803.445128] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8803.453236] pending: wb_workfn, wb_workfn [ 8803.458373] workqueue xprtiod: flags=0xa [ 8803.462581] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8803.469512] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 8803.473237] workqueue ipv6_addrconf: flags=0x40008 [ 8803.476615] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8803.480603] pending: addrconf_verify_work [ 8803.483139] workqueue knbd0-recv: flags=0x1a [ 8803.485484] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8803.488729] in-flight: 657:recv_work [ 8803.491163] workqueue ptlrpc_pinger: flags=0x2 [ 8803.493418] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8803.496442] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 8803.499119] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6844s workers=4 idle: 20831 12642 28859 [ 8803.503370] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8793s workers=2 idle: 659 [ 8804.065322] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 8804.075645] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8804.124084] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8804.130627] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8804.135830] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8804.144681] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8804.162863] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8804.167754] RAX: 0000000000007524 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 8804.179571] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 8804.190299] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8804.195497] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 8804.199575] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 8804.215531] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 8804.223820] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8804.228627] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 8804.232678] Call Trace: [ 8804.234205] [ 8804.235121] ? show_regs.cold.9+0x22/0x2f [ 8804.238465] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8804.241259] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8804.243360] ? watchdog+0x40/0x40 [ 8804.244736] ? hrtimer_interrupt+0x128/0x2c0 [ 8804.247874] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8804.250679] ? apic_timer_interrupt+0xf/0x20 [ 8804.253503] [ 8804.255627] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8804.261689] do_raw_spin_lock+0x177/0x190 [ 8804.268712] _raw_spin_lock+0x21/0x30 [ 8804.273963] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8804.282848] ? do_raw_spin_unlock+0x75/0x190 [ 8804.296271] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8804.298269] ? do_wait_intr+0xf0/0xf0 [ 8804.304605] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8804.310386] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8804.316526] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8804.322747] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8804.326861] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8804.330902] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8804.334425] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8804.338520] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8804.341630] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8804.347763] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8804.349675] lmv_intent_open+0x338/0x1150 [lmv] [ 8804.350633] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8804.360484] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8804.363770] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8804.365725] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8804.368597] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8804.370655] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8804.372803] ? do_raw_spin_unlock+0x75/0x190 [ 8804.374769] ? _raw_spin_unlock+0x12/0x30 [ 8804.376724] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8804.384550] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8804.386975] ll_atomic_open+0x36a/0xd70 [lustre] [ 8804.391038] lookup_open+0x356/0xb40 [ 8804.392160] do_last+0x313/0xfc0 [ 8804.397697] ? nd_jump_root+0xe5/0x160 [ 8804.401598] ? path_init+0x437/0x520 [ 8804.405652] path_openat+0xf7/0x500 [ 8804.409520] ? do_raw_spin_unlock+0x75/0x190 [ 8804.419682] do_filp_open+0x99/0x140 [ 8804.422382] ? getname_flags+0x6e/0x330 [ 8804.424396] ? __check_object_size+0xff/0x256 [ 8804.427236] ? do_raw_spin_unlock+0x75/0x190 [ 8804.432600] ? _raw_spin_unlock+0x12/0x30 [ 8804.437302] do_sys_openat2+0x2b4/0x410 [ 8804.438739] do_sys_open+0x73/0xa0 [ 8804.442524] __x64_sys_openat+0x24/0x30 [ 8804.446445] do_syscall_64+0xc1/0x440 [ 8804.450379] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8804.455910] RIP: 0033:0x7ff54af45332 [ 8804.460479] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8804.475304] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8804.481453] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 8804.485378] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 8804.488585] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 8804.494560] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 8804.500494] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 8812.061689] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 8812.063209] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 8812.064411] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) [ 8812.073036] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8812.073493] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8812.073499] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8812.073502] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8812.073516] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8812.073520] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8812.073524] RAX: 0000000000007e61 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 8812.073526] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 8812.073528] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8812.073531] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 8812.073533] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 8812.073535] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 8812.073538] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8812.073540] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 8812.073547] Call Trace: [ 8812.073553] [ 8812.073558] ? show_regs.cold.9+0x22/0x2f [ 8812.073567] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8812.073573] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8812.073577] ? watchdog+0x40/0x40 [ 8812.073581] ? hrtimer_interrupt+0x128/0x2c0 [ 8812.073585] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8812.073593] ? apic_timer_interrupt+0xf/0x20 [ 8812.073597] [ 8812.073600] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8812.073606] do_raw_spin_lock+0x177/0x190 [ 8812.073610] _raw_spin_lock+0x21/0x30 [ 8812.073615] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8812.074259] ? do_raw_spin_unlock+0x75/0x190 [ 8812.074266] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8812.074494] ? do_wait_intr+0xf0/0xf0 [ 8812.074503] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8812.074712] mdc_close+0x53b/0xd70 [mdc] [ 8812.074759] lmv_close+0x18d/0x490 [lmv] [ 8812.074787] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 8812.075523] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8812.075639] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8812.075724] ll_md_close+0x274/0x800 [lustre] [ 8812.075809] ll_file_release+0x128/0xc50 [lustre] [ 8812.084530] obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8812.084594] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8812.084601] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8812.084608] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8812.119002] __fput+0x10d/0x440 [ 8812.119024] ____fput+0x12/0x20 [ 8812.119028] task_work_run+0xc6/0x110 [ 8812.119036] exit_to_usermode_loop+0x1dd/0x1f0 [ 8812.119043] do_syscall_64+0x3d6/0x440 [ 8812.119047] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8812.119102] RIP: 0033:0x7ff54af45cd8 [ 8812.119110] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8812.119115] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8812.119120] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 8812.119122] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 8812.119125] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 8812.119127] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 8812.119129] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8812.389038] [ 8812.389620] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8812.391460] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8812.401679] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8812.405449] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 8812.412744] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 8812.418031] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 8812.421405] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 8812.425491] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 8812.431349] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 8812.435024] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8812.436936] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 8812.439502] Call Trace: [ 8812.440443] [ 8812.446926] ? show_regs.cold.9+0x22/0x2f [ 8812.448797] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8812.451409] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8812.458315] ? watchdog+0x40/0x40 [ 8812.460518] ? hrtimer_interrupt+0x128/0x2c0 [ 8812.464203] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8812.466506] ? apic_timer_interrupt+0xf/0x20 [ 8812.470702] [ 8812.471952] ? native_safe_halt+0x2e/0x50 [ 8812.473681] kvm_wait+0xcd/0xe0 [ 8812.475139] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8812.477327] do_raw_spin_lock+0x177/0x190 [ 8812.479759] _raw_spin_lock+0x21/0x30 [ 8812.481697] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8812.486355] process_one_work+0x2c8/0x700 [ 8812.487299] worker_thread+0x296/0x6e0 [ 8812.488257] ? rescuer_thread+0x570/0x570 [ 8812.489126] kthread+0x1d1/0x200 [ 8812.489788] ? set_kthread_struct+0x70/0x70 [ 8812.492554] ret_from_fork+0x1f/0x30 [ 8828.055060] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 8828.058227] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8828.073775] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8828.077628] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8828.081434] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8828.083236] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8828.089422] RSP: 0000:ffffaf9fc1b97bb8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 8828.091536] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 8828.094516] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 8828.097425] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 8828.099628] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 8828.102230] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8828.104332] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 8828.106994] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8828.108570] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 8828.111075] Call Trace: [ 8828.111704] [ 8828.112215] ? show_regs.cold.9+0x22/0x2f [ 8828.113140] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8828.114273] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8828.116135] ? watchdog+0x40/0x40 [ 8828.117163] ? hrtimer_interrupt+0x128/0x2c0 [ 8828.118298] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8828.120396] ? apic_timer_interrupt+0xf/0x20 [ 8828.121816] [ 8828.122377] ? native_safe_halt+0x2e/0x50 [ 8828.123934] kvm_wait+0xcd/0xe0 [ 8828.124902] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8828.126553] do_raw_spin_lock+0x177/0x190 [ 8828.128085] _raw_spin_lock+0x21/0x30 [ 8828.129452] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8828.130947] obd_mod_free+0x46/0x140 [ptlrpc] [ 8828.132915] mdc_commit_open+0x91/0xb0 [mdc] [ 8828.134308] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8828.136524] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8828.138441] after_reply+0xe2c/0x1780 [ptlrpc] [ 8828.140286] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8828.142106] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8828.143466] ? do_wait_intr+0xf0/0xf0 [ 8828.144398] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8828.146263] kthread+0x1d1/0x200 [ 8828.147505] ? set_kthread_struct+0x70/0x70 [ 8828.148906] ret_from_fork+0x1f/0x30 [ 8832.060020] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 8832.063993] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8832.090517] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8832.095617] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8832.099074] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8832.100166] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8832.110773] RSP: 0000:ffffaf9fc15234f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8832.113806] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 8832.116649] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 8832.119469] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8832.127315] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 8832.130189] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 8832.132452] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 8832.137237] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8832.139568] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 8832.146843] Call Trace: [ 8832.147545] [ 8832.149493] ? show_regs.cold.9+0x22/0x2f [ 8832.151426] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8832.153509] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8832.155690] ? watchdog+0x40/0x40 [ 8832.157418] ? hrtimer_interrupt+0x128/0x2c0 [ 8832.160387] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8832.162498] ? apic_timer_interrupt+0xf/0x20 [ 8832.168561] [ 8832.170374] ? native_safe_halt+0x2e/0x50 [ 8832.172783] kvm_wait+0xcd/0xe0 [ 8832.173685] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8832.175465] do_raw_spin_lock+0x177/0x190 [ 8832.176942] _raw_spin_lock+0x21/0x30 [ 8832.178022] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8832.181003] ? do_raw_spin_unlock+0x75/0x190 [ 8832.182786] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8832.187483] ? do_wait_intr+0xf0/0xf0 [ 8832.190511] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8832.195328] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8832.199247] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8832.200999] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8832.203425] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8832.205639] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8832.210365] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8832.214734] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8832.218484] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8832.223718] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8832.225878] lmv_intent_open+0x338/0x1150 [lmv] [ 8832.227587] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8832.229937] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8832.232482] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8832.234143] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8832.235950] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8832.238800] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8832.240532] ? do_raw_spin_unlock+0x75/0x190 [ 8832.241644] ? _raw_spin_unlock+0x12/0x30 [ 8832.243711] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8832.245962] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8832.248540] ll_atomic_open+0x36a/0xd70 [lustre] [ 8832.251005] lookup_open+0x356/0xb40 [ 8832.253642] do_last+0x313/0xfc0 [ 8832.254981] ? nd_jump_root+0xe5/0x160 [ 8832.258808] ? path_init+0x437/0x520 [ 8832.260602] path_openat+0xf7/0x500 [ 8832.261903] ? do_raw_spin_unlock+0x75/0x190 [ 8832.266500] do_filp_open+0x99/0x140 [ 8832.268835] ? getname_flags+0x6e/0x330 [ 8832.272022] ? __check_object_size+0xff/0x256 [ 8832.275469] ? do_raw_spin_unlock+0x75/0x190 [ 8832.277008] ? _raw_spin_unlock+0x12/0x30 [ 8832.278481] do_sys_openat2+0x2b4/0x410 [ 8832.279899] do_sys_open+0x73/0xa0 [ 8832.281518] __x64_sys_openat+0x24/0x30 [ 8832.282924] do_syscall_64+0xc1/0x440 [ 8832.284446] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8832.286324] RIP: 0033:0x7ff54af45332 [ 8832.287490] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8832.293473] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8832.295828] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 8832.298289] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 8832.300686] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 8832.303725] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 8832.306150] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 8834.016435] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6875s! [ 8834.021931] Showing busy workqueues and worker pools: [ 8834.025509] workqueue events: flags=0x0 [ 8834.031687] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8834.036847] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 8834.044239] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8834.048016] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 8834.055102] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8834.059333] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 8834.064539] workqueue events_long: flags=0x0 [ 8834.065989] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8834.068520] pending: lu_objects_flush [obdclass] [ 8834.070323] workqueue events_unbound: flags=0x2 [ 8834.071727] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8834.073646] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8834.077042] workqueue events_power_efficient: flags=0x80 [ 8834.078927] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8834.081093] pending: neigh_periodic_work [ 8834.082442] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8834.084234] pending: do_cache_clean, check_lifetime [ 8834.085728] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8834.087432] pending: neigh_periodic_work [ 8834.088697] workqueue writeback: flags=0x4a [ 8834.090114] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8834.091761] pending: wb_workfn, wb_workfn [ 8834.092984] workqueue xprtiod: flags=0xa [ 8834.094377] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8834.096152] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 8834.098276] workqueue ipv6_addrconf: flags=0x40008 [ 8834.099931] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8834.101505] pending: addrconf_verify_work [ 8834.103093] workqueue knbd0-recv: flags=0x1a [ 8834.104508] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8834.107745] in-flight: 657:recv_work [ 8834.109669] workqueue ptlrpc_pinger: flags=0x2 [ 8834.111818] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8834.114720] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 8834.117785] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6875s workers=4 idle: 20831 12642 28859 [ 8834.121974] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8824s workers=2 idle: 659 [ 8840.058108] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 8840.059055] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 8840.062816] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8840.062910] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8840.062918] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8840.062921] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8840.062935] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8840.062940] RSP: 0018:ffffaf9fc1543ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8840.062945] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 8840.062947] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 8840.062949] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8840.062951] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 8840.062954] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 8840.062957] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 8840.062960] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8840.062963] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 8840.062971] Call Trace: [ 8840.062977] [ 8840.062983] ? show_regs.cold.9+0x22/0x2f [ 8840.062993] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8840.063000] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8840.063004] ? watchdog+0x40/0x40 [ 8840.063008] ? hrtimer_interrupt+0x128/0x2c0 [ 8840.063012] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8840.063019] ? apic_timer_interrupt+0xf/0x20 [ 8840.063023] [ 8840.063026] ? native_safe_halt+0x2e/0x50 [ 8840.063030] kvm_wait+0xcd/0xe0 [ 8840.063037] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8840.063045] do_raw_spin_lock+0x177/0x190 [ 8840.063049] _raw_spin_lock+0x21/0x30 [ 8840.063053] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8840.063386] ? do_raw_spin_unlock+0x75/0x190 [ 8840.063392] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8840.063586] ? do_wait_intr+0xf0/0xf0 [ 8840.063594] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8840.063802] mdc_close+0x53b/0xd70 [mdc] [ 8840.063841] lmv_close+0x18d/0x490 [lmv] [ 8840.063872] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 8840.064040] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8840.064152] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8840.064232] ll_md_close+0x274/0x800 [lustre] [ 8840.064314] ll_file_release+0x128/0xc50 [lustre] [ 8840.064401] __fput+0x10d/0x440 [ 8840.064410] ____fput+0x12/0x20 [ 8840.064415] task_work_run+0xc6/0x110 [ 8840.064423] exit_to_usermode_loop+0x1dd/0x1f0 [ 8840.064431] do_syscall_64+0x3d6/0x440 [ 8840.064436] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8840.064442] RIP: 0033:0x7ff54af45cd8 [ 8840.064450] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8840.064455] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8840.064459] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 8840.064461] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 8840.064464] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 8840.064466] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 8840.064469] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8840.341268] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8840.388094] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8840.395895] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8840.400817] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8840.405450] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8840.408356] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8840.418728] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8840.423920] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 8840.433433] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 8840.438609] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 8840.441791] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 8840.444805] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 8840.447095] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 8840.449821] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8840.451922] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 8840.457336] Call Trace: [ 8840.459089] [ 8840.459626] ? show_regs.cold.9+0x22/0x2f [ 8840.460664] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8840.462664] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8840.463885] ? watchdog+0x40/0x40 [ 8840.465622] ? hrtimer_interrupt+0x128/0x2c0 [ 8840.466657] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8840.470192] ? apic_timer_interrupt+0xf/0x20 [ 8840.473279] [ 8840.474466] ? native_safe_halt+0x2e/0x50 [ 8840.476366] kvm_wait+0xcd/0xe0 [ 8840.477419] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8840.480937] do_raw_spin_lock+0x177/0x190 [ 8840.483303] _raw_spin_lock+0x21/0x30 [ 8840.485722] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8840.489351] process_one_work+0x2c8/0x700 [ 8840.491924] worker_thread+0x296/0x6e0 [ 8840.494954] ? rescuer_thread+0x570/0x570 [ 8840.497771] kthread+0x1d1/0x200 [ 8840.499698] ? set_kthread_struct+0x70/0x70 [ 8840.502364] ret_from_fork+0x1f/0x30 [ 8854.040984] rcu: INFO: rcu_sched self-detected stall on CPU [ 8854.047046] rcu: 0-...!: (3583900 ticks this GP) idle=98a/1/0x4000000000000002 softirq=125154/125154 fqs=959 [ 8854.063174] (t=6900200 jiffies g=253585 q=5094) [ 8854.071190] rcu: rcu_sched kthread starved for 6895160 jiffies! g253585 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=3 [ 8854.080036] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 8854.087587] rcu: RCU grace-period kthread stack dump: [ 8854.095868] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 8854.109552] Call Trace: [ 8854.115194] __schedule+0x351/0xcb0 [ 8854.118929] schedule+0xc0/0x180 [ 8854.120164] schedule_timeout+0xb4/0x190 [ 8854.124310] ? do_raw_spin_unlock+0x75/0x190 [ 8854.128912] ? __next_timer_interrupt+0x160/0x160 [ 8854.136136] ? prepare_to_swait_event+0xf3/0x140 [ 8854.146945] rcu_gp_kthread+0x8dd/0x15b0 [ 8854.149850] ? force_qs_rnp+0x2c0/0x2c0 [ 8854.152571] kthread+0x1d1/0x200 [ 8854.155166] ? set_kthread_struct+0x70/0x70 [ 8854.164000] ret_from_fork+0x1f/0x30 [ 8854.171482] rcu: Stack dump where RCU GP kthread last ran: [ 8854.177274] Sending NMI from CPU 0 to CPUs 3: [ 8854.197499] NMI backtrace for cpu 3 [ 8854.197507] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8854.197510] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8854.197511] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8854.197513] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8854.197515] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 [ 8854.197517] RAX: 0000000000006cac RBX: ffff96848811cae0 RCX: 0000000000000001 [ 8854.197519] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 8854.197520] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8854.197521] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 8854.197522] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 8854.197524] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 8854.197525] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8854.197526] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 8854.197527] Call Trace: [ 8854.197528] [ 8854.197529] ? show_regs.cold.9+0x22/0x2f [ 8854.197530] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 8854.197531] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 8854.197532] ? nmi_handle+0xbd/0x210 [ 8854.197533] ? default_do_nmi+0x4b/0x1b0 [ 8854.197533] ? do_nmi+0x280/0x380 [ 8854.197534] ? end_repeat_nmi+0x16/0x1f [ 8854.197535] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8854.197536] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8854.197548] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8854.197549] [ 8854.197550] do_raw_spin_lock+0x177/0x190 [ 8854.197552] _raw_spin_lock+0x21/0x30 [ 8854.197553] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8854.197554] ? do_raw_spin_unlock+0x75/0x190 [ 8854.197555] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8854.197556] ? do_wait_intr+0xf0/0xf0 [ 8854.197557] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8854.197558] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8854.197559] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8854.197561] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8854.197562] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8854.197563] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8854.197564] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8854.197564] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8854.197565] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8854.197567] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8854.197567] lmv_intent_open+0x338/0x1150 [lmv] [ 8854.197569] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8854.197569] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8854.197571] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8854.197571] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8854.197572] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8854.197573] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8854.197574] ? do_raw_spin_unlock+0x75/0x190 [ 8854.197575] ? _raw_spin_unlock+0x12/0x30 [ 8854.197576] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8854.197577] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8854.197578] ll_atomic_open+0x36a/0xd70 [lustre] [ 8854.197579] lookup_open+0x356/0xb40 [ 8854.197580] do_last+0x313/0xfc0 [ 8854.197581] ? nd_jump_root+0xe5/0x160 [ 8854.197582] ? path_init+0x437/0x520 [ 8854.197582] path_openat+0xf7/0x500 [ 8854.197583] ? do_raw_spin_unlock+0x75/0x190 [ 8854.197584] do_filp_open+0x99/0x140 [ 8854.197585] ? getname_flags+0x6e/0x330 [ 8854.197585] ? __check_object_size+0xff/0x256 [ 8854.197586] ? do_raw_spin_unlock+0x75/0x190 [ 8854.197587] ? _raw_spin_unlock+0x12/0x30 [ 8854.197588] do_sys_openat2+0x2b4/0x410 [ 8854.197589] do_sys_open+0x73/0xa0 [ 8854.197590] __x64_sys_openat+0x24/0x30 [ 8854.197591] do_syscall_64+0xc1/0x440 [ 8854.197591] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8854.197592] RIP: 0033:0x7ff54af45332 [ 8854.197594] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8854.197596] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8854.197599] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 8854.197600] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 8854.197601] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 8854.197603] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 8854.197604] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 8854.205632] NMI backtrace for cpu 0 [ 8854.668915] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8854.690355] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8854.693171] Call Trace: [ 8854.699313] [ 8854.711483] ? dump_stack+0xbb/0x10e [ 8854.715267] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 8854.721225] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 8854.724432] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 8854.730435] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 8854.738418] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 8854.743686] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 8854.745302] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 8854.754677] ? update_process_times+0x6d/0xc0 [ 8854.759642] ? tick_sched_handle.isra.18+0x27/0x90 [ 8854.767874] ? tick_sched_timer+0x53/0xb0 [ 8854.771646] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8854.776169] ? tick_sched_handle.isra.18+0x90/0x90 [ 8854.777457] ? hrtimer_interrupt+0x128/0x2c0 [ 8854.778450] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8854.784648] ? apic_timer_interrupt+0xf/0x20 [ 8854.790559] [ 8854.793068] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8854.798924] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8854.808364] ? do_raw_spin_lock+0x177/0x190 [ 8854.810321] ? _raw_spin_lock+0x21/0x30 [ 8854.825873] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8854.832823] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 8854.839032] ? mdc_commit_open+0x91/0xb0 [mdc] [ 8854.842788] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8854.845855] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8854.849605] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 8854.862812] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8854.868488] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8854.870878] ? do_wait_intr+0xf0/0xf0 [ 8854.874664] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8854.883078] ? kthread+0x1d1/0x200 [ 8854.887856] ? set_kthread_struct+0x70/0x70 [ 8854.889949] ? ret_from_fork+0x1f/0x30 [ 8864.226904] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6905s! [ 8864.237903] Showing busy workqueues and worker pools: [ 8864.240885] workqueue events: flags=0x0 [ 8864.242778] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8864.246145] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 8864.254081] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8864.260552] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 8864.270599] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8864.281933] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 8864.288292] workqueue events_long: flags=0x0 [ 8864.293248] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8864.304421] pending: lu_objects_flush [obdclass] [ 8864.307138] workqueue events_unbound: flags=0x2 [ 8864.312105] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8864.315858] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8864.321205] workqueue events_power_efficient: flags=0x80 [ 8864.323665] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8864.328371] pending: neigh_periodic_work [ 8864.330377] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8864.333917] pending: do_cache_clean, check_lifetime [ 8864.337961] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8864.342944] pending: neigh_periodic_work [ 8864.346066] workqueue writeback: flags=0x4a [ 8864.354149] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8864.361647] pending: wb_workfn, wb_workfn [ 8864.363540] workqueue xprtiod: flags=0xa [ 8864.369156] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8864.376274] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 8864.384702] workqueue ipv6_addrconf: flags=0x40008 [ 8864.392197] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8864.398710] pending: addrconf_verify_work [ 8864.402389] workqueue knbd0-recv: flags=0x1a [ 8864.405051] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8864.409262] in-flight: 657:recv_work [ 8864.411230] workqueue ptlrpc_pinger: flags=0x2 [ 8864.413527] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8864.417041] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 8864.419920] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6905s workers=4 idle: 20831 12642 28859 [ 8864.424396] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8854s workers=2 idle: 659 [ 8868.058530] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [ 8868.064582] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 8868.064867] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8868.064940] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8868.064946] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8868.064949] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8868.064964] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8868.064969] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8868.064973] RAX: 0000000000007d0a RBX: ffff96848811cae0 RCX: 0000000000000001 [ 8868.064975] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 8868.064978] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8868.064980] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 8868.064982] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 8868.064985] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 8868.064988] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8868.064991] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 8868.065000] Call Trace: [ 8868.065005] [ 8868.065011] ? show_regs.cold.9+0x22/0x2f [ 8868.065020] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8868.065028] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8868.065032] ? watchdog+0x40/0x40 [ 8868.065036] ? hrtimer_interrupt+0x128/0x2c0 [ 8868.065040] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8868.065049] ? apic_timer_interrupt+0xf/0x20 [ 8868.065054] [ 8868.065058] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8868.065064] do_raw_spin_lock+0x177/0x190 [ 8868.065069] _raw_spin_lock+0x21/0x30 [ 8868.065074] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8868.065571] ? do_raw_spin_unlock+0x75/0x190 [ 8868.065578] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8868.066035] ? do_wait_intr+0xf0/0xf0 [ 8868.066044] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8868.067702] mdc_close+0x53b/0xd70 [mdc] [ 8868.067746] lmv_close+0x18d/0x490 [lmv] [ 8868.067776] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 8868.067941] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8868.068057] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8868.068139] ll_md_close+0x274/0x800 [lustre] [ 8868.068242] ll_file_release+0x128/0xc50 [lustre] [ 8868.074092] __fput+0x10d/0x440 [ 8868.074104] ____fput+0x12/0x20 [ 8868.074110] task_work_run+0xc6/0x110 [ 8868.074119] exit_to_usermode_loop+0x1dd/0x1f0 [ 8868.074127] do_syscall_64+0x3d6/0x440 [ 8868.074132] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8868.074142] RIP: 0033:0x7ff54af45cd8 [ 8868.074150] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8868.074155] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8868.074161] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 8868.074163] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 8868.074166] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 8868.074168] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 8868.074170] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8868.462156] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8868.515525] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8868.528013] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8868.547287] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8868.560818] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8868.570200] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8868.588952] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8868.594177] RAX: 0000000000007a85 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 8868.602754] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 8868.607302] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 8868.613908] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 8868.620910] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 8868.631653] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 8868.635422] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8868.647647] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 8868.659871] Call Trace: [ 8868.661123] [ 8868.666150] ? show_regs.cold.9+0x22/0x2f [ 8868.676177] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8868.680321] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8868.691210] ? watchdog+0x40/0x40 [ 8868.695185] ? hrtimer_interrupt+0x128/0x2c0 [ 8868.700516] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8868.704179] ? apic_timer_interrupt+0xf/0x20 [ 8868.706809] [ 8868.712964] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8868.715937] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8868.725701] do_raw_spin_lock+0x177/0x190 [ 8868.727855] _raw_spin_lock+0x21/0x30 [ 8868.732556] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8868.736857] process_one_work+0x2c8/0x700 [ 8868.741687] worker_thread+0x296/0x6e0 [ 8868.747986] ? rescuer_thread+0x570/0x570 [ 8868.751702] kthread+0x1d1/0x200 [ 8868.764590] ? set_kthread_struct+0x70/0x70 [ 8868.769151] ret_from_fork+0x1f/0x30 [ 8880.060804] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [ 8880.071685] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8880.122642] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8880.133003] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8880.142548] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8880.148481] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8880.177695] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8880.180897] RAX: 0000000000006c9f RBX: ffff96848811cae0 RCX: 0000000000000001 [ 8880.194454] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 8880.202048] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8880.207520] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 8880.220364] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 8880.229921] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 8880.238404] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8880.243010] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 8880.246779] Call Trace: [ 8880.248005] [ 8880.248787] ? show_regs.cold.9+0x22/0x2f [ 8880.250185] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8880.254705] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8880.256611] ? watchdog+0x40/0x40 [ 8880.258139] ? hrtimer_interrupt+0x128/0x2c0 [ 8880.261100] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8880.263712] ? apic_timer_interrupt+0xf/0x20 [ 8880.266791] [ 8880.267547] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8880.274724] do_raw_spin_lock+0x177/0x190 [ 8880.277156] _raw_spin_lock+0x21/0x30 [ 8880.278588] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8880.291318] ? do_raw_spin_unlock+0x75/0x190 [ 8880.301295] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8880.307813] ? do_wait_intr+0xf0/0xf0 [ 8880.311279] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8880.321521] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8880.323627] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8880.332576] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8880.338824] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8880.347533] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8880.349716] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8880.352776] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8880.356617] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8880.366114] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8880.370485] lmv_intent_open+0x338/0x1150 [lmv] [ 8880.374620] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8880.376040] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8880.383790] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8880.385026] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8880.387651] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8880.391429] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8880.393029] ? do_raw_spin_unlock+0x75/0x190 [ 8880.395449] ? _raw_spin_unlock+0x12/0x30 [ 8880.396640] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8880.399021] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8880.401260] ll_atomic_open+0x36a/0xd70 [lustre] [ 8880.403751] lookup_open+0x356/0xb40 [ 8880.404840] do_last+0x313/0xfc0 [ 8880.405919] ? nd_jump_root+0xe5/0x160 [ 8880.407805] ? path_init+0x437/0x520 [ 8880.409628] path_openat+0xf7/0x500 [ 8880.411380] ? do_raw_spin_unlock+0x75/0x190 [ 8880.413589] do_filp_open+0x99/0x140 [ 8880.415537] ? getname_flags+0x6e/0x330 [ 8880.417369] ? __check_object_size+0xff/0x256 [ 8880.419677] ? do_raw_spin_unlock+0x75/0x190 [ 8880.421447] ? _raw_spin_unlock+0x12/0x30 [ 8880.423775] do_sys_openat2+0x2b4/0x410 [ 8880.425577] do_sys_open+0x73/0xa0 [ 8880.427374] __x64_sys_openat+0x24/0x30 [ 8880.429471] do_syscall_64+0xc1/0x440 [ 8880.431741] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8880.433942] RIP: 0033:0x7ff54af45332 [ 8880.435684] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8880.444658] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8880.448511] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 8880.451805] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 8880.455034] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 8880.457391] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 8880.460780] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 8892.055309] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 8892.067473] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8892.123290] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8892.141789] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8892.151817] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8892.161820] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8892.182925] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8892.188308] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 8892.195801] RDX: 0000000000002521 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 8892.203611] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 8892.212767] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 8892.230633] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8892.238696] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 8892.242975] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8892.251464] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 8892.263405] Call Trace: [ 8892.264766] [ 8892.267306] ? show_regs.cold.9+0x22/0x2f [ 8892.270606] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8892.274363] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8892.275994] ? watchdog+0x40/0x40 [ 8892.287064] ? hrtimer_interrupt+0x128/0x2c0 [ 8892.288601] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8892.299529] ? apic_timer_interrupt+0xf/0x20 [ 8892.301030] [ 8892.301700] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8892.312462] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8892.317880] do_raw_spin_lock+0x177/0x190 [ 8892.327241] _raw_spin_lock+0x21/0x30 [ 8892.329331] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8892.331373] obd_mod_free+0x46/0x140 [ptlrpc] [ 8892.344103] mdc_commit_open+0x91/0xb0 [mdc] [ 8892.345654] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8892.363924] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8892.369933] after_reply+0xe2c/0x1780 [ptlrpc] [ 8892.374807] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8892.378084] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8892.384628] ? do_wait_intr+0xf0/0xf0 [ 8892.386495] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8892.396792] kthread+0x1d1/0x200 [ 8892.406439] ? set_kthread_struct+0x70/0x70 [ 8892.409377] ret_from_fork+0x1f/0x30 [ 8894.437783] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6935s! [ 8894.463636] Showing busy workqueues and worker pools: [ 8894.465521] workqueue events: flags=0x0 [ 8894.466974] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8894.482452] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 8894.497382] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8894.513705] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 8894.520372] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8894.531371] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 8894.540533] workqueue events_long: flags=0x0 [ 8894.555814] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8894.570784] pending: lu_objects_flush [obdclass] [ 8894.578973] workqueue events_unbound: flags=0x2 [ 8894.580606] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8894.598276] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8894.608258] workqueue events_power_efficient: flags=0x80 [ 8894.615627] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8894.629038] pending: neigh_periodic_work [ 8894.639292] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8894.648121] pending: do_cache_clean, check_lifetime [ 8894.667386] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8894.676071] pending: neigh_periodic_work [ 8894.688658] workqueue writeback: flags=0x4a [ 8894.694054] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8894.706364] pending: wb_workfn, wb_workfn [ 8894.707947] workqueue xprtiod: flags=0xa [ 8894.721301] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8894.735263] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 8894.745498] workqueue ipv6_addrconf: flags=0x40008 [ 8894.751506] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8894.771402] pending: addrconf_verify_work [ 8894.773058] workqueue knbd0-recv: flags=0x1a [ 8894.788348] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8894.794915] in-flight: 657:recv_work [ 8894.796363] workqueue ptlrpc_pinger: flags=0x2 [ 8894.816976] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8894.825644] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 8894.832356] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6935s workers=4 idle: 20831 12642 28859 [ 8894.840017] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8885s workers=2 idle: 659 [ 8896.060491] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [ 8896.062313] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 8896.078743] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8896.078821] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8896.078828] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8896.078832] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8896.079301] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8896.079307] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8896.079312] RAX: 0000000000002299 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 8896.079315] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 8896.079317] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8896.079319] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 8896.079321] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 8896.079324] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 8896.079326] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8896.079328] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 8896.079336] Call Trace: [ 8896.079342] [ 8896.079348] ? show_regs.cold.9+0x22/0x2f [ 8896.079358] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8896.079365] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8896.079370] ? watchdog+0x40/0x40 [ 8896.079375] ? hrtimer_interrupt+0x128/0x2c0 [ 8896.079379] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8896.079387] ? apic_timer_interrupt+0xf/0x20 [ 8896.079392] [ 8896.079396] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8896.079403] do_raw_spin_lock+0x177/0x190 [ 8896.079408] _raw_spin_lock+0x21/0x30 [ 8896.079413] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8896.085735] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8896.085930] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8896.085937] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8896.085956] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8896.086292] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8896.086304] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8896.086308] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8896.086313] RAX: 0000000000004585 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 8896.086315] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 8896.086317] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 8896.086320] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 8896.086322] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 8896.086326] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 8896.086328] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8896.086330] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 8896.086338] Call Trace: [ 8896.086343] [ 8896.086349] ? show_regs.cold.9+0x22/0x2f [ 8896.086359] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8896.086366] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8896.086370] ? watchdog+0x40/0x40 [ 8896.086375] ? hrtimer_interrupt+0x128/0x2c0 [ 8896.086379] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8896.086386] ? apic_timer_interrupt+0xf/0x20 [ 8896.086391] [ 8896.086394] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8896.086399] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8896.086404] do_raw_spin_lock+0x177/0x190 [ 8896.086409] _raw_spin_lock+0x21/0x30 [ 8896.086414] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8896.086617] process_one_work+0x2c8/0x700 [ 8896.086625] worker_thread+0x296/0x6e0 [ 8896.086629] ? rescuer_thread+0x570/0x570 [ 8896.086633] kthread+0x1d1/0x200 [ 8896.086638] ? set_kthread_struct+0x70/0x70 [ 8896.086641] ret_from_fork+0x1f/0x30 [ 8896.583783] ? do_raw_spin_unlock+0x75/0x190 [ 8896.590765] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8896.604623] ? do_wait_intr+0xf0/0xf0 [ 8896.609226] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8896.622931] mdc_close+0x53b/0xd70 [mdc] [ 8896.630398] lmv_close+0x18d/0x490 [lmv] [ 8896.633495] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 8896.642282] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8896.650473] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8896.659834] ll_md_close+0x274/0x800 [lustre] [ 8896.670292] ll_file_release+0x128/0xc50 [lustre] [ 8896.675913] __fput+0x10d/0x440 [ 8896.677699] ____fput+0x12/0x20 [ 8896.679684] task_work_run+0xc6/0x110 [ 8896.684910] exit_to_usermode_loop+0x1dd/0x1f0 [ 8896.693417] do_syscall_64+0x3d6/0x440 [ 8896.696609] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8896.709321] RIP: 0033:0x7ff54af45cd8 [ 8896.712647] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8896.744716] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8896.757777] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 8896.764434] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 8896.779484] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 8896.788498] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 8896.798137] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8908.060659] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [ 8908.080150] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8908.111543] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8908.116956] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8908.120472] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8908.123985] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8908.134546] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8908.138027] RAX: 0000000000006f86 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 8908.141475] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 8908.146105] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8908.149331] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 8908.152656] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 8908.156268] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 8908.160099] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8908.162986] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 8908.166233] Call Trace: [ 8908.167498] [ 8908.168449] ? show_regs.cold.9+0x22/0x2f [ 8908.171779] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8908.175509] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8908.178265] ? watchdog+0x40/0x40 [ 8908.180252] ? hrtimer_interrupt+0x128/0x2c0 [ 8908.182273] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8908.184479] ? apic_timer_interrupt+0xf/0x20 [ 8908.186247] [ 8908.187385] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8908.189972] do_raw_spin_lock+0x177/0x190 [ 8908.191668] _raw_spin_lock+0x21/0x30 [ 8908.193484] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8908.196493] ? do_raw_spin_unlock+0x75/0x190 [ 8908.198658] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8908.201539] ? do_wait_intr+0xf0/0xf0 [ 8908.202970] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8908.205825] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8908.208483] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8908.211983] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8908.214692] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8908.217999] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8908.219606] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8908.221389] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8908.224114] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8908.227501] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8908.230335] lmv_intent_open+0x338/0x1150 [lmv] [ 8908.232480] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8908.235217] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8908.238557] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8908.240713] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8908.243146] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8908.246420] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8908.249867] ? do_raw_spin_unlock+0x75/0x190 [ 8908.253574] ? _raw_spin_unlock+0x12/0x30 [ 8908.255951] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8908.259116] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8908.264454] ll_atomic_open+0x36a/0xd70 [lustre] [ 8908.266392] lookup_open+0x356/0xb40 [ 8908.269095] do_last+0x313/0xfc0 [ 8908.270354] ? nd_jump_root+0xe5/0x160 [ 8908.273141] ? path_init+0x437/0x520 [ 8908.274679] path_openat+0xf7/0x500 [ 8908.276500] ? do_raw_spin_unlock+0x75/0x190 [ 8908.278463] do_filp_open+0x99/0x140 [ 8908.280596] ? getname_flags+0x6e/0x330 [ 8908.281622] ? __check_object_size+0xff/0x256 [ 8908.283318] ? do_raw_spin_unlock+0x75/0x190 [ 8908.285670] ? _raw_spin_unlock+0x12/0x30 [ 8908.289492] do_sys_openat2+0x2b4/0x410 [ 8908.290920] do_sys_open+0x73/0xa0 [ 8908.293675] __x64_sys_openat+0x24/0x30 [ 8908.295195] do_syscall_64+0xc1/0x440 [ 8908.298230] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8908.299821] RIP: 0033:0x7ff54af45332 [ 8908.300919] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8908.306508] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8908.308491] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 8908.310516] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 8908.312509] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 8908.314546] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 8908.316353] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 8920.057182] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 8920.062974] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8920.118446] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8920.128508] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8920.136210] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8920.138395] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8920.166523] RSP: 0000:ffffaf9fc1b97bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8920.173630] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 8920.179797] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 8920.185618] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 8920.190696] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 8920.195713] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8920.203712] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 8920.205999] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8920.208836] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 8920.214255] Call Trace: [ 8920.215187] [ 8920.215741] ? show_regs.cold.9+0x22/0x2f [ 8920.217023] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8920.220978] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8920.222610] ? watchdog+0x40/0x40 [ 8920.224558] ? hrtimer_interrupt+0x128/0x2c0 [ 8920.229324] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8920.231357] ? apic_timer_interrupt+0xf/0x20 [ 8920.237727] [ 8920.238615] ? native_safe_halt+0x2e/0x50 [ 8920.239820] kvm_wait+0xcd/0xe0 [ 8920.240751] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8920.245029] do_raw_spin_lock+0x177/0x190 [ 8920.246762] _raw_spin_lock+0x21/0x30 [ 8920.249139] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8920.253922] obd_mod_free+0x46/0x140 [ptlrpc] [ 8920.258996] mdc_commit_open+0x91/0xb0 [mdc] [ 8920.262481] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8920.266469] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8920.268770] after_reply+0xe2c/0x1780 [ptlrpc] [ 8920.278975] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8920.290266] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8920.295475] ? do_wait_intr+0xf0/0xf0 [ 8920.297717] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8920.307509] kthread+0x1d1/0x200 [ 8920.309383] ? set_kthread_struct+0x70/0x70 [ 8920.324909] ret_from_fork+0x1f/0x30 [ 8924.059764] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [ 8924.060635] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 8924.070074] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8924.070152] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8924.070160] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8924.070163] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8924.070177] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8924.070181] RSP: 0018:ffffaf9fc1543ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8924.070186] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 8924.070188] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 8924.070191] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8924.070193] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 8924.070196] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 8924.070199] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 8924.070203] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8924.070205] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 8924.070213] Call Trace: [ 8924.070219] [ 8924.070225] ? show_regs.cold.9+0x22/0x2f [ 8924.070234] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8924.070241] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8924.070245] ? watchdog+0x40/0x40 [ 8924.070250] ? hrtimer_interrupt+0x128/0x2c0 [ 8924.070254] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8924.070260] ? apic_timer_interrupt+0xf/0x20 [ 8924.070265] [ 8924.070268] ? native_safe_halt+0x2e/0x50 [ 8924.070273] kvm_wait+0xcd/0xe0 [ 8924.070280] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8924.070289] do_raw_spin_lock+0x177/0x190 [ 8924.070293] _raw_spin_lock+0x21/0x30 [ 8924.070298] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8924.076139] Modules linked in: lustre(O) osp(O) ofd(O) [ 8924.111118] ? do_raw_spin_unlock+0x75/0x190 [ 8924.119476] lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8924.119545] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8924.119553] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8924.119563] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8924.128183] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8924.128503] ? do_wait_intr+0xf0/0xf0 [ 8924.128514] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8924.128752] mdc_close+0x53b/0xd70 [mdc] [ 8924.128792] lmv_close+0x18d/0x490 [lmv] [ 8924.129076] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 8924.129235] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8924.129355] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8924.129444] ll_md_close+0x274/0x800 [lustre] [ 8924.129531] ll_file_release+0x128/0xc50 [lustre] [ 8924.129620] __fput+0x10d/0x440 [ 8924.129629] ____fput+0x12/0x20 [ 8924.129633] task_work_run+0xc6/0x110 [ 8924.129747] exit_to_usermode_loop+0x1dd/0x1f0 [ 8924.129755] do_syscall_64+0x3d6/0x440 [ 8924.129760] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8924.129769] RIP: 0033:0x7ff54af45cd8 [ 8924.129778] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8924.129782] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8924.129787] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 8924.129790] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 8924.129792] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 8924.129795] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 8924.129798] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8924.134973] [ 8924.134986] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8924.135007] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8924.135012] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8924.135017] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 8924.135019] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 8924.135022] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 8924.135024] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 8924.135027] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 8924.135030] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 8924.135034] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8924.135037] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 8924.135046] Call Trace: [ 8924.135053] [ 8924.135060] ? show_regs.cold.9+0x22/0x2f [ 8924.135072] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8924.135079] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8924.135084] ? watchdog+0x40/0x40 [ 8924.135088] ? hrtimer_interrupt+0x128/0x2c0 [ 8924.135093] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8924.135100] ? apic_timer_interrupt+0xf/0x20 [ 8924.135104] [ 8924.135108] ? native_safe_halt+0x2e/0x50 [ 8924.135112] kvm_wait+0xcd/0xe0 [ 8924.135119] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8924.135129] do_raw_spin_lock+0x177/0x190 [ 8924.135133] _raw_spin_lock+0x21/0x30 [ 8924.135137] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8924.135630] process_one_work+0x2c8/0x700 [ 8924.135638] worker_thread+0x296/0x6e0 [ 8924.135642] ? rescuer_thread+0x570/0x570 [ 8924.135646] kthread+0x1d1/0x200 [ 8924.135651] ? set_kthread_struct+0x70/0x70 [ 8924.135654] ret_from_fork+0x1f/0x30 [ 8925.151832] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6966s! [ 8925.170666] Showing busy workqueues and worker pools: [ 8925.174204] workqueue events: flags=0x0 [ 8925.182578] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8925.184320] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 8925.213031] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8925.227262] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 8925.242411] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8925.252017] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 8925.260803] workqueue events_long: flags=0x0 [ 8925.266453] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8925.274449] pending: lu_objects_flush [obdclass] [ 8925.280634] workqueue events_unbound: flags=0x2 [ 8925.290037] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8925.294677] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8925.315465] workqueue events_power_efficient: flags=0x80 [ 8925.322483] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8925.329003] pending: neigh_periodic_work [ 8925.332735] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8925.335553] pending: do_cache_clean, check_lifetime [ 8925.337918] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8925.342648] pending: neigh_periodic_work [ 8925.344714] workqueue writeback: flags=0x4a [ 8925.351820] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8925.356902] pending: wb_workfn, wb_workfn [ 8925.360642] workqueue xprtiod: flags=0xa [ 8925.364468] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8925.370068] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 8925.375484] workqueue ipv6_addrconf: flags=0x40008 [ 8925.385319] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8925.391918] pending: addrconf_verify_work [ 8925.393826] workqueue knbd0-recv: flags=0x1a [ 8925.396952] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8925.406115] in-flight: 657:recv_work [ 8925.414072] workqueue ptlrpc_pinger: flags=0x2 [ 8925.417011] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8925.420299] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 8925.422710] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6966s workers=4 idle: 20831 12642 28859 [ 8925.438924] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8915s workers=2 idle: 659 [ 8936.060251] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 8936.065036] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8936.096366] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8936.102424] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8936.108069] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8936.118121] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8936.136960] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8936.145529] RAX: 0000000000003e92 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 8936.169461] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 8936.179011] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8936.181452] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 8936.185340] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 8936.189052] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 8936.194122] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8936.196557] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 8936.199530] Call Trace: [ 8936.200532] [ 8936.201499] ? show_regs.cold.9+0x22/0x2f [ 8936.203950] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8936.205542] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8936.207062] ? watchdog+0x40/0x40 [ 8936.208738] ? hrtimer_interrupt+0x128/0x2c0 [ 8936.209879] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8936.211455] ? apic_timer_interrupt+0xf/0x20 [ 8936.212662] [ 8936.213202] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8936.215290] do_raw_spin_lock+0x177/0x190 [ 8936.216213] _raw_spin_lock+0x21/0x30 [ 8936.217149] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8936.218557] ? do_raw_spin_unlock+0x75/0x190 [ 8936.222128] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8936.224108] ? do_wait_intr+0xf0/0xf0 [ 8936.225396] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8936.227337] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8936.229351] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8936.231533] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8936.233405] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8936.235510] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8936.237147] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8936.238635] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8936.240802] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8936.246435] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8936.247915] lmv_intent_open+0x338/0x1150 [lmv] [ 8936.249055] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8936.251054] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8936.253226] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8936.254970] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8936.256879] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8936.258912] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8936.260726] ? do_raw_spin_unlock+0x75/0x190 [ 8936.262287] ? _raw_spin_unlock+0x12/0x30 [ 8936.263724] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8936.265801] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8936.267814] ll_atomic_open+0x36a/0xd70 [lustre] [ 8936.269704] lookup_open+0x356/0xb40 [ 8936.271102] do_last+0x313/0xfc0 [ 8936.272391] ? nd_jump_root+0xe5/0x160 [ 8936.273939] ? path_init+0x437/0x520 [ 8936.275533] path_openat+0xf7/0x500 [ 8936.276980] ? do_raw_spin_unlock+0x75/0x190 [ 8936.278677] do_filp_open+0x99/0x140 [ 8936.279994] ? getname_flags+0x6e/0x330 [ 8936.281411] ? __check_object_size+0xff/0x256 [ 8936.283006] ? do_raw_spin_unlock+0x75/0x190 [ 8936.284534] ? _raw_spin_unlock+0x12/0x30 [ 8936.286056] do_sys_openat2+0x2b4/0x410 [ 8936.287455] do_sys_open+0x73/0xa0 [ 8936.288859] __x64_sys_openat+0x24/0x30 [ 8936.290180] do_syscall_64+0xc1/0x440 [ 8936.291469] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8936.293047] RIP: 0033:0x7ff54af45332 [ 8936.294127] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8936.303245] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8936.306126] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 8936.308679] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 8936.311124] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 8936.313649] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 8936.316007] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 8952.056457] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [ 8952.060166] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 8952.063398] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8952.063475] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8952.063481] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8952.063484] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8952.063497] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8952.063502] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8952.063507] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 8952.063509] RDX: 00000000000065a9 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 8952.063511] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 8952.063513] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 8952.063516] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8952.063520] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 8952.063523] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8952.063525] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 8952.063534] Call Trace: [ 8952.063540] [ 8952.063546] ? show_regs.cold.9+0x22/0x2f [ 8952.063555] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8952.063561] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8952.063566] ? watchdog+0x40/0x40 [ 8952.063571] ? hrtimer_interrupt+0x128/0x2c0 [ 8952.063575] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8952.063583] ? apic_timer_interrupt+0xf/0x20 [ 8952.063721] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [ 8952.063738] [ 8952.063743] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8952.063749] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8952.063754] do_raw_spin_lock+0x177/0x190 [ 8952.063759] _raw_spin_lock+0x21/0x30 [ 8952.063766] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8952.064529] obd_mod_free+0x46/0x140 [ptlrpc] [ 8952.064740] mdc_commit_open+0x91/0xb0 [mdc] [ 8952.064778] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8952.064980] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8952.065362] after_reply+0xe2c/0x1780 [ptlrpc] [ 8952.065556] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8952.065763] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8952.065985] ? do_wait_intr+0xf0/0xf0 [ 8952.067652] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8952.067918] kthread+0x1d1/0x200 [ 8952.067926] ? set_kthread_struct+0x70/0x70 [ 8952.067930] ret_from_fork+0x1f/0x30 [ 8952.067936] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8952.068024] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8952.068030] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8952.068036] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8952.068763] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8952.068775] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8952.068779] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8952.068784] RAX: 0000000000004ee9 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 8952.068786] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 8952.068788] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 8952.068790] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 8952.068793] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 8952.068796] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 8952.068799] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8952.068801] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 8952.068809] Call Trace: [ 8952.068814] [ 8952.068818] ? show_regs.cold.9+0x22/0x2f [ 8952.068828] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8952.068836] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8952.068841] ? watchdog+0x40/0x40 [ 8952.068846] ? hrtimer_interrupt+0x128/0x2c0 [ 8952.068850] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8952.068857] ? apic_timer_interrupt+0xf/0x20 [ 8952.068861] [ 8952.068864] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8952.068869] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8952.068875] do_raw_spin_lock+0x177/0x190 [ 8952.068880] _raw_spin_lock+0x21/0x30 [ 8952.068885] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8952.069754] process_one_work+0x2c8/0x700 [ 8952.069763] worker_thread+0x296/0x6e0 [ 8952.069768] ? rescuer_thread+0x570/0x570 [ 8952.069772] kthread+0x1d1/0x200 [ 8952.069776] ? set_kthread_struct+0x70/0x70 [ 8952.069780] ret_from_fork+0x1f/0x30 [ 8952.434682] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8952.458841] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8952.465187] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8952.469125] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8952.477208] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8952.493709] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 8952.498852] RAX: 0000000000003e14 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 8952.506293] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 8952.509739] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8952.515336] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 8952.522156] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 8952.530346] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 8952.538813] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8952.544926] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 8952.553184] Call Trace: [ 8952.556113] [ 8952.559664] ? show_regs.cold.9+0x22/0x2f [ 8952.561332] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8952.563938] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8952.565667] ? watchdog+0x40/0x40 [ 8952.567573] ? hrtimer_interrupt+0x128/0x2c0 [ 8952.569537] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8952.571700] ? apic_timer_interrupt+0xf/0x20 [ 8952.573725] [ 8952.575170] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8952.577760] do_raw_spin_lock+0x177/0x190 [ 8952.579756] _raw_spin_lock+0x21/0x30 [ 8952.581771] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8952.584714] ? do_raw_spin_unlock+0x75/0x190 [ 8952.587184] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8952.590203] ? do_wait_intr+0xf0/0xf0 [ 8952.592342] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8952.596155] mdc_close+0x53b/0xd70 [mdc] [ 8952.598625] lmv_close+0x18d/0x490 [lmv] [ 8952.602084] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 8952.604942] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8952.614259] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8952.616096] ll_md_close+0x274/0x800 [lustre] [ 8952.619915] ll_file_release+0x128/0xc50 [lustre] [ 8952.622670] __fput+0x10d/0x440 [ 8952.623979] ____fput+0x12/0x20 [ 8952.627868] task_work_run+0xc6/0x110 [ 8952.631351] exit_to_usermode_loop+0x1dd/0x1f0 [ 8952.632893] do_syscall_64+0x3d6/0x440 [ 8952.635370] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8952.639841] RIP: 0033:0x7ff54af45cd8 [ 8952.641977] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8952.652786] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8952.660300] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 8952.668956] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 8952.673643] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 8952.677677] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 8952.681877] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8955.871626] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6996s! [ 8955.881239] Showing busy workqueues and worker pools: [ 8955.886606] workqueue events: flags=0x0 [ 8955.890169] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8955.901840] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 8955.911795] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8955.919340] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 8955.930489] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8955.937524] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 8955.942900] workqueue events_long: flags=0x0 [ 8955.944831] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8955.953773] pending: lu_objects_flush [obdclass] [ 8955.961235] workqueue events_unbound: flags=0x2 [ 8955.965897] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8955.976028] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8955.984331] workqueue events_power_efficient: flags=0x80 [ 8955.987160] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8955.994894] pending: neigh_periodic_work [ 8956.001939] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8956.010518] pending: do_cache_clean, check_lifetime [ 8956.015314] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8956.018314] pending: neigh_periodic_work [ 8956.020733] workqueue writeback: flags=0x4a [ 8956.023640] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8956.030180] pending: wb_workfn, wb_workfn [ 8956.035583] workqueue xprtiod: flags=0xa [ 8956.038978] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8956.045361] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 8956.051185] workqueue ipv6_addrconf: flags=0x40008 [ 8956.052665] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8956.055533] pending: addrconf_verify_work [ 8956.057642] workqueue knbd0-recv: flags=0x1a [ 8956.059811] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8956.064752] in-flight: 657:recv_work [ 8956.068164] workqueue ptlrpc_pinger: flags=0x2 [ 8956.072567] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8956.076826] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 8956.082624] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6997s workers=4 idle: 20831 12642 28859 [ 8956.086762] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8946s workers=2 idle: 659 [ 8964.060020] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 8964.062781] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8964.077218] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8964.080590] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8964.082858] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8964.084211] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8964.089234] RSP: 0000:ffffaf9fc15234f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8964.091382] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 8964.093178] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 8964.095629] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8964.097672] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 8964.099885] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 8964.102123] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 8964.104457] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8964.106104] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 8964.108070] Call Trace: [ 8964.108850] [ 8964.109593] ? show_regs.cold.9+0x22/0x2f [ 8964.110733] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8964.112099] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8964.113439] ? watchdog+0x40/0x40 [ 8964.114433] ? hrtimer_interrupt+0x128/0x2c0 [ 8964.115641] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8964.117039] ? apic_timer_interrupt+0xf/0x20 [ 8964.118490] [ 8964.118998] ? native_safe_halt+0x2e/0x50 [ 8964.120598] kvm_wait+0xcd/0xe0 [ 8964.121773] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8964.123263] do_raw_spin_lock+0x177/0x190 [ 8964.124194] _raw_spin_lock+0x21/0x30 [ 8964.125114] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8964.126969] ? do_raw_spin_unlock+0x75/0x190 [ 8964.128643] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8964.130322] ? do_wait_intr+0xf0/0xf0 [ 8964.131246] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8964.133640] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8964.135067] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8964.136821] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8964.138807] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8964.140797] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8964.142655] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8964.143853] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8964.146072] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8964.147672] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8964.149173] lmv_intent_open+0x338/0x1150 [lmv] [ 8964.150236] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8964.152174] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8964.154193] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8964.155987] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8964.157745] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8964.159826] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8964.161378] ? do_raw_spin_unlock+0x75/0x190 [ 8964.162899] ? _raw_spin_unlock+0x12/0x30 [ 8964.164377] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8964.166035] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8964.167996] ll_atomic_open+0x36a/0xd70 [lustre] [ 8964.169672] lookup_open+0x356/0xb40 [ 8964.170641] do_last+0x313/0xfc0 [ 8964.171568] ? nd_jump_root+0xe5/0x160 [ 8964.172602] ? path_init+0x437/0x520 [ 8964.173513] path_openat+0xf7/0x500 [ 8964.174768] ? do_raw_spin_unlock+0x75/0x190 [ 8964.176290] do_filp_open+0x99/0x140 [ 8964.177624] ? getname_flags+0x6e/0x330 [ 8964.179139] ? __check_object_size+0xff/0x256 [ 8964.180680] ? do_raw_spin_unlock+0x75/0x190 [ 8964.181837] ? _raw_spin_unlock+0x12/0x30 [ 8964.182975] do_sys_openat2+0x2b4/0x410 [ 8964.184401] do_sys_open+0x73/0xa0 [ 8964.185139] __x64_sys_openat+0x24/0x30 [ 8964.186266] do_syscall_64+0xc1/0x440 [ 8964.187761] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8964.189635] RIP: 0033:0x7ff54af45332 [ 8964.191052] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8964.197446] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8964.200442] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 8964.203035] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 8964.205738] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 8964.208377] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 8964.211001] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 8980.058060] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [ 8980.059031] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 8980.063428] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8980.063688] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8980.063695] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8980.063698] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8980.063710] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8980.063715] RSP: 0018:ffffaf9fc1543ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8980.064381] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 8980.064383] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 8980.064386] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8980.064388] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 8980.064391] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 8980.064394] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 8980.064396] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8980.064399] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 8980.064407] Call Trace: [ 8980.064412] [ 8980.064418] ? show_regs.cold.9+0x22/0x2f [ 8980.064429] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8980.064435] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8980.064439] ? watchdog+0x40/0x40 [ 8980.064444] ? hrtimer_interrupt+0x128/0x2c0 [ 8980.064447] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8980.064455] ? apic_timer_interrupt+0xf/0x20 [ 8980.064459] [ 8980.064462] ? native_safe_halt+0x2e/0x50 [ 8980.064467] kvm_wait+0xcd/0xe0 [ 8980.064474] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8980.064483] do_raw_spin_lock+0x177/0x190 [ 8980.064487] _raw_spin_lock+0x21/0x30 [ 8980.064491] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8980.065965] ? do_raw_spin_unlock+0x75/0x190 [ 8980.065975] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8980.068160] ? do_wait_intr+0xf0/0xf0 [ 8980.068172] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8980.073298] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8980.073387] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8980.073395] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8980.073403] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8980.100906] mdc_close+0x53b/0xd70 [mdc] [ 8980.106413] [ 8980.106424] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8980.106443] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8980.106447] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8980.106452] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 8980.106455] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 8980.106457] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 8980.106460] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 8980.106463] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 8980.106466] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 8980.106469] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8980.106472] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 8980.106571] Call Trace: [ 8980.111983] lmv_close+0x18d/0x490 [lmv] [ 8980.114633] [ 8980.125079] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 8980.129502] ? show_regs.cold.9+0x22/0x2f [ 8980.129523] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8980.129530] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8980.129535] ? watchdog+0x40/0x40 [ 8980.129539] ? hrtimer_interrupt+0x128/0x2c0 [ 8980.129543] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8980.129550] ? apic_timer_interrupt+0xf/0x20 [ 8980.129555] [ 8980.129560] ? native_safe_halt+0x2e/0x50 [ 8980.129812] kvm_wait+0xcd/0xe0 [ 8980.129822] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8980.129833] do_raw_spin_lock+0x177/0x190 [ 8980.129838] _raw_spin_lock+0x21/0x30 [ 8980.129843] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8980.133677] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8980.136713] process_one_work+0x2c8/0x700 [ 8980.136734] worker_thread+0x296/0x6e0 [ 8980.136739] ? rescuer_thread+0x570/0x570 [ 8980.136743] kthread+0x1d1/0x200 [ 8980.136747] ? set_kthread_struct+0x70/0x70 [ 8980.136751] ret_from_fork+0x1f/0x30 [ 8980.283465] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8980.284785] ll_md_close+0x274/0x800 [lustre] [ 8980.286034] ll_file_release+0x128/0xc50 [lustre] [ 8980.287749] __fput+0x10d/0x440 [ 8980.289359] ____fput+0x12/0x20 [ 8980.290107] task_work_run+0xc6/0x110 [ 8980.291911] exit_to_usermode_loop+0x1dd/0x1f0 [ 8980.293716] do_syscall_64+0x3d6/0x440 [ 8980.295000] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8980.297891] RIP: 0033:0x7ff54af45cd8 [ 8980.298991] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8980.308308] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8980.312373] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 8980.315370] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 8980.317959] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 8980.320126] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 8980.322808] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8984.055290] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [ 8984.061656] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8984.091585] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8984.097779] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8984.102388] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8984.105544] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8984.138941] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8984.152799] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 8984.159705] RDX: 000000000000073c RSI: 0000000000000003 RDI: ffff96848811cae0 [ 8984.170009] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 8984.172581] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 8984.175548] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8984.178871] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 8984.182217] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8984.184369] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 8984.187291] Call Trace: [ 8984.188318] [ 8984.189085] ? show_regs.cold.9+0x22/0x2f [ 8984.190788] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8984.192665] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8984.194540] ? watchdog+0x40/0x40 [ 8984.195850] ? hrtimer_interrupt+0x128/0x2c0 [ 8984.197478] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8984.199272] ? apic_timer_interrupt+0xf/0x20 [ 8984.201208] [ 8984.202011] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8984.204303] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8984.206368] do_raw_spin_lock+0x177/0x190 [ 8984.207996] _raw_spin_lock+0x21/0x30 [ 8984.209465] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8984.211806] obd_mod_free+0x46/0x140 [ptlrpc] [ 8984.213841] mdc_commit_open+0x91/0xb0 [mdc] [ 8984.215496] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8984.217804] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8984.220165] after_reply+0xe2c/0x1780 [ptlrpc] [ 8984.222317] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8984.224690] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8984.226656] ? do_wait_intr+0xf0/0xf0 [ 8984.228164] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8984.230779] kthread+0x1d1/0x200 [ 8984.232065] ? set_kthread_struct+0x70/0x70 [ 8984.233754] ret_from_fork+0x1f/0x30 [ 8986.591111] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7027s! [ 8986.604067] Showing busy workqueues and worker pools: [ 8986.615709] workqueue events: flags=0x0 [ 8986.620453] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8986.627326] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 8986.633106] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8986.645438] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 8986.656852] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8986.662569] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 8986.669922] workqueue events_long: flags=0x0 [ 8986.676786] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8986.679052] pending: lu_objects_flush [obdclass] [ 8986.686815] workqueue events_unbound: flags=0x2 [ 8986.694430] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8986.699243] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8986.711575] workqueue events_power_efficient: flags=0x80 [ 8986.713910] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8986.719870] pending: neigh_periodic_work [ 8986.722814] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8986.727788] pending: do_cache_clean, check_lifetime [ 8986.730767] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8986.735833] pending: neigh_periodic_work [ 8986.741251] workqueue writeback: flags=0x4a [ 8986.743913] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8986.748832] pending: wb_workfn, wb_workfn [ 8986.751782] workqueue xprtiod: flags=0xa [ 8986.754012] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8986.756957] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 8986.764495] workqueue ipv6_addrconf: flags=0x40008 [ 8986.771786] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8986.778830] pending: addrconf_verify_work [ 8986.780449] workqueue knbd0-recv: flags=0x1a [ 8986.786536] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8986.791824] in-flight: 657:recv_work [ 8986.793153] workqueue ptlrpc_pinger: flags=0x2 [ 8986.796738] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8986.806206] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 8986.814161] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7027s workers=4 idle: 20831 12642 28859 [ 8986.822788] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8977s workers=2 idle: 659 [ 8992.060737] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 8992.067043] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8992.121630] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8992.125388] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8992.129338] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8992.131345] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8992.149763] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8992.152616] RAX: 0000000000004c96 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 8992.156658] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 8992.159860] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8992.162825] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 8992.166649] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 8992.169374] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 8992.172868] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8992.175130] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 8992.177775] Call Trace: [ 8992.180416] [ 8992.181558] ? show_regs.cold.9+0x22/0x2f [ 8992.183604] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8992.191467] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8992.194952] ? watchdog+0x40/0x40 [ 8992.199840] ? hrtimer_interrupt+0x128/0x2c0 [ 8992.209952] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8992.215633] ? apic_timer_interrupt+0xf/0x20 [ 8992.218884] [ 8992.221223] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8992.226975] do_raw_spin_lock+0x177/0x190 [ 8992.230572] _raw_spin_lock+0x21/0x30 [ 8992.239650] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8992.241082] ? do_raw_spin_unlock+0x75/0x190 [ 8992.249473] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8992.252635] ? do_wait_intr+0xf0/0xf0 [ 8992.254956] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8992.260129] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8992.263473] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8992.268687] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8992.271154] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8992.274913] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8992.279515] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8992.281462] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8992.284774] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8992.287215] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8992.294818] lmv_intent_open+0x338/0x1150 [lmv] [ 8992.296139] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8992.303112] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8992.313708] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8992.315353] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8992.322949] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8992.332893] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8992.339623] ? do_raw_spin_unlock+0x75/0x190 [ 8992.344886] ? _raw_spin_unlock+0x12/0x30 [ 8992.349374] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8992.351031] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8992.358077] ll_atomic_open+0x36a/0xd70 [lustre] [ 8992.360494] lookup_open+0x356/0xb40 [ 8992.361953] do_last+0x313/0xfc0 [ 8992.365980] ? nd_jump_root+0xe5/0x160 [ 8992.368879] ? path_init+0x437/0x520 [ 8992.371723] path_openat+0xf7/0x500 [ 8992.374918] ? do_raw_spin_unlock+0x75/0x190 [ 8992.377479] do_filp_open+0x99/0x140 [ 8992.378978] ? getname_flags+0x6e/0x330 [ 8992.382394] ? __check_object_size+0xff/0x256 [ 8992.386104] ? do_raw_spin_unlock+0x75/0x190 [ 8992.388730] ? _raw_spin_unlock+0x12/0x30 [ 8992.390519] do_sys_openat2+0x2b4/0x410 [ 8992.396562] do_sys_open+0x73/0xa0 [ 8992.398206] __x64_sys_openat+0x24/0x30 [ 8992.402006] do_syscall_64+0xc1/0x440 [ 8992.410153] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8992.414636] RIP: 0033:0x7ff54af45332 [ 8992.418486] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8992.429306] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8992.437434] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 8992.442852] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 8992.447063] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 8992.451720] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 8992.454995] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 9008.059004] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 9008.061070] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 9008.062608] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9008.062682] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9008.062688] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9008.062690] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9008.062702] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9008.062705] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9008.062708] RAX: 0000000000001c88 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 9008.062710] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 9008.062712] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9008.062713] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 9008.062715] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 9008.062717] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 9008.062719] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9008.062721] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 9008.062728] Call Trace: [ 9008.062732] [ 9008.062736] ? show_regs.cold.9+0x22/0x2f [ 9008.062743] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9008.062747] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9008.062750] ? watchdog+0x40/0x40 [ 9008.062753] ? hrtimer_interrupt+0x128/0x2c0 [ 9008.062755] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9008.062762] ? apic_timer_interrupt+0xf/0x20 [ 9008.062765] [ 9008.062767] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9008.063088] do_raw_spin_lock+0x177/0x190 [ 9008.063095] _raw_spin_lock+0x21/0x30 [ 9008.063101] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9008.064097] ? do_raw_spin_unlock+0x75/0x190 [ 9008.064104] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9008.064400] ? do_wait_intr+0xf0/0xf0 [ 9008.064408] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9008.064778] mdc_close+0x53b/0xd70 [mdc] [ 9008.064819] lmv_close+0x18d/0x490 [lmv] [ 9008.064850] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 9008.072568] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash [ 9008.099189] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9008.106660] dm_log dm_mod sha512_ssse3 sha512_generic [ 9008.106679] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9008.106687] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9008.106693] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9008.112923] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9008.113145] ll_md_close+0x274/0x800 [lustre] [ 9008.113291] ll_file_release+0x128/0xc50 [lustre] [ 9008.113377] __fput+0x10d/0x440 [ 9008.113386] ____fput+0x12/0x20 [ 9008.113391] task_work_run+0xc6/0x110 [ 9008.113399] exit_to_usermode_loop+0x1dd/0x1f0 [ 9008.113405] do_syscall_64+0x3d6/0x440 [ 9008.113410] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9008.113418] RIP: 0033:0x7ff54af45cd8 [ 9008.113425] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9008.113430] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9008.113436] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 9008.113493] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 9008.113496] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 9008.113498] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 9008.113501] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9008.450821] [ 9008.451424] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9008.459614] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9008.476368] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9008.484781] RAX: 00000000000037c1 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 9008.489101] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 9008.496437] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 9008.506071] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 9008.509719] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 9008.516299] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 9008.525600] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9008.528968] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 9008.538432] Call Trace: [ 9008.541546] [ 9008.542915] ? show_regs.cold.9+0x22/0x2f [ 9008.544146] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9008.546325] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9008.548105] ? watchdog+0x40/0x40 [ 9008.553723] ? hrtimer_interrupt+0x128/0x2c0 [ 9008.555884] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9008.556914] ? apic_timer_interrupt+0xf/0x20 [ 9008.558030] [ 9008.558494] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9008.563286] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9008.564719] do_raw_spin_lock+0x177/0x190 [ 9008.566327] _raw_spin_lock+0x21/0x30 [ 9008.567461] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9008.569483] process_one_work+0x2c8/0x700 [ 9008.570506] worker_thread+0x296/0x6e0 [ 9008.571549] ? rescuer_thread+0x570/0x570 [ 9008.572807] kthread+0x1d1/0x200 [ 9008.574677] ? set_kthread_struct+0x70/0x70 [ 9008.581521] ret_from_fork+0x1f/0x30 [ 9012.055044] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [ 9012.062044] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9012.107311] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9012.121090] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9012.124891] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9012.128121] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9012.136348] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9012.140623] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 9012.144279] RDX: 00000000000055f0 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 9012.147601] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 9012.154821] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 9012.158577] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9012.170640] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 9012.176787] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9012.180322] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 9012.184643] Call Trace: [ 9012.185894] [ 9012.186727] ? show_regs.cold.9+0x22/0x2f [ 9012.189146] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9012.192439] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9012.194705] ? watchdog+0x40/0x40 [ 9012.196536] ? hrtimer_interrupt+0x128/0x2c0 [ 9012.198512] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9012.201950] ? apic_timer_interrupt+0xf/0x20 [ 9012.205618] [ 9012.206822] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9012.210646] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9012.214503] do_raw_spin_lock+0x177/0x190 [ 9012.218099] _raw_spin_lock+0x21/0x30 [ 9012.221600] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9012.232615] obd_mod_free+0x46/0x140 [ptlrpc] [ 9012.239917] mdc_commit_open+0x91/0xb0 [mdc] [ 9012.244304] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9012.248093] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9012.251655] after_reply+0xe2c/0x1780 [ptlrpc] [ 9012.254617] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9012.259613] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9012.264698] ? do_wait_intr+0xf0/0xf0 [ 9012.267257] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9012.270680] kthread+0x1d1/0x200 [ 9012.273264] ? set_kthread_struct+0x70/0x70 [ 9012.275403] ret_from_fork+0x1f/0x30 [ 9017.312138] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7058s! [ 9017.316726] Showing busy workqueues and worker pools: [ 9017.319696] workqueue events: flags=0x0 [ 9017.322128] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9017.324199] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 9017.332270] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 9017.338884] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 9017.348066] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9017.352488] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 9017.358637] workqueue events_long: flags=0x0 [ 9017.366878] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9017.378169] pending: lu_objects_flush [obdclass] [ 9017.380013] workqueue events_unbound: flags=0x2 [ 9017.393716] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 9017.406167] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9017.417604] workqueue events_power_efficient: flags=0x80 [ 9017.419464] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9017.429757] pending: neigh_periodic_work [ 9017.436456] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9017.442325] pending: do_cache_clean, check_lifetime [ 9017.447706] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9017.456968] pending: neigh_periodic_work [ 9017.458305] workqueue writeback: flags=0x4a [ 9017.459744] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9017.463636] pending: wb_workfn, wb_workfn [ 9017.466750] workqueue xprtiod: flags=0xa [ 9017.470359] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9017.477337] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 9017.486312] workqueue ipv6_addrconf: flags=0x40008 [ 9017.492437] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9017.497560] pending: addrconf_verify_work [ 9017.501353] workqueue knbd0-recv: flags=0x1a [ 9017.504699] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9017.516243] in-flight: 657:recv_work [ 9017.523161] workqueue ptlrpc_pinger: flags=0x2 [ 9017.524758] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9017.533551] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 9017.541159] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7058s workers=4 idle: 20831 12642 28859 [ 9017.547546] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9007s workers=2 idle: 659 [ 9020.062758] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 9020.068081] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9020.130662] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9020.142708] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9020.149386] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9020.154730] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9020.173573] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9020.177656] RAX: 000000000000308d RBX: ffff96848811cae0 RCX: 0000000000000001 [ 9020.193371] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 9020.201501] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9020.206740] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 9020.211336] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 9020.218090] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 9020.220621] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9020.222501] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 9020.225541] Call Trace: [ 9020.226522] [ 9020.228273] ? show_regs.cold.9+0x22/0x2f [ 9020.231425] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9020.233885] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9020.237187] ? watchdog+0x40/0x40 [ 9020.239042] ? hrtimer_interrupt+0x128/0x2c0 [ 9020.241338] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9020.244543] ? apic_timer_interrupt+0xf/0x20 [ 9020.249325] [ 9020.250066] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9020.252689] do_raw_spin_lock+0x177/0x190 [ 9020.257447] _raw_spin_lock+0x21/0x30 [ 9020.259719] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9020.267836] ? do_raw_spin_unlock+0x75/0x190 [ 9020.273297] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9020.275743] ? do_wait_intr+0xf0/0xf0 [ 9020.277871] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9020.280956] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9020.283627] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9020.288358] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9020.290579] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9020.292614] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9020.294432] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9020.296445] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9020.299157] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9020.302149] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9020.305390] lmv_intent_open+0x338/0x1150 [lmv] [ 9020.308336] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9020.310561] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9020.317000] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9020.318684] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9020.321168] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9020.323398] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9020.325337] ? do_raw_spin_unlock+0x75/0x190 [ 9020.326585] ? _raw_spin_unlock+0x12/0x30 [ 9020.327730] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9020.329252] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9020.331124] ll_atomic_open+0x36a/0xd70 [lustre] [ 9020.333042] lookup_open+0x356/0xb40 [ 9020.334467] do_last+0x313/0xfc0 [ 9020.335689] ? nd_jump_root+0xe5/0x160 [ 9020.337769] ? path_init+0x437/0x520 [ 9020.340565] path_openat+0xf7/0x500 [ 9020.342025] ? do_raw_spin_unlock+0x75/0x190 [ 9020.346014] do_filp_open+0x99/0x140 [ 9020.347464] ? getname_flags+0x6e/0x330 [ 9020.349094] ? __check_object_size+0xff/0x256 [ 9020.350945] ? do_raw_spin_unlock+0x75/0x190 [ 9020.352568] ? _raw_spin_unlock+0x12/0x30 [ 9020.354300] do_sys_openat2+0x2b4/0x410 [ 9020.355669] do_sys_open+0x73/0xa0 [ 9020.357048] __x64_sys_openat+0x24/0x30 [ 9020.358402] do_syscall_64+0xc1/0x440 [ 9020.359987] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9020.362134] RIP: 0033:0x7ff54af45332 [ 9020.363586] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9020.374909] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9020.377880] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 9020.381195] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 9020.384242] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 9020.387116] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 9020.390313] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 9034.054506] rcu: INFO: rcu_sched self-detected stall on CPU [ 9034.060223] rcu: 0-...!: (3673264 ticks this GP) idle=98a/1/0x4000000000000002 softirq=125154/125154 fqs=959 [ 9034.078033] (t=7080214 jiffies g=253585 q=5209) [ 9034.086865] rcu: rcu_sched kthread starved for 7075174 jiffies! g253585 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [ 9034.096930] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 9034.127991] rcu: RCU grace-period kthread stack dump: [ 9034.137816] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 9034.156030] Call Trace: [ 9034.156983] __schedule+0x351/0xcb0 [ 9034.158377] schedule+0xc0/0x180 [ 9034.183961] schedule_timeout+0xb4/0x190 [ 9034.192386] ? do_raw_spin_unlock+0x75/0x190 [ 9034.193822] ? __next_timer_interrupt+0x160/0x160 [ 9034.204185] ? prepare_to_swait_event+0xf3/0x140 [ 9034.208139] rcu_gp_kthread+0x8dd/0x15b0 [ 9034.222805] ? force_qs_rnp+0x2c0/0x2c0 [ 9034.229807] kthread+0x1d1/0x200 [ 9034.234783] ? set_kthread_struct+0x70/0x70 [ 9034.245701] ret_from_fork+0x1f/0x30 [ 9034.253022] rcu: Stack dump where RCU GP kthread last ran: [ 9034.268498] Sending NMI from CPU 0 to CPUs 1: [ 9034.286822] NMI backtrace for cpu 1 [ 9034.286829] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9034.286832] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9034.286833] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9034.286835] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9034.286837] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 [ 9034.286839] RAX: 0000000000006f0a RBX: ffff96848811cae0 RCX: 0000000000000001 [ 9034.286841] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 9034.286842] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9034.286843] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 9034.286845] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 9034.286846] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 9034.286848] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9034.286849] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 9034.286850] Call Trace: [ 9034.286851] [ 9034.286852] ? show_regs.cold.9+0x22/0x2f [ 9034.286853] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 9034.286854] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 9034.286855] ? nmi_handle+0xbd/0x210 [ 9034.286856] ? default_do_nmi+0x4b/0x1b0 [ 9034.286858] ? do_nmi+0x280/0x380 [ 9034.286859] ? end_repeat_nmi+0x16/0x1f [ 9034.286860] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9034.286861] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9034.286862] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9034.286864] [ 9034.286864] do_raw_spin_lock+0x177/0x190 [ 9034.286866] _raw_spin_lock+0x21/0x30 [ 9034.286867] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9034.286868] ? do_raw_spin_unlock+0x75/0x190 [ 9034.286869] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9034.286871] ? do_wait_intr+0xf0/0xf0 [ 9034.286872] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9034.286873] mdc_close+0x53b/0xd70 [mdc] [ 9034.286874] lmv_close+0x18d/0x490 [lmv] [ 9034.286875] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 9034.286877] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9034.286878] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9034.286880] ll_md_close+0x274/0x800 [lustre] [ 9034.286881] ll_file_release+0x128/0xc50 [lustre] [ 9034.286882] __fput+0x10d/0x440 [ 9034.286883] ____fput+0x12/0x20 [ 9034.286884] task_work_run+0xc6/0x110 [ 9034.286885] exit_to_usermode_loop+0x1dd/0x1f0 [ 9034.286887] do_syscall_64+0x3d6/0x440 [ 9034.286888] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9034.286889] RIP: 0033:0x7ff54af45cd8 [ 9034.286891] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9034.286893] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9034.286896] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 9034.286897] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 9034.286899] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 9034.286901] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 9034.286902] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9034.290375] NMI backtrace for cpu 0 [ 9034.915932] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9034.926937] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9034.934242] Call Trace: [ 9034.935934] [ 9034.937268] ? dump_stack+0xbb/0x10e [ 9034.938304] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 9034.940919] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 9034.944313] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 9034.947958] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 9034.950536] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 9034.956312] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 9034.964092] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 9034.971744] ? update_process_times+0x6d/0xc0 [ 9034.976379] ? tick_sched_handle.isra.18+0x27/0x90 [ 9034.985394] ? tick_sched_timer+0x53/0xb0 [ 9034.992383] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9035.000914] ? tick_sched_handle.isra.18+0x90/0x90 [ 9035.013134] ? hrtimer_interrupt+0x128/0x2c0 [ 9035.022326] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9035.027850] ? apic_timer_interrupt+0xf/0x20 [ 9035.036199] [ 9035.040245] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9035.048419] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9035.055141] ? do_raw_spin_lock+0x177/0x190 [ 9035.057943] ? _raw_spin_lock+0x21/0x30 [ 9035.061944] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9035.068423] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 9035.078865] ? mdc_commit_open+0x91/0xb0 [mdc] [ 9035.082935] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9035.090848] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9035.102326] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 9035.104398] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9035.117372] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9035.128219] ? do_wait_intr+0xf0/0xf0 [ 9035.133517] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9035.140036] ? kthread+0x1d1/0x200 [ 9035.141235] ? set_kthread_struct+0x70/0x70 [ 9035.154458] ? ret_from_fork+0x1f/0x30 [ 9036.064306] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 9036.070955] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9036.107025] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9036.117967] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9036.137861] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9036.146466] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9036.148538] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9036.166808] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9036.170276] RAX: 0000000000006788 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 9036.177436] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 9036.187240] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 9036.207574] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 9036.227998] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 9036.230445] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 9036.243225] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9036.251509] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 9036.260589] Call Trace: [ 9036.265898] [ 9036.267497] ? show_regs.cold.9+0x22/0x2f [ 9036.269804] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9036.274406] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9036.278391] ? watchdog+0x40/0x40 [ 9036.280488] ? hrtimer_interrupt+0x128/0x2c0 [ 9036.282678] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9036.294322] ? apic_timer_interrupt+0xf/0x20 [ 9036.297875] [ 9036.300576] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9036.302474] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9036.304459] do_raw_spin_lock+0x177/0x190 [ 9036.315577] _raw_spin_lock+0x21/0x30 [ 9036.319576] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9036.327869] process_one_work+0x2c8/0x700 [ 9036.334522] worker_thread+0x296/0x6e0 [ 9036.337845] ? rescuer_thread+0x570/0x570 [ 9036.339212] kthread+0x1d1/0x200 [ 9036.348908] ? set_kthread_struct+0x70/0x70 [ 9036.353079] ret_from_fork+0x1f/0x30 [ 9048.032608] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7089s! [ 9048.042034] Showing busy workqueues and worker pools: [ 9048.046222] workqueue events: flags=0x0 [ 9048.048359] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9048.052182] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 9048.059271] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 9048.061802] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 9048.068276] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4] [ 9048.072653] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9048.072742] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9048.072749] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9048.072753] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9048.072768] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9048.072771] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9048.072776] RAX: 0000000000006c07 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 9048.072778] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 9048.072781] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9048.072783] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 9048.072785] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 9048.072789] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 9048.072792] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9048.072794] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 9048.072804] Call Trace: [ 9048.072810] [ 9048.072816] ? show_regs.cold.9+0x22/0x2f [ 9048.072825] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9048.072832] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9048.072836] ? watchdog+0x40/0x40 [ 9048.072841] ? hrtimer_interrupt+0x128/0x2c0 [ 9048.072844] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9048.072852] ? apic_timer_interrupt+0xf/0x20 [ 9048.072857] [ 9048.072860] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9048.072867] do_raw_spin_lock+0x177/0x190 [ 9048.072872] _raw_spin_lock+0x21/0x30 [ 9048.072877] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9048.077136] , osc_grant_work_handler [osc], key_garbage_collector [ 9048.077248] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9048.077261] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 9048.108905] ? do_raw_spin_unlock+0x75/0x190 [ 9048.108927] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9048.109220] ? do_wait_intr+0xf0/0xf0 [ 9048.109229] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9048.109428] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9048.110099] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9048.115747] [ 9048.115786] workqueue events_long: flags=0x0 [ 9048.115793] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9048.115807] pending: lu_objects_flush [obdclass] [ 9048.115993] workqueue events_unbound: flags=0x2 [ 9048.115996] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 9048.116008] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9048.116045] workqueue events_power_efficient: flags=0x80 [ 9048.116049] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9048.116055] pending: neigh_periodic_work [ 9048.116063] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9048.116070] pending: do_cache_clean, check_lifetime [ 9048.116085] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9048.116092] pending: neigh_periodic_work [ 9048.116102] workqueue writeback: flags=0x4a [ 9048.116105] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9048.116110] pending: wb_workfn, wb_workfn [ 9048.116124] workqueue xprtiod: flags=0xa [ 9048.116127] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9048.116133] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 9048.116154] workqueue ipv6_addrconf: flags=0x40008 [ 9048.116157] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9048.116164] pending: addrconf_verify_work [ 9048.116181] workqueue knbd0-recv: flags=0x1a [ 9048.116184] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9048.116192] in-flight: 657:recv_work [ 9048.116202] workqueue ptlrpc_pinger: flags=0x2 [ 9048.116204] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9048.116210] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 9048.120563] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9048.120725] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9048.120766] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9048.120800] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9048.120826] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9048.120906] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9048.121219] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9048.121249] lmv_intent_open+0x338/0x1150 [lmv] [ 9048.121276] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9048.121365] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9048.121460] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9048.121483] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9048.121645] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9048.121752] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9048.121840] ? do_raw_spin_unlock+0x75/0x190 [ 9048.121850] ? _raw_spin_unlock+0x12/0x30 [ 9048.121858] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9048.121892] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9048.121915] ll_atomic_open+0x36a/0xd70 [lustre] [ 9048.122006] lookup_open+0x356/0xb40 [ 9048.122018] do_last+0x313/0xfc0 [ 9048.122024] ? nd_jump_root+0xe5/0x160 [ 9048.122029] ? path_init+0x437/0x520 [ 9048.122034] path_openat+0xf7/0x500 [ 9048.122040] ? do_raw_spin_unlock+0x75/0x190 [ 9048.122045] do_filp_open+0x99/0x140 [ 9048.122053] ? getname_flags+0x6e/0x330 [ 9048.122058] ? __check_object_size+0xff/0x256 [ 9048.122067] ? do_raw_spin_unlock+0x75/0x190 [ 9048.122071] ? _raw_spin_unlock+0x12/0x30 [ 9048.122076] do_sys_openat2+0x2b4/0x410 [ 9048.122080] do_sys_open+0x73/0xa0 [ 9048.122085] __x64_sys_openat+0x24/0x30 [ 9048.122089] do_syscall_64+0xc1/0x440 [ 9048.122097] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9048.122103] RIP: 0033:0x7ff54af45332 [ 9048.122113] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9048.122118] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9048.122125] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 9048.122129] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 9048.122218] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 9048.122221] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 9048.122224] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 9048.432060] [ 9048.432741] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7089s workers=4 idle: 20831 12642 28859 [ 9048.441339] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9038s workers=2 idle: 659 [ 9060.058437] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 9060.066135] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9060.112980] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9060.120489] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9060.132767] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9060.140640] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9060.161061] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9060.172066] RAX: 0000000000007f0e RBX: ffff96848811cae0 RCX: 0000000000000001 [ 9060.181957] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 9060.184153] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9060.186754] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 9060.189363] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 9060.192124] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 9060.194895] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9060.203690] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 9060.211112] Call Trace: [ 9060.216098] [ 9060.217000] ? show_regs.cold.9+0x22/0x2f [ 9060.219162] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9060.224015] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9060.228212] ? watchdog+0x40/0x40 [ 9060.231063] ? hrtimer_interrupt+0x128/0x2c0 [ 9060.235812] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9060.238442] ? apic_timer_interrupt+0xf/0x20 [ 9060.240953] [ 9060.241748] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9060.244069] do_raw_spin_lock+0x177/0x190 [ 9060.247823] _raw_spin_lock+0x21/0x30 [ 9060.250238] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9060.255939] ? do_raw_spin_unlock+0x75/0x190 [ 9060.259166] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9060.263947] ? do_wait_intr+0xf0/0xf0 [ 9060.266265] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9060.271794] mdc_close+0x53b/0xd70 [mdc] [ 9060.277816] lmv_close+0x18d/0x490 [lmv] [ 9060.279321] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 9060.282267] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9060.284300] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9060.296045] ll_md_close+0x274/0x800 [lustre] [ 9060.301210] ll_file_release+0x128/0xc50 [lustre] [ 9060.305191] __fput+0x10d/0x440 [ 9060.307857] ____fput+0x12/0x20 [ 9060.310895] task_work_run+0xc6/0x110 [ 9060.313952] exit_to_usermode_loop+0x1dd/0x1f0 [ 9060.317282] do_syscall_64+0x3d6/0x440 [ 9060.320296] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9060.325258] RIP: 0033:0x7ff54af45cd8 [ 9060.330305] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9060.346449] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9060.356755] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 9060.359901] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 9060.364625] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 9060.373858] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 9060.383803] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9064.060159] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 9064.066762] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9064.105061] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9064.114524] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9064.122218] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9064.127982] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9064.133867] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9064.154385] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9064.180651] RAX: 0000000000007d82 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 9064.198559] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 9064.207077] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 9064.234722] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 9064.237846] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 9064.265007] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 9064.277519] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9064.298227] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 9064.321652] Call Trace: [ 9064.329946] [ 9064.335713] ? show_regs.cold.9+0x22/0x2f [ 9064.340883] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9064.352191] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9064.355053] ? watchdog+0x40/0x40 [ 9064.365150] ? hrtimer_interrupt+0x128/0x2c0 [ 9064.378254] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9064.390356] ? apic_timer_interrupt+0xf/0x20 [ 9064.392223] [ 9064.409117] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9064.424838] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9064.427606] do_raw_spin_lock+0x177/0x190 [ 9064.433789] _raw_spin_lock+0x21/0x30 [ 9064.435802] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9064.440195] process_one_work+0x2c8/0x700 [ 9064.442295] worker_thread+0x296/0x6e0 [ 9064.444166] ? rescuer_thread+0x570/0x570 [ 9064.446291] kthread+0x1d1/0x200 [ 9064.448047] ? set_kthread_struct+0x70/0x70 [ 9064.449452] ret_from_fork+0x1f/0x30 [ 9076.055189] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 9076.059141] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9076.066598] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 9076.084305] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9076.084382] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9076.084393] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9076.084408] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9076.084416] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9076.084424] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 9076.084426] RDX: 0000000000002408 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 9076.084428] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 9076.084430] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 9076.084435] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9076.084438] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 9076.084441] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9076.084443] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 9076.084449] Call Trace: [ 9076.084454] [ 9076.084464] ? show_regs.cold.9+0x22/0x2f [ 9076.084475] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9076.084481] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9076.084486] ? watchdog+0x40/0x40 [ 9076.084490] ? hrtimer_interrupt+0x128/0x2c0 [ 9076.084494] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9076.084503] ? apic_timer_interrupt+0xf/0x20 [ 9076.084507] [ 9076.084510] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9076.084515] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9076.084520] do_raw_spin_lock+0x177/0x190 [ 9076.084525] _raw_spin_lock+0x21/0x30 [ 9076.084531] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9076.084890] obd_mod_free+0x46/0x140 [ptlrpc] [ 9076.085095] mdc_commit_open+0x91/0xb0 [mdc] [ 9076.085135] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9076.089895] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9076.089969] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9076.089976] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9076.089978] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9076.089991] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9076.089995] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9076.089998] RAX: 0000000000007382 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 9076.090001] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 9076.090002] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9076.090005] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 9076.090007] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 9076.090010] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 9076.090014] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9076.090017] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 9076.090582] Call Trace: [ 9076.090816] [ 9076.090826] ? show_regs.cold.9+0x22/0x2f [ 9076.090841] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9076.090847] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9076.090852] ? watchdog+0x40/0x40 [ 9076.090857] ? hrtimer_interrupt+0x128/0x2c0 [ 9076.090860] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9076.090873] ? apic_timer_interrupt+0xf/0x20 [ 9076.090878] [ 9076.090881] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9076.090889] do_raw_spin_lock+0x177/0x190 [ 9076.090894] _raw_spin_lock+0x21/0x30 [ 9076.090900] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9076.091191] ? do_raw_spin_unlock+0x75/0x190 [ 9076.091198] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9076.091930] ? do_wait_intr+0xf0/0xf0 [ 9076.091940] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9076.100002] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9076.100685] after_reply+0xe2c/0x1780 [ptlrpc] [ 9076.100947] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9076.106212] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9076.106523] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9076.106758] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9076.106880] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9076.106932] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9076.106965] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9076.106995] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9076.107087] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9076.107300] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9076.107328] lmv_intent_open+0x338/0x1150 [lmv] [ 9076.107360] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9076.107465] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9076.107555] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9076.107581] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9076.107667] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9076.107750] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9076.107837] ? do_raw_spin_unlock+0x75/0x190 [ 9076.107845] ? _raw_spin_unlock+0x12/0x30 [ 9076.107851] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9076.107879] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9076.107899] ll_atomic_open+0x36a/0xd70 [lustre] [ 9076.107992] lookup_open+0x356/0xb40 [ 9076.108002] do_last+0x313/0xfc0 [ 9076.108007] ? nd_jump_root+0xe5/0x160 [ 9076.108012] ? path_init+0x437/0x520 [ 9076.108016] path_openat+0xf7/0x500 [ 9076.108022] ? do_raw_spin_unlock+0x75/0x190 [ 9076.108027] do_filp_open+0x99/0x140 [ 9076.108034] ? getname_flags+0x6e/0x330 [ 9076.108039] ? __check_object_size+0xff/0x256 [ 9076.108303] ? do_raw_spin_unlock+0x75/0x190 [ 9076.108308] ? _raw_spin_unlock+0x12/0x30 [ 9076.108313] do_sys_openat2+0x2b4/0x410 [ 9076.108318] do_sys_open+0x73/0xa0 [ 9076.108322] __x64_sys_openat+0x24/0x30 [ 9076.108326] do_syscall_64+0xc1/0x440 [ 9076.108333] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9076.108339] RIP: 0033:0x7ff54af45332 [ 9076.108347] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9076.108352] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9076.108357] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 9076.108360] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 9076.108362] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 9076.108364] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 9076.108366] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 9076.596896] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9076.605593] ? do_wait_intr+0xf0/0xf0 [ 9076.608750] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9076.615027] kthread+0x1d1/0x200 [ 9076.618112] ? set_kthread_struct+0x70/0x70 [ 9076.622734] ret_from_fork+0x1f/0x30 [ 9078.751089] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7119s! [ 9078.756495] Showing busy workqueues and worker pools: [ 9078.759625] workqueue events: flags=0x0 [ 9078.762409] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9078.765275] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 9078.772629] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 9078.778161] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 9078.787700] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9078.793105] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 9078.797447] workqueue events_long: flags=0x0 [ 9078.801110] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9078.805317] pending: lu_objects_flush [obdclass] [ 9078.808857] workqueue events_unbound: flags=0x2 [ 9078.812592] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 9078.818052] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9078.824779] workqueue events_power_efficient: flags=0x80 [ 9078.828334] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9078.832585] pending: neigh_periodic_work [ 9078.835465] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9078.839743] pending: do_cache_clean, check_lifetime [ 9078.844538] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9078.849494] pending: neigh_periodic_work [ 9078.853927] workqueue writeback: flags=0x4a [ 9078.857178] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9078.859843] pending: wb_workfn, wb_workfn [ 9078.862587] workqueue xprtiod: flags=0xa [ 9078.865688] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9078.870294] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 9078.876529] workqueue ipv6_addrconf: flags=0x40008 [ 9078.880668] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9078.884512] pending: addrconf_verify_work [ 9078.887680] workqueue knbd0-recv: flags=0x1a [ 9078.890476] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9078.894693] in-flight: 657:recv_work [ 9078.897531] workqueue ptlrpc_pinger: flags=0x2 [ 9078.899791] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9078.905519] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 9078.910177] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7120s workers=4 idle: 20831 12642 28859 [ 9078.916303] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9069s workers=2 idle: 659 [ 9088.065502] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 9088.075130] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9088.177444] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9088.196669] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9088.218822] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9088.222979] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9088.247286] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9088.255102] RAX: 0000000000006b78 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 9088.265677] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 9088.280552] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9088.288853] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 9088.301475] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 9088.310428] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 9088.316789] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9088.322093] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 9088.331351] Call Trace: [ 9088.332154] [ 9088.332760] ? show_regs.cold.9+0x22/0x2f [ 9088.345076] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9088.351045] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9088.357783] ? watchdog+0x40/0x40 [ 9088.366649] ? hrtimer_interrupt+0x128/0x2c0 [ 9088.368206] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9088.374467] ? apic_timer_interrupt+0xf/0x20 [ 9088.381498] [ 9088.385496] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9088.392652] do_raw_spin_lock+0x177/0x190 [ 9088.402065] _raw_spin_lock+0x21/0x30 [ 9088.407852] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9088.418970] ? do_raw_spin_unlock+0x75/0x190 [ 9088.425015] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9088.430480] ? do_wait_intr+0xf0/0xf0 [ 9088.436330] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9088.443746] mdc_close+0x53b/0xd70 [mdc] [ 9088.448000] lmv_close+0x18d/0x490 [lmv] [ 9088.451874] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 9088.455392] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9088.457865] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9088.461383] ll_md_close+0x274/0x800 [lustre] [ 9088.463569] ll_file_release+0x128/0xc50 [lustre] [ 9088.465246] __fput+0x10d/0x440 [ 9088.466795] ____fput+0x12/0x20 [ 9088.467910] task_work_run+0xc6/0x110 [ 9088.469174] exit_to_usermode_loop+0x1dd/0x1f0 [ 9088.472367] do_syscall_64+0x3d6/0x440 [ 9088.476888] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9088.492480] RIP: 0033:0x7ff54af45cd8 [ 9088.496005] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9088.522489] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9088.536804] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 9088.546902] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 9088.557932] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 9088.570460] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 9088.582837] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9092.064184] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 9092.071391] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9092.097896] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9092.106844] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9092.116467] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9092.124995] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9092.128717] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9092.148345] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9092.152246] RAX: 0000000000007f0c RBX: ffff96848811cae0 RCX: 0000000000000001 [ 9092.154777] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 9092.158631] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 9092.160952] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 9092.163876] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 9092.167780] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 9092.170879] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9092.172616] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 9092.175437] Call Trace: [ 9092.176765] [ 9092.177389] ? show_regs.cold.9+0x22/0x2f [ 9092.178517] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9092.182758] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9092.183903] ? watchdog+0x40/0x40 [ 9092.184828] ? hrtimer_interrupt+0x128/0x2c0 [ 9092.186482] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9092.187802] ? apic_timer_interrupt+0xf/0x20 [ 9092.189198] [ 9092.189867] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9092.191627] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9092.192947] do_raw_spin_lock+0x177/0x190 [ 9092.195199] _raw_spin_lock+0x21/0x30 [ 9092.196315] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9092.200527] process_one_work+0x2c8/0x700 [ 9092.202522] worker_thread+0x296/0x6e0 [ 9092.204490] ? rescuer_thread+0x570/0x570 [ 9092.205820] kthread+0x1d1/0x200 [ 9092.206801] ? set_kthread_struct+0x70/0x70 [ 9092.208380] ret_from_fork+0x1f/0x30 [ 9104.056852] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 9104.061275] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9104.063078] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 9104.102350] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9104.102365] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9104.102368] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9104.102771] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9104.102777] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9104.102783] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 9104.102786] RDX: 0000000000004588 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 9104.102788] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 9104.102790] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 9104.102792] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9104.102795] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 9104.102797] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9104.102800] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 9104.102807] Call Trace: [ 9104.102814] [ 9104.102820] ? show_regs.cold.9+0x22/0x2f [ 9104.102842] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9104.102850] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9104.102854] ? watchdog+0x40/0x40 [ 9104.102859] ? hrtimer_interrupt+0x128/0x2c0 [ 9104.102863] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9104.102873] ? apic_timer_interrupt+0xf/0x20 [ 9104.102877] [ 9104.102880] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9104.102884] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9104.102889] do_raw_spin_lock+0x177/0x190 [ 9104.102893] _raw_spin_lock+0x21/0x30 [ 9104.102900] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9104.103497] obd_mod_free+0x46/0x140 [ptlrpc] [ 9104.103691] mdc_commit_open+0x91/0xb0 [mdc] [ 9104.103729] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9104.110055] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9104.110371] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9104.110380] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9104.110383] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9104.110463] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9104.110468] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9104.110472] RAX: 000000000000480a RBX: ffff96848811cae0 RCX: 0000000000000001 [ 9104.110474] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 9104.110476] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9104.110479] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 9104.110481] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 9104.110484] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 9104.110488] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9104.110490] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 9104.110498] Call Trace: [ 9104.110504] [ 9104.110510] ? show_regs.cold.9+0x22/0x2f [ 9104.110583] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9104.110591] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9104.110595] ? watchdog+0x40/0x40 [ 9104.110712] ? hrtimer_interrupt+0x128/0x2c0 [ 9104.110717] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9104.110726] ? apic_timer_interrupt+0xf/0x20 [ 9104.110730] [ 9104.110733] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9104.110740] do_raw_spin_lock+0x177/0x190 [ 9104.110745] _raw_spin_lock+0x21/0x30 [ 9104.110749] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9104.111736] ? do_raw_spin_unlock+0x75/0x190 [ 9104.111744] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9104.111963] ? do_wait_intr+0xf0/0xf0 [ 9104.111972] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9104.112197] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9104.112432] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9104.112647] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9104.112785] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9104.112823] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9104.112853] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9104.112880] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9104.112964] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9104.113192] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9104.113223] lmv_intent_open+0x338/0x1150 [lmv] [ 9104.113253] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9104.113339] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9104.113421] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9104.113443] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9104.113530] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9104.113610] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9104.121198] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9104.121564] after_reply+0xe2c/0x1780 [ptlrpc] [ 9104.121752] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9104.122172] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9104.129336] ? do_raw_spin_unlock+0x75/0x190 [ 9104.129355] ? _raw_spin_unlock+0x12/0x30 [ 9104.129362] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9104.129459] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9104.129482] ll_atomic_open+0x36a/0xd70 [lustre] [ 9104.133225] ? do_wait_intr+0xf0/0xf0 [ 9104.150019] lookup_open+0x356/0xb40 [ 9104.150039] do_last+0x313/0xfc0 [ 9104.150044] ? nd_jump_root+0xe5/0x160 [ 9104.150049] ? path_init+0x437/0x520 [ 9104.150054] path_openat+0xf7/0x500 [ 9104.150145] ? do_raw_spin_unlock+0x75/0x190 [ 9104.150153] do_filp_open+0x99/0x140 [ 9104.150160] ? getname_flags+0x6e/0x330 [ 9104.150165] ? __check_object_size+0xff/0x256 [ 9104.150171] ? do_raw_spin_unlock+0x75/0x190 [ 9104.150175] ? _raw_spin_unlock+0x12/0x30 [ 9104.150181] do_sys_openat2+0x2b4/0x410 [ 9104.150185] do_sys_open+0x73/0xa0 [ 9104.150189] __x64_sys_openat+0x24/0x30 [ 9104.150193] do_syscall_64+0xc1/0x440 [ 9104.150198] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9104.150203] RIP: 0033:0x7ff54af45332 [ 9104.150211] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9104.150215] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9104.150219] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 9104.150221] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 9104.150224] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 9104.150226] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 9104.150228] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 9104.643709] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9104.647533] kthread+0x1d1/0x200 [ 9104.649524] ? set_kthread_struct+0x70/0x70 [ 9104.652097] ret_from_fork+0x1f/0x30 [ 9108.959133] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7150s! [ 9108.970777] Showing busy workqueues and worker pools: [ 9108.976288] workqueue events: flags=0x0 [ 9108.982639] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9108.990961] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 9108.999151] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 9109.005270] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 9109.017738] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9109.026582] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 9109.033704] workqueue events_long: flags=0x0 [ 9109.040868] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9109.052789] pending: lu_objects_flush [obdclass] [ 9109.060727] workqueue events_unbound: flags=0x2 [ 9109.067815] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 9109.084369] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9109.102773] workqueue events_power_efficient: flags=0x80 [ 9109.117305] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9109.123219] pending: neigh_periodic_work [ 9109.134771] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9109.149185] pending: do_cache_clean, check_lifetime [ 9109.157532] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9109.164953] pending: neigh_periodic_work [ 9109.176365] workqueue writeback: flags=0x4a [ 9109.184348] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9109.195699] pending: wb_workfn, wb_workfn [ 9109.203459] workqueue xprtiod: flags=0xa [ 9109.205693] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9109.212441] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 9109.217131] workqueue ipv6_addrconf: flags=0x40008 [ 9109.220461] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9109.225394] pending: addrconf_verify_work [ 9109.231984] workqueue knbd0-recv: flags=0x1a [ 9109.237250] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9109.242526] in-flight: 657:recv_work [ 9109.247392] workqueue ptlrpc_pinger: flags=0x2 [ 9109.249464] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9109.253417] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 9109.256646] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7150s workers=4 idle: 20831 12642 28859 [ 9109.260913] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9099s workers=2 idle: 659 [ 9116.058058] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 9116.070694] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9116.132677] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9116.142047] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9116.149483] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9116.153460] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9116.166702] RSP: 0018:ffffaf9fc1543ae0 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 9116.172331] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 9116.178139] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 9116.183441] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9116.188242] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 9116.195442] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 9116.199579] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 9116.205669] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9116.209329] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 9116.213364] Call Trace: [ 9116.214285] [ 9116.217696] ? show_regs.cold.9+0x22/0x2f [ 9116.220061] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9116.223570] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9116.228422] ? watchdog+0x40/0x40 [ 9116.231939] ? hrtimer_interrupt+0x128/0x2c0 [ 9116.237069] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9116.246491] ? apic_timer_interrupt+0xf/0x20 [ 9116.249230] [ 9116.250962] ? native_safe_halt+0x2e/0x50 [ 9116.257053] kvm_wait+0xcd/0xe0 [ 9116.261014] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9116.262805] do_raw_spin_lock+0x177/0x190 [ 9116.273655] _raw_spin_lock+0x21/0x30 [ 9116.274959] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9116.286218] ? do_raw_spin_unlock+0x75/0x190 [ 9116.287791] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9116.293522] ? do_wait_intr+0xf0/0xf0 [ 9116.298602] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9116.303090] mdc_close+0x53b/0xd70 [mdc] [ 9116.305227] lmv_close+0x18d/0x490 [lmv] [ 9116.308055] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 9116.310134] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9116.314057] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9116.317127] ll_md_close+0x274/0x800 [lustre] [ 9116.319230] ll_file_release+0x128/0xc50 [lustre] [ 9116.324099] __fput+0x10d/0x440 [ 9116.326719] ____fput+0x12/0x20 [ 9116.328435] task_work_run+0xc6/0x110 [ 9116.330633] exit_to_usermode_loop+0x1dd/0x1f0 [ 9116.333183] do_syscall_64+0x3d6/0x440 [ 9116.335991] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9116.339521] RIP: 0033:0x7ff54af45cd8 [ 9116.341790] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9116.351844] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9116.356989] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 9116.360572] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 9116.363924] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 9116.369879] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 9116.373509] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9120.060024] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 9120.065068] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9120.107560] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9120.119149] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9120.130293] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9120.139423] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9120.141569] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9120.155996] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9120.159636] RAX: 0000000000007bf7 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 9120.164768] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 9120.174409] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 9120.178031] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 9120.183038] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 9120.186293] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 9120.190184] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9120.194353] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 9120.203736] Call Trace: [ 9120.208496] [ 9120.217170] ? show_regs.cold.9+0x22/0x2f [ 9120.222645] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9120.227737] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9120.229675] ? watchdog+0x40/0x40 [ 9120.233233] ? hrtimer_interrupt+0x128/0x2c0 [ 9120.238124] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9120.241510] ? apic_timer_interrupt+0xf/0x20 [ 9120.245374] [ 9120.247363] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9120.249570] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9120.255203] do_raw_spin_lock+0x177/0x190 [ 9120.258705] _raw_spin_lock+0x21/0x30 [ 9120.260604] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9120.265087] process_one_work+0x2c8/0x700 [ 9120.270338] worker_thread+0x296/0x6e0 [ 9120.276492] ? rescuer_thread+0x570/0x570 [ 9120.278047] kthread+0x1d1/0x200 [ 9120.285240] ? set_kthread_struct+0x70/0x70 [ 9120.290304] ret_from_fork+0x1f/0x30 [ 9132.060033] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 9132.065643] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9132.095573] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9132.104335] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9132.116231] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9132.119230] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9132.130753] RSP: 0000:ffffaf9fc15234f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9132.137485] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 9132.146224] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 9132.151717] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9132.159234] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 9132.163866] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 9132.168836] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 9132.174869] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9132.181833] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 9132.197470] Call Trace: [ 9132.198914] [ 9132.202500] ? show_regs.cold.9+0x22/0x2f [ 9132.209853] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9132.214951] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9132.219120] ? watchdog+0x40/0x40 [ 9132.225684] ? hrtimer_interrupt+0x128/0x2c0 [ 9132.230655] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9132.237593] ? apic_timer_interrupt+0xf/0x20 [ 9132.241820] [ 9132.244064] ? native_safe_halt+0x2e/0x50 [ 9132.249585] kvm_wait+0xcd/0xe0 [ 9132.253676] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9132.260858] do_raw_spin_lock+0x177/0x190 [ 9132.262571] _raw_spin_lock+0x21/0x30 [ 9132.267808] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9132.274109] ? do_raw_spin_unlock+0x75/0x190 [ 9132.278450] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9132.281125] ? do_wait_intr+0xf0/0xf0 [ 9132.282261] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9132.286245] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9132.292588] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9132.301870] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9132.309211] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9132.316035] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9132.320020] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9132.323775] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9132.349536] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9132.362581] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9132.371588] lmv_intent_open+0x338/0x1150 [lmv] [ 9132.381718] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9132.393313] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9132.402022] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9132.411393] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9132.413190] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9132.417996] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9132.420601] ? do_raw_spin_unlock+0x75/0x190 [ 9132.423549] ? _raw_spin_unlock+0x12/0x30 [ 9132.425616] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9132.428358] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9132.432641] ll_atomic_open+0x36a/0xd70 [lustre] [ 9132.438749] lookup_open+0x356/0xb40 [ 9132.441897] do_last+0x313/0xfc0 [ 9132.444636] ? nd_jump_root+0xe5/0x160 [ 9132.449588] ? path_init+0x437/0x520 [ 9132.452069] path_openat+0xf7/0x500 [ 9132.454582] ? do_raw_spin_unlock+0x75/0x190 [ 9132.459208] do_filp_open+0x99/0x140 [ 9132.461717] ? getname_flags+0x6e/0x330 [ 9132.463014] ? __check_object_size+0xff/0x256 [ 9132.467009] ? do_raw_spin_unlock+0x75/0x190 [ 9132.469801] ? _raw_spin_unlock+0x12/0x30 [ 9132.473432] do_sys_openat2+0x2b4/0x410 [ 9132.477237] do_sys_open+0x73/0xa0 [ 9132.479888] __x64_sys_openat+0x24/0x30 [ 9132.483477] do_syscall_64+0xc1/0x440 [ 9132.486037] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9132.490636] RIP: 0033:0x7ff54af45332 [ 9132.493917] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9132.505984] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9132.514037] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 9132.519209] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 9132.524565] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 9132.529208] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 9132.533927] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 9136.059116] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 9136.065792] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9136.116222] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9136.132555] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9136.144118] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9136.150816] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9136.161484] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9136.164282] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 9136.166291] RDX: 0000000000007c2f RSI: 0000000000000003 RDI: ffff96848811cae0 [ 9136.167901] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 9136.169933] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 9136.171912] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9136.174266] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 9136.181382] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9136.183409] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 9136.185437] Call Trace: [ 9136.186307] [ 9136.187151] ? show_regs.cold.9+0x22/0x2f [ 9136.188776] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9136.190378] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9136.193326] ? watchdog+0x40/0x40 [ 9136.194379] ? hrtimer_interrupt+0x128/0x2c0 [ 9136.199204] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9136.202880] ? apic_timer_interrupt+0xf/0x20 [ 9136.204929] [ 9136.205484] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9136.207445] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9136.209242] do_raw_spin_lock+0x177/0x190 [ 9136.211238] _raw_spin_lock+0x21/0x30 [ 9136.212612] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9136.214264] obd_mod_free+0x46/0x140 [ptlrpc] [ 9136.216211] mdc_commit_open+0x91/0xb0 [mdc] [ 9136.218881] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9136.220690] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9136.226506] after_reply+0xe2c/0x1780 [ptlrpc] [ 9136.231512] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9136.236616] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9136.239857] ? do_wait_intr+0xf0/0xf0 [ 9136.240981] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9136.252055] kthread+0x1d1/0x200 [ 9136.254516] ? set_kthread_struct+0x70/0x70 [ 9136.260083] ret_from_fork+0x1f/0x30 [ 9139.683515] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7180s! [ 9139.687672] Showing busy workqueues and worker pools: [ 9139.694984] workqueue events: flags=0x0 [ 9139.699333] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9139.703456] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 9139.707040] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 9139.709785] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 9139.715438] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9139.721520] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 9139.729255] workqueue events_long: flags=0x0 [ 9139.735018] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9139.738721] pending: lu_objects_flush [obdclass] [ 9139.741913] workqueue events_unbound: flags=0x2 [ 9139.746654] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 9139.753304] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9139.765498] workqueue events_power_efficient: flags=0x80 [ 9139.773632] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9139.781882] pending: neigh_periodic_work [ 9139.784668] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9139.793243] pending: do_cache_clean, check_lifetime [ 9139.798138] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9139.803571] pending: neigh_periodic_work [ 9139.806384] workqueue writeback: flags=0x4a [ 9139.811694] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9139.817885] pending: wb_workfn, wb_workfn [ 9139.822929] workqueue xprtiod: flags=0xa [ 9139.826462] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9139.833492] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 9139.838656] workqueue ipv6_addrconf: flags=0x40008 [ 9139.840921] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9139.845947] pending: addrconf_verify_work [ 9139.851518] workqueue knbd0-recv: flags=0x1a [ 9139.857715] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9139.872968] in-flight: 657:recv_work [ 9139.879426] workqueue ptlrpc_pinger: flags=0x2 [ 9139.881896] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9139.892731] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 9139.898884] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7181s workers=4 idle: 20831 12642 28859 [ 9139.909367] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9130s workers=2 idle: 659 [ 9144.058028] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [ 9144.062421] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9144.117262] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9144.130287] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9144.137827] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9144.143050] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9144.166508] RSP: 0018:ffffaf9fc1543ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9144.174384] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 9144.177908] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 9144.182429] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9144.187751] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 9144.192928] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 9144.197430] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 9144.203816] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9144.209904] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 9144.218064] Call Trace: [ 9144.219898] [ 9144.221285] ? show_regs.cold.9+0x22/0x2f [ 9144.226324] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9144.229877] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9144.234310] ? watchdog+0x40/0x40 [ 9144.237683] ? hrtimer_interrupt+0x128/0x2c0 [ 9144.244127] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9144.248361] ? apic_timer_interrupt+0xf/0x20 [ 9144.253762] [ 9144.256256] ? native_safe_halt+0x2e/0x50 [ 9144.259322] kvm_wait+0xcd/0xe0 [ 9144.265126] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9144.270846] do_raw_spin_lock+0x177/0x190 [ 9144.273132] _raw_spin_lock+0x21/0x30 [ 9144.274605] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9144.278615] ? do_raw_spin_unlock+0x75/0x190 [ 9144.284396] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9144.289491] ? do_wait_intr+0xf0/0xf0 [ 9144.291381] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9144.294491] mdc_close+0x53b/0xd70 [mdc] [ 9144.295984] lmv_close+0x18d/0x490 [lmv] [ 9144.297742] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 9144.300417] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9144.303523] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9144.305794] ll_md_close+0x274/0x800 [lustre] [ 9144.310422] ll_file_release+0x128/0xc50 [lustre] [ 9144.312716] __fput+0x10d/0x440 [ 9144.313921] ____fput+0x12/0x20 [ 9144.315527] task_work_run+0xc6/0x110 [ 9144.318046] exit_to_usermode_loop+0x1dd/0x1f0 [ 9144.320347] do_syscall_64+0x3d6/0x440 [ 9144.322322] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9144.334009] RIP: 0033:0x7ff54af45cd8 [ 9144.336828] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9144.356281] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9144.361948] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 9144.369995] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 9144.379406] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 9144.393048] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 9144.397665] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9148.059010] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 9148.062310] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9148.086651] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9148.095798] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9148.103161] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9148.108921] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9148.111305] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9148.126827] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9148.129243] RAX: 0000000000004dd5 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 9148.138465] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 9148.149291] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 9148.162683] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 9148.168873] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 9148.175784] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 9148.183679] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9148.186864] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 9148.190253] Call Trace: [ 9148.191140] [ 9148.191968] ? show_regs.cold.9+0x22/0x2f [ 9148.193464] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9148.196118] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9148.198727] ? watchdog+0x40/0x40 [ 9148.199783] ? hrtimer_interrupt+0x128/0x2c0 [ 9148.204820] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9148.206959] ? apic_timer_interrupt+0xf/0x20 [ 9148.213616] [ 9148.214379] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9148.216064] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9148.222712] do_raw_spin_lock+0x177/0x190 [ 9148.227949] _raw_spin_lock+0x21/0x30 [ 9148.229700] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9148.232151] process_one_work+0x2c8/0x700 [ 9148.237934] worker_thread+0x296/0x6e0 [ 9148.240125] ? rescuer_thread+0x570/0x570 [ 9148.242096] kthread+0x1d1/0x200 [ 9148.244832] ? set_kthread_struct+0x70/0x70 [ 9148.251321] ret_from_fork+0x1f/0x30 [ 9160.060015] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 9160.064487] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9160.103641] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9160.113766] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9160.119355] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9160.124175] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9160.134479] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9160.138360] RAX: 00000000000043a7 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 9160.142508] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 9160.146812] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9160.149457] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 9160.153989] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 9160.156472] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 9160.160249] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9160.163410] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 9160.166963] Call Trace: [ 9160.167944] [ 9160.168942] ? show_regs.cold.9+0x22/0x2f [ 9160.171531] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9160.173051] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9160.175833] ? watchdog+0x40/0x40 [ 9160.177178] ? hrtimer_interrupt+0x128/0x2c0 [ 9160.179985] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9160.182934] ? apic_timer_interrupt+0xf/0x20 [ 9160.185264] [ 9160.186225] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9160.188940] do_raw_spin_lock+0x177/0x190 [ 9160.191913] _raw_spin_lock+0x21/0x30 [ 9160.193843] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9160.196864] ? do_raw_spin_unlock+0x75/0x190 [ 9160.198309] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9160.202422] ? do_wait_intr+0xf0/0xf0 [ 9160.204798] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9160.207838] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9160.210318] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9160.213817] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9160.216340] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9160.218829] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9160.221472] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9160.223992] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9160.227080] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9160.233229] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9160.236475] lmv_intent_open+0x338/0x1150 [lmv] [ 9160.239083] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9160.242455] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9160.245720] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9160.248098] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9160.252300] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9160.255485] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9160.258883] ? do_raw_spin_unlock+0x75/0x190 [ 9160.260902] ? _raw_spin_unlock+0x12/0x30 [ 9160.262798] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9160.265053] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9160.267316] ll_atomic_open+0x36a/0xd70 [lustre] [ 9160.269829] lookup_open+0x356/0xb40 [ 9160.271830] do_last+0x313/0xfc0 [ 9160.274323] ? nd_jump_root+0xe5/0x160 [ 9160.278546] ? path_init+0x437/0x520 [ 9160.280205] path_openat+0xf7/0x500 [ 9160.283965] ? do_raw_spin_unlock+0x75/0x190 [ 9160.287135] do_filp_open+0x99/0x140 [ 9160.290336] ? getname_flags+0x6e/0x330 [ 9160.294282] ? __check_object_size+0xff/0x256 [ 9160.296329] ? do_raw_spin_unlock+0x75/0x190 [ 9160.298335] ? _raw_spin_unlock+0x12/0x30 [ 9160.300700] do_sys_openat2+0x2b4/0x410 [ 9160.302308] do_sys_open+0x73/0xa0 [ 9160.303907] __x64_sys_openat+0x24/0x30 [ 9160.305340] do_syscall_64+0xc1/0x440 [ 9160.306496] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9160.310155] RIP: 0033:0x7ff54af45332 [ 9160.311831] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9160.320146] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9160.323400] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 9160.326348] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 9160.329175] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 9160.332744] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 9160.336218] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 9164.055016] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 9164.062039] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9164.116832] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9164.134672] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9164.143709] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9164.153916] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9164.182726] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9164.186197] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 9164.195658] RDX: 0000000000004988 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 9164.204884] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 9164.210332] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 9164.223648] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9164.225861] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 9164.232690] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9164.236715] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 9164.248249] Call Trace: [ 9164.249171] [ 9164.252571] ? show_regs.cold.9+0x22/0x2f [ 9164.258928] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9164.263281] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9164.268710] ? watchdog+0x40/0x40 [ 9164.275069] ? hrtimer_interrupt+0x128/0x2c0 [ 9164.280670] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9164.283575] ? apic_timer_interrupt+0xf/0x20 [ 9164.286065] [ 9164.289666] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9164.294861] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9164.305002] do_raw_spin_lock+0x177/0x190 [ 9164.311635] _raw_spin_lock+0x21/0x30 [ 9164.315468] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9164.322350] obd_mod_free+0x46/0x140 [ptlrpc] [ 9164.327670] mdc_commit_open+0x91/0xb0 [mdc] [ 9164.336554] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9164.348526] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9164.359500] after_reply+0xe2c/0x1780 [ptlrpc] [ 9164.363843] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9164.371911] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9164.376782] ? do_wait_intr+0xf0/0xf0 [ 9164.382874] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9164.391807] kthread+0x1d1/0x200 [ 9164.397589] ? set_kthread_struct+0x70/0x70 [ 9164.399153] ret_from_fork+0x1f/0x30 [ 9170.399319] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7211s! [ 9170.408058] Showing busy workqueues and worker pools: [ 9170.412633] workqueue events: flags=0x0 [ 9170.418145] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9170.421558] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 9170.426534] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 9170.434165] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 9170.449141] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9170.460195] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 9170.466744] workqueue events_long: flags=0x0 [ 9170.469946] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9170.475176] pending: lu_objects_flush [obdclass] [ 9170.478808] workqueue events_unbound: flags=0x2 [ 9170.481727] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 9170.485734] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9170.492408] workqueue events_power_efficient: flags=0x80 [ 9170.495838] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9170.498148] pending: neigh_periodic_work [ 9170.500408] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9170.516849] pending: do_cache_clean, check_lifetime [ 9170.526020] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9170.532822] pending: neigh_periodic_work [ 9170.537780] workqueue writeback: flags=0x4a [ 9170.542584] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9170.545610] pending: wb_workfn, wb_workfn [ 9170.550126] workqueue xprtiod: flags=0xa [ 9170.557239] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9170.562904] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 9170.566027] workqueue ipv6_addrconf: flags=0x40008 [ 9170.575896] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9170.582369] pending: addrconf_verify_work [ 9170.583912] workqueue knbd0-recv: flags=0x1a [ 9170.585330] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9170.596354] in-flight: 657:recv_work [ 9170.598517] workqueue ptlrpc_pinger: flags=0x2 [ 9170.603700] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9170.610213] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 9170.619224] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7211s workers=4 idle: 20831 12642 28859 [ 9170.628110] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9160s workers=2 idle: 659 [ 9172.062429] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [ 9172.072181] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9172.141804] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9172.149166] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9172.154137] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9172.157220] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9172.166854] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9172.170136] RAX: 00000000000071ad RBX: ffff96848811cae0 RCX: 0000000000000001 [ 9172.172675] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 9172.179191] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9172.184753] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 9172.190030] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 9172.193124] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 9172.196848] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9172.200042] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 9172.204037] Call Trace: [ 9172.205419] [ 9172.206526] ? show_regs.cold.9+0x22/0x2f [ 9172.208912] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9172.211707] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9172.213817] ? watchdog+0x40/0x40 [ 9172.215817] ? hrtimer_interrupt+0x128/0x2c0 [ 9172.218002] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9172.220867] ? apic_timer_interrupt+0xf/0x20 [ 9172.223186] [ 9172.224427] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9172.227593] do_raw_spin_lock+0x177/0x190 [ 9172.230106] _raw_spin_lock+0x21/0x30 [ 9172.232170] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9172.235417] ? do_raw_spin_unlock+0x75/0x190 [ 9172.237836] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9172.240880] ? do_wait_intr+0xf0/0xf0 [ 9172.243003] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9172.246134] mdc_close+0x53b/0xd70 [mdc] [ 9172.248478] lmv_close+0x18d/0x490 [lmv] [ 9172.250683] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 9172.254270] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9172.260371] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9172.267784] ll_md_close+0x274/0x800 [lustre] [ 9172.270260] ll_file_release+0x128/0xc50 [lustre] [ 9172.276084] __fput+0x10d/0x440 [ 9172.277925] ____fput+0x12/0x20 [ 9172.279761] task_work_run+0xc6/0x110 [ 9172.281826] exit_to_usermode_loop+0x1dd/0x1f0 [ 9172.284659] do_syscall_64+0x3d6/0x440 [ 9172.288490] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9172.291758] RIP: 0033:0x7ff54af45cd8 [ 9172.297951] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9172.310122] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9172.314720] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 9172.321942] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 9172.326366] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 9172.330423] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 9172.334077] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9176.060683] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 9176.072978] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9176.118424] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9176.137821] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9176.142092] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9176.149934] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9176.155219] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9176.174151] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9176.182386] RAX: 00000000000078e9 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 9176.186664] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 9176.190925] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 9176.195629] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 9176.199945] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 9176.203127] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 9176.206604] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9176.209654] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 9176.214652] Call Trace: [ 9176.215465] [ 9176.216190] ? show_regs.cold.9+0x22/0x2f [ 9176.219666] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9176.223756] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9176.226141] ? watchdog+0x40/0x40 [ 9176.228539] ? hrtimer_interrupt+0x128/0x2c0 [ 9176.231033] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9176.238224] ? apic_timer_interrupt+0xf/0x20 [ 9176.242790] [ 9176.245653] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9176.250132] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9176.255566] do_raw_spin_lock+0x177/0x190 [ 9176.258338] _raw_spin_lock+0x21/0x30 [ 9176.263216] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9176.268325] process_one_work+0x2c8/0x700 [ 9176.270904] worker_thread+0x296/0x6e0 [ 9176.273617] ? rescuer_thread+0x570/0x570 [ 9176.277046] kthread+0x1d1/0x200 [ 9176.278510] ? set_kthread_struct+0x70/0x70 [ 9176.282540] ret_from_fork+0x1f/0x30 [ 9188.060119] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [ 9188.063385] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9188.097471] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9188.101918] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9188.107176] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9188.110279] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9188.129879] RSP: 0000:ffffaf9fc15234f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9188.135895] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 9188.144623] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 9188.149257] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9188.154079] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 9188.158884] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 9188.164540] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 9188.169347] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9188.173674] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 9188.178511] Call Trace: [ 9188.180177] [ 9188.181121] ? show_regs.cold.9+0x22/0x2f [ 9188.184102] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9188.186686] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9188.189329] ? watchdog+0x40/0x40 [ 9188.190552] ? hrtimer_interrupt+0x128/0x2c0 [ 9188.196330] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9188.200269] ? apic_timer_interrupt+0xf/0x20 [ 9188.204835] [ 9188.205910] ? native_safe_halt+0x2e/0x50 [ 9188.210848] kvm_wait+0xcd/0xe0 [ 9188.213862] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9188.216543] do_raw_spin_lock+0x177/0x190 [ 9188.218694] _raw_spin_lock+0x21/0x30 [ 9188.223977] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9188.229762] ? do_raw_spin_unlock+0x75/0x190 [ 9188.233627] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9188.239749] ? do_wait_intr+0xf0/0xf0 [ 9188.241718] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9188.244957] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9188.249743] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9188.254291] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9188.261425] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9188.264967] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9188.269210] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9188.271986] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9188.280950] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9188.286917] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9188.289969] lmv_intent_open+0x338/0x1150 [lmv] [ 9188.294870] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9188.301825] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9188.312261] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9188.317958] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9188.325912] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9188.337599] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9188.340783] ? do_raw_spin_unlock+0x75/0x190 [ 9188.348654] ? _raw_spin_unlock+0x12/0x30 [ 9188.350070] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9188.359070] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9188.371834] ll_atomic_open+0x36a/0xd70 [lustre] [ 9188.379909] lookup_open+0x356/0xb40 [ 9188.381264] do_last+0x313/0xfc0 [ 9188.383511] ? nd_jump_root+0xe5/0x160 [ 9188.385807] ? path_init+0x437/0x520 [ 9188.387955] path_openat+0xf7/0x500 [ 9188.389804] ? do_raw_spin_unlock+0x75/0x190 [ 9188.392778] do_filp_open+0x99/0x140 [ 9188.400315] ? getname_flags+0x6e/0x330 [ 9188.402464] ? __check_object_size+0xff/0x256 [ 9188.405851] ? do_raw_spin_unlock+0x75/0x190 [ 9188.408276] ? _raw_spin_unlock+0x12/0x30 [ 9188.411211] do_sys_openat2+0x2b4/0x410 [ 9188.413548] do_sys_open+0x73/0xa0 [ 9188.416233] __x64_sys_openat+0x24/0x30 [ 9188.418348] do_syscall_64+0xc1/0x440 [ 9188.422260] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9188.425345] RIP: 0033:0x7ff54af45332 [ 9188.430214] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9188.443418] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9188.448913] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 9188.454006] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 9188.458848] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 9188.463186] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 9188.467808] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 9196.055065] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 9196.057083] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9196.071953] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9196.083732] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9196.095105] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9196.099817] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9196.107446] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9196.116330] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 9196.122487] RDX: 0000000000006a53 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 9196.127896] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 9196.132777] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 9196.135294] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9196.137631] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 9196.141454] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9196.143779] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 9196.146676] Call Trace: [ 9196.147860] [ 9196.148675] ? show_regs.cold.9+0x22/0x2f [ 9196.150337] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9196.151893] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9196.153639] ? watchdog+0x40/0x40 [ 9196.155127] ? hrtimer_interrupt+0x128/0x2c0 [ 9196.156696] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9196.159660] ? apic_timer_interrupt+0xf/0x20 [ 9196.161886] [ 9196.162877] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9196.165359] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9196.170339] do_raw_spin_lock+0x177/0x190 [ 9196.172955] _raw_spin_lock+0x21/0x30 [ 9196.174968] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9196.177975] obd_mod_free+0x46/0x140 [ptlrpc] [ 9196.181944] mdc_commit_open+0x91/0xb0 [mdc] [ 9196.184512] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9196.187974] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9196.191746] after_reply+0xe2c/0x1780 [ptlrpc] [ 9196.194401] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9196.197474] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9196.200359] ? do_wait_intr+0xf0/0xf0 [ 9196.202462] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9196.205489] kthread+0x1d1/0x200 [ 9196.212205] ? set_kthread_struct+0x70/0x70 [ 9196.214323] ret_from_fork+0x1f/0x30 [ 9200.058014] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [ 9200.060031] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9200.084868] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9200.093109] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9200.099177] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9200.101753] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9200.112894] RSP: 0018:ffffaf9fc1543ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9200.117205] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 9200.121372] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 9200.125681] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9200.129904] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 9200.134309] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 9200.138651] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 9200.143239] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9200.146643] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 9200.151197] Call Trace: [ 9200.152637] [ 9200.153807] ? show_regs.cold.9+0x22/0x2f [ 9200.156126] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9200.162579] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9200.170806] ? watchdog+0x40/0x40 [ 9200.175237] ? hrtimer_interrupt+0x128/0x2c0 [ 9200.177850] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9200.181142] ? apic_timer_interrupt+0xf/0x20 [ 9200.184131] [ 9200.185467] ? native_safe_halt+0x2e/0x50 [ 9200.188442] kvm_wait+0xcd/0xe0 [ 9200.190308] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9200.193200] do_raw_spin_lock+0x177/0x190 [ 9200.194958] _raw_spin_lock+0x21/0x30 [ 9200.197124] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9200.200346] ? do_raw_spin_unlock+0x75/0x190 [ 9200.203903] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9200.207670] ? do_wait_intr+0xf0/0xf0 [ 9200.210059] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9200.213268] mdc_close+0x53b/0xd70 [mdc] [ 9200.215584] lmv_close+0x18d/0x490 [lmv] [ 9200.217946] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 9200.222306] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9200.225699] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9200.229471] ll_md_close+0x274/0x800 [lustre] [ 9200.232661] ll_file_release+0x128/0xc50 [lustre] [ 9200.235703] __fput+0x10d/0x440 [ 9200.237095] ____fput+0x12/0x20 [ 9200.239115] task_work_run+0xc6/0x110 [ 9200.241275] exit_to_usermode_loop+0x1dd/0x1f0 [ 9200.243960] do_syscall_64+0x3d6/0x440 [ 9200.246338] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9200.249488] RIP: 0033:0x7ff54af45cd8 [ 9200.251505] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9200.262739] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9200.267603] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 9200.271854] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 9200.276157] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 9200.280202] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 9200.284597] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9201.119099] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7242s! [ 9201.123879] Showing busy workqueues and worker pools: [ 9201.126343] workqueue events: flags=0x0 [ 9201.128584] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9201.133963] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 9201.139834] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 9201.144408] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 9201.150754] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9201.154800] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 9201.159850] workqueue events_long: flags=0x0 [ 9201.163340] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9201.166326] pending: lu_objects_flush [obdclass] [ 9201.168264] workqueue events_unbound: flags=0x2 [ 9201.170801] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 9201.175279] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9201.181440] workqueue events_power_efficient: flags=0x80 [ 9201.185850] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9201.189323] pending: neigh_periodic_work [ 9201.191457] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9201.194762] pending: do_cache_clean, check_lifetime [ 9201.197475] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9201.200692] pending: neigh_periodic_work [ 9201.202514] workqueue writeback: flags=0x4a [ 9201.204454] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9201.206860] pending: wb_workfn, wb_workfn [ 9201.208784] workqueue xprtiod: flags=0xa [ 9201.210428] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9201.213308] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 9201.216444] workqueue ipv6_addrconf: flags=0x40008 [ 9201.218436] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9201.221558] pending: addrconf_verify_work [ 9201.222968] workqueue knbd0-recv: flags=0x1a [ 9201.225112] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9201.227798] in-flight: 657:recv_work [ 9201.229844] workqueue ptlrpc_pinger: flags=0x2 [ 9201.232790] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9201.236552] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 9201.240510] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7242s workers=4 idle: 20831 12642 28859 [ 9201.244812] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9191s workers=2 idle: 659 [ 9204.060428] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 9204.067326] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9204.116256] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9204.125052] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9204.132312] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9204.145860] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9204.153017] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9204.166924] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9204.172469] RAX: 0000000000007cf9 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 9204.177377] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 9204.181909] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 9204.186848] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 9204.193945] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 9204.201496] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 9204.209358] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9204.213499] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 9204.219788] Call Trace: [ 9204.223563] [ 9204.224261] ? show_regs.cold.9+0x22/0x2f [ 9204.225626] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9204.237453] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9204.243969] ? watchdog+0x40/0x40 [ 9204.248339] ? hrtimer_interrupt+0x128/0x2c0 [ 9204.257765] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9204.260322] ? apic_timer_interrupt+0xf/0x20 [ 9204.262913] [ 9204.263697] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9204.265593] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9204.270527] do_raw_spin_lock+0x177/0x190 [ 9204.273886] _raw_spin_lock+0x21/0x30 [ 9204.275915] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9204.283064] process_one_work+0x2c8/0x700 [ 9204.285775] worker_thread+0x296/0x6e0 [ 9204.288406] ? rescuer_thread+0x570/0x570 [ 9204.291589] kthread+0x1d1/0x200 [ 9204.294962] ? set_kthread_struct+0x70/0x70 [ 9204.296246] ret_from_fork+0x1f/0x30 [ 9214.057091] rcu: INFO: rcu_sched self-detected stall on CPU [ 9214.060252] rcu: 0-...!: (3770766 ticks this GP) idle=98a/1/0x4000000000000002 softirq=125154/125154 fqs=959 [ 9214.065526] (t=7260217 jiffies g=253585 q=5290) [ 9214.068967] rcu: rcu_sched kthread starved for 7255177 jiffies! g253585 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [ 9214.079626] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 9214.086521] rcu: RCU grace-period kthread stack dump: [ 9214.091235] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 9214.099120] Call Trace: [ 9214.100987] __schedule+0x351/0xcb0 [ 9214.102670] schedule+0xc0/0x180 [ 9214.104429] schedule_timeout+0xb4/0x190 [ 9214.106700] ? do_raw_spin_unlock+0x75/0x190 [ 9214.109212] ? __next_timer_interrupt+0x160/0x160 [ 9214.112150] ? prepare_to_swait_event+0xf3/0x140 [ 9214.116249] rcu_gp_kthread+0x8dd/0x15b0 [ 9214.118582] ? force_qs_rnp+0x2c0/0x2c0 [ 9214.120925] kthread+0x1d1/0x200 [ 9214.124416] ? set_kthread_struct+0x70/0x70 [ 9214.128199] ret_from_fork+0x1f/0x30 [ 9214.129454] rcu: Stack dump where RCU GP kthread last ran: [ 9214.138381] NMI backtrace for cpu 0 [ 9214.140934] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9214.150624] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9214.155420] Call Trace: [ 9214.156651] [ 9214.157657] ? dump_stack+0xbb/0x10e [ 9214.159565] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 9214.162673] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 9214.165792] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 9214.167401] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 9214.179626] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 9214.184865] ? rcu_check_gp_kthread_starvation+0x16f/0x188 [ 9214.187023] ? rcu_sched_clock_irq.cold.51+0xce/0x307 [ 9214.188957] ? update_process_times+0x6d/0xc0 [ 9214.190829] ? tick_sched_handle.isra.18+0x27/0x90 [ 9214.192819] ? tick_sched_timer+0x53/0xb0 [ 9214.194323] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9214.196064] ? tick_sched_handle.isra.18+0x90/0x90 [ 9214.198208] ? hrtimer_interrupt+0x128/0x2c0 [ 9214.200061] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9214.202087] ? apic_timer_interrupt+0xf/0x20 [ 9214.203822] [ 9214.204835] ? native_safe_halt+0x2e/0x50 [ 9214.206707] ? kvm_wait+0xcd/0xe0 [ 9214.208178] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9214.210406] ? do_raw_spin_lock+0x177/0x190 [ 9214.212226] ? _raw_spin_lock+0x21/0x30 [ 9214.213836] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9214.216639] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 9214.218983] ? mdc_commit_open+0x91/0xb0 [mdc] [ 9214.220950] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9214.223377] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9214.225979] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 9214.228224] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9214.230545] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9214.233191] ? do_wait_intr+0xf0/0xf0 [ 9214.234526] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9214.236686] ? kthread+0x1d1/0x200 [ 9214.238122] ? set_kthread_struct+0x70/0x70 [ 9214.240674] ? ret_from_fork+0x1f/0x30 [ 9214.244149] NMI backtrace for cpu 0 [ 9214.250007] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9214.263103] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9214.269139] Call Trace: [ 9214.270656] [ 9214.271741] ? dump_stack+0xbb/0x10e [ 9214.273709] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 9214.278355] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 9214.287287] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 9214.289140] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 9214.297489] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 9214.299484] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 9214.301667] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 9214.307750] ? update_process_times+0x6d/0xc0 [ 9214.311202] ? tick_sched_handle.isra.18+0x27/0x90 [ 9214.314097] ? tick_sched_timer+0x53/0xb0 [ 9214.317289] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9214.320600] ? tick_sched_handle.isra.18+0x90/0x90 [ 9214.322861] ? hrtimer_interrupt+0x128/0x2c0 [ 9214.327075] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9214.330142] ? apic_timer_interrupt+0xf/0x20 [ 9214.335549] [ 9214.337884] ? native_safe_halt+0x2e/0x50 [ 9214.340823] ? kvm_wait+0xcd/0xe0 [ 9214.343362] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9214.346540] ? do_raw_spin_lock+0x177/0x190 [ 9214.348295] ? _raw_spin_lock+0x21/0x30 [ 9214.353513] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9214.358463] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 9214.364206] ? mdc_commit_open+0x91/0xb0 [mdc] [ 9214.369055] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9214.371146] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9214.378994] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 9214.397861] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9214.404583] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9214.413143] ? do_wait_intr+0xf0/0xf0 [ 9214.414433] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9214.424566] ? kthread+0x1d1/0x200 [ 9214.429184] ? set_kthread_struct+0x70/0x70 [ 9214.430683] ? ret_from_fork+0x1f/0x30 [ 9216.062408] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [ 9216.092834] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9216.143644] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9216.151190] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9216.157822] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9216.169115] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9216.190257] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9216.205298] RAX: 0000000000005e83 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 9216.214596] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 9216.224460] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9216.232766] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 9216.240699] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 9216.246670] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 9216.251435] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9216.260636] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 9216.267722] Call Trace: [ 9216.270588] [ 9216.273896] ? show_regs.cold.9+0x22/0x2f [ 9216.278022] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9216.282204] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9216.289574] ? watchdog+0x40/0x40 [ 9216.292437] ? hrtimer_interrupt+0x128/0x2c0 [ 9216.299623] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9216.306437] ? apic_timer_interrupt+0xf/0x20 [ 9216.310501] [ 9216.312133] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9216.315936] do_raw_spin_lock+0x177/0x190 [ 9216.320541] _raw_spin_lock+0x21/0x30 [ 9216.323976] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9216.327607] ? do_raw_spin_unlock+0x75/0x190 [ 9216.329731] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9216.332180] ? do_wait_intr+0xf0/0xf0 [ 9216.334003] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9216.336615] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9216.342749] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9216.348433] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9216.357587] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9216.362715] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9216.365413] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9216.371200] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9216.375357] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9216.381510] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9216.389098] lmv_intent_open+0x338/0x1150 [lmv] [ 9216.396692] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9216.406138] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9216.411737] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9216.417480] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9216.422481] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9216.429192] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9216.430883] ? do_raw_spin_unlock+0x75/0x190 [ 9216.437192] ? _raw_spin_unlock+0x12/0x30 [ 9216.438251] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9216.444690] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9216.449004] ll_atomic_open+0x36a/0xd70 [lustre] [ 9216.450257] lookup_open+0x356/0xb40 [ 9216.453167] do_last+0x313/0xfc0 [ 9216.454328] ? nd_jump_root+0xe5/0x160 [ 9216.455573] ? path_init+0x437/0x520 [ 9216.464517] path_openat+0xf7/0x500 [ 9216.465573] ? do_raw_spin_unlock+0x75/0x190 [ 9216.468631] do_filp_open+0x99/0x140 [ 9216.470365] ? getname_flags+0x6e/0x330 [ 9216.471521] ? __check_object_size+0xff/0x256 [ 9216.473365] ? do_raw_spin_unlock+0x75/0x190 [ 9216.474594] ? _raw_spin_unlock+0x12/0x30 [ 9216.475773] do_sys_openat2+0x2b4/0x410 [ 9216.476964] do_sys_open+0x73/0xa0 [ 9216.479462] __x64_sys_openat+0x24/0x30 [ 9216.481535] do_syscall_64+0xc1/0x440 [ 9216.483196] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9216.485237] RIP: 0033:0x7ff54af45332 [ 9216.486746] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9216.494556] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9216.499799] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 9216.503311] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 9216.511235] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 9216.513480] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 9216.518578] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 9228.059284] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [ 9228.068938] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9228.141127] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9228.148163] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9228.159486] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9228.166985] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9228.189325] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9228.201835] RAX: 0000000000005c83 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 9228.209548] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 9228.213894] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9228.219276] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 9228.225851] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 9228.236106] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 9228.246781] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9228.257333] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 9228.261479] Call Trace: [ 9228.262332] [ 9228.263147] ? show_regs.cold.9+0x22/0x2f [ 9228.265698] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9228.270507] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9228.272161] ? watchdog+0x40/0x40 [ 9228.276117] ? hrtimer_interrupt+0x128/0x2c0 [ 9228.278569] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9228.288577] ? apic_timer_interrupt+0xf/0x20 [ 9228.293557] [ 9228.295438] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9228.301131] do_raw_spin_lock+0x177/0x190 [ 9228.303264] _raw_spin_lock+0x21/0x30 [ 9228.305325] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9228.308529] ? do_raw_spin_unlock+0x75/0x190 [ 9228.311992] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9228.315092] ? do_wait_intr+0xf0/0xf0 [ 9228.317776] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9228.322545] mdc_close+0x53b/0xd70 [mdc] [ 9228.328713] lmv_close+0x18d/0x490 [lmv] [ 9228.331623] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 9228.334519] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9228.337405] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9228.339616] ll_md_close+0x274/0x800 [lustre] [ 9228.341699] ll_file_release+0x128/0xc50 [lustre] [ 9228.343983] __fput+0x10d/0x440 [ 9228.345436] ____fput+0x12/0x20 [ 9228.346890] task_work_run+0xc6/0x110 [ 9228.348607] exit_to_usermode_loop+0x1dd/0x1f0 [ 9228.350523] do_syscall_64+0x3d6/0x440 [ 9228.352155] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9228.354337] RIP: 0033:0x7ff54af45cd8 [ 9228.355969] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9228.364096] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9228.367028] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 9228.370042] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 9228.373265] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 9228.376507] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 9228.379897] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9231.327570] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7272s! [ 9231.335482] Showing busy workqueues and worker pools: [ 9231.338485] workqueue events: flags=0x0 [ 9231.341142] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9231.345519] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 9231.356369] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 9231.364953] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 9231.370320] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9231.380143] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 9231.385701] workqueue events_long: flags=0x0 [ 9231.388426] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9231.392122] pending: lu_objects_flush [obdclass] [ 9231.395057] workqueue events_unbound: flags=0x2 [ 9231.397276] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 9231.401303] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9231.413291] workqueue events_power_efficient: flags=0x80 [ 9231.417482] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9231.421409] pending: neigh_periodic_work [ 9231.425685] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9231.430637] pending: do_cache_clean, check_lifetime [ 9231.435970] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9231.440395] pending: neigh_periodic_work [ 9231.446239] workqueue writeback: flags=0x4a [ 9231.455359] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9231.464864] pending: wb_workfn, wb_workfn [ 9231.470653] workqueue xprtiod: flags=0xa [ 9231.472574] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9231.476812] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 9231.493489] workqueue ipv6_addrconf: flags=0x40008 [ 9231.502507] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9231.510838] pending: addrconf_verify_work [ 9231.518084] workqueue knbd0-recv: flags=0x1a [ 9231.521345] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9231.526275] in-flight: 657:recv_work [ 9231.528497] workqueue ptlrpc_pinger: flags=0x2 [ 9231.534898] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9231.539433] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 9231.545522] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7272s workers=4 idle: 20831 12642 28859 [ 9231.551256] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9221s workers=2 idle: 659 [ 9232.061139] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 9232.076997] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9232.122894] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9232.129596] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9232.134468] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9232.138153] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9232.142359] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9232.153482] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9232.157639] RAX: 0000000000007f4d RBX: ffff96848811cae0 RCX: 0000000000000001 [ 9232.161571] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 9232.165387] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 9232.169335] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 9232.173589] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 9232.177502] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 9232.182020] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9232.184993] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 9232.188469] Call Trace: [ 9232.189946] [ 9232.191079] ? show_regs.cold.9+0x22/0x2f [ 9232.194227] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9232.197943] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9232.204484] ? watchdog+0x40/0x40 [ 9232.208537] ? hrtimer_interrupt+0x128/0x2c0 [ 9232.217819] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9232.221063] ? apic_timer_interrupt+0xf/0x20 [ 9232.224662] [ 9232.226489] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9232.229691] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9232.235517] do_raw_spin_lock+0x177/0x190 [ 9232.236987] _raw_spin_lock+0x21/0x30 [ 9232.238543] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9232.241126] process_one_work+0x2c8/0x700 [ 9232.243033] worker_thread+0x296/0x6e0 [ 9232.248103] ? rescuer_thread+0x570/0x570 [ 9232.251269] kthread+0x1d1/0x200 [ 9232.254691] ? set_kthread_struct+0x70/0x70 [ 9232.256169] ret_from_fork+0x1f/0x30 [ 9244.061194] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [ 9244.070285] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9244.120190] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9244.131521] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9244.156503] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9244.165358] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9244.193320] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9244.204338] RAX: 0000000000007bf7 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 9244.216009] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 9244.223172] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9244.228922] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 9244.239333] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 9244.243411] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 9244.255507] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9244.268209] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 9244.274824] Call Trace: [ 9244.276250] [ 9244.278774] ? show_regs.cold.9+0x22/0x2f [ 9244.283118] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9244.286738] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9244.289023] ? watchdog+0x40/0x40 [ 9244.290561] ? hrtimer_interrupt+0x128/0x2c0 [ 9244.294064] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9244.296822] ? apic_timer_interrupt+0xf/0x20 [ 9244.298869] [ 9244.301906] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9244.304698] do_raw_spin_lock+0x177/0x190 [ 9244.306928] _raw_spin_lock+0x21/0x30 [ 9244.308940] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9244.318281] ? do_raw_spin_unlock+0x75/0x190 [ 9244.322051] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9244.337766] ? do_wait_intr+0xf0/0xf0 [ 9244.340137] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9244.345795] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9244.350082] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9244.353966] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9244.362357] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9244.369548] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9244.373444] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9244.376803] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9244.384139] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9244.391018] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9244.395165] lmv_intent_open+0x338/0x1150 [lmv] [ 9244.400121] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9244.406015] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9244.412229] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9244.415330] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9244.418804] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9244.424621] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9244.426610] ? do_raw_spin_unlock+0x75/0x190 [ 9244.435347] ? _raw_spin_unlock+0x12/0x30 [ 9244.439550] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9244.444968] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9244.448059] ll_atomic_open+0x36a/0xd70 [lustre] [ 9244.451052] lookup_open+0x356/0xb40 [ 9244.454531] do_last+0x313/0xfc0 [ 9244.456484] ? nd_jump_root+0xe5/0x160 [ 9244.457907] ? path_init+0x437/0x520 [ 9244.459474] path_openat+0xf7/0x500 [ 9244.461079] ? do_raw_spin_unlock+0x75/0x190 [ 9244.463023] do_filp_open+0x99/0x140 [ 9244.464887] ? getname_flags+0x6e/0x330 [ 9244.467031] ? __check_object_size+0xff/0x256 [ 9244.469385] ? do_raw_spin_unlock+0x75/0x190 [ 9244.471839] ? _raw_spin_unlock+0x12/0x30 [ 9244.473856] do_sys_openat2+0x2b4/0x410 [ 9244.476053] do_sys_open+0x73/0xa0 [ 9244.477663] __x64_sys_openat+0x24/0x30 [ 9244.479479] do_syscall_64+0xc1/0x440 [ 9244.481282] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9244.483542] RIP: 0033:0x7ff54af45332 [ 9244.485491] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9244.494475] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9244.507947] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 9244.519377] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 9244.526093] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 9244.529337] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 9244.532161] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 9256.056684] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 9256.059602] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 9256.060679] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9256.060764] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9256.060771] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9256.060775] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9256.060796] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9256.060801] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9256.060806] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 9256.060809] RDX: 0000000000007e7b RSI: 0000000000000003 RDI: ffff96848811cae0 [ 9256.060811] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 9256.060813] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 9256.060816] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9256.060819] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 9256.060822] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9256.060824] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 9256.060834] Call Trace: [ 9256.060840] [ 9256.060847] ? show_regs.cold.9+0x22/0x2f [ 9256.060856] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9256.060861] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9256.060866] ? watchdog+0x40/0x40 [ 9256.060870] ? hrtimer_interrupt+0x128/0x2c0 [ 9256.060874] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9256.060884] ? apic_timer_interrupt+0xf/0x20 [ 9256.060888] [ 9256.060891] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9256.060896] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9256.060901] do_raw_spin_lock+0x177/0x190 [ 9256.060906] _raw_spin_lock+0x21/0x30 [ 9256.060912] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9256.061212] obd_mod_free+0x46/0x140 [ptlrpc] [ 9256.061406] mdc_commit_open+0x91/0xb0 [mdc] [ 9256.061446] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9256.066919] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9256.067016] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9256.067023] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9256.067027] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9256.067044] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9256.067048] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9256.067053] RAX: 0000000000003610 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 9256.067061] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 9256.067063] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9256.067066] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 9256.067068] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 9256.067071] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 9256.067074] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9256.067076] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 9256.067090] Call Trace: [ 9256.067097] [ 9256.067104] ? show_regs.cold.9+0x22/0x2f [ 9256.067229] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9256.067238] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9256.067242] ? watchdog+0x40/0x40 [ 9256.067246] ? hrtimer_interrupt+0x128/0x2c0 [ 9256.067250] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9256.067260] ? apic_timer_interrupt+0xf/0x20 [ 9256.067265] [ 9256.067268] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9256.067274] do_raw_spin_lock+0x177/0x190 [ 9256.067279] _raw_spin_lock+0x21/0x30 [ 9256.067284] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9256.067628] ? do_raw_spin_unlock+0x75/0x190 [ 9256.067633] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9256.067909] ? do_wait_intr+0xf0/0xf0 [ 9256.067917] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9256.068123] mdc_close+0x53b/0xd70 [mdc] [ 9256.069073] lmv_close+0x18d/0x490 [lmv] [ 9256.069102] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 9256.069289] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9256.069402] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9256.069483] ll_md_close+0x274/0x800 [lustre] [ 9256.069565] ll_file_release+0x128/0xc50 [lustre] [ 9256.069832] __fput+0x10d/0x440 [ 9256.069843] ____fput+0x12/0x20 [ 9256.069848] task_work_run+0xc6/0x110 [ 9256.069855] exit_to_usermode_loop+0x1dd/0x1f0 [ 9256.069863] do_syscall_64+0x3d6/0x440 [ 9256.069868] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9256.069876] RIP: 0033:0x7ff54af45cd8 [ 9256.069885] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9256.069888] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9256.069894] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 9256.069896] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 9256.069898] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 9256.069901] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 9256.069903] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9256.488028] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9256.492594] after_reply+0xe2c/0x1780 [ptlrpc] [ 9256.494963] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9256.498098] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9256.500427] ? do_wait_intr+0xf0/0xf0 [ 9256.502069] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9256.504629] kthread+0x1d1/0x200 [ 9256.506410] ? set_kthread_struct+0x70/0x70 [ 9256.508431] ret_from_fork+0x1f/0x30 [ 9260.063665] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 9260.072619] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9260.115697] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9260.119613] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9260.126198] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9260.130278] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9260.136821] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9260.149658] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9260.153894] RAX: 000000000000162a RBX: ffff96848811cae0 RCX: 0000000000000001 [ 9260.157494] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 9260.163701] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 9260.167772] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 9260.171938] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 9260.175125] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 9260.178431] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9260.180822] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 9260.183822] Call Trace: [ 9260.184713] [ 9260.185869] ? show_regs.cold.9+0x22/0x2f [ 9260.187647] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9260.189412] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9260.191143] ? watchdog+0x40/0x40 [ 9260.192397] ? hrtimer_interrupt+0x128/0x2c0 [ 9260.194080] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9260.195981] ? apic_timer_interrupt+0xf/0x20 [ 9260.197855] [ 9260.198760] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9260.201166] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9260.203379] do_raw_spin_lock+0x177/0x190 [ 9260.205809] _raw_spin_lock+0x21/0x30 [ 9260.207205] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9260.209481] process_one_work+0x2c8/0x700 [ 9260.211128] worker_thread+0x296/0x6e0 [ 9260.212634] ? rescuer_thread+0x570/0x570 [ 9260.214507] kthread+0x1d1/0x200 [ 9260.215504] ? set_kthread_struct+0x70/0x70 [ 9260.216675] ret_from_fork+0x1f/0x30 [ 9262.047042] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7303s! [ 9262.050854] Showing busy workqueues and worker pools: [ 9262.053328] workqueue events: flags=0x0 [ 9262.054980] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9262.059000] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 9262.063323] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 9262.066277] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 9262.071861] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9262.075825] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 9262.079253] workqueue events_long: flags=0x0 [ 9262.081384] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9262.084031] pending: lu_objects_flush [obdclass] [ 9262.086033] workqueue events_unbound: flags=0x2 [ 9262.088255] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 9262.091091] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9262.095382] workqueue events_power_efficient: flags=0x80 [ 9262.097712] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9262.101053] pending: neigh_periodic_work [ 9262.103083] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9262.105660] pending: do_cache_clean, check_lifetime [ 9262.107963] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9262.110280] pending: neigh_periodic_work [ 9262.112352] workqueue writeback: flags=0x4a [ 9262.114188] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9262.115991] pending: wb_workfn, wb_workfn [ 9262.117119] workqueue xprtiod: flags=0xa [ 9262.118232] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9262.120339] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 9262.123374] workqueue ipv6_addrconf: flags=0x40008 [ 9262.125057] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9262.127225] pending: addrconf_verify_work [ 9262.128470] workqueue knbd0-recv: flags=0x1a [ 9262.129713] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9262.131632] in-flight: 657:recv_work [ 9262.132779] workqueue ptlrpc_pinger: flags=0x2 [ 9262.133963] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9262.135817] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 9262.138000] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7303s workers=4 idle: 20831 12642 28859 [ 9262.140454] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9252s workers=2 idle: 659 [ 9272.063129] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [ 9272.069048] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9272.094621] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9272.102084] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9272.110301] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9272.114611] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9272.127977] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9272.135497] RAX: 0000000000007d8a RBX: ffff96848811cae0 RCX: 0000000000000001 [ 9272.140817] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 9272.145728] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9272.151089] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 9272.155538] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 9272.160035] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 9272.164768] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9272.168107] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 9272.172077] Call Trace: [ 9272.173512] [ 9272.174575] ? show_regs.cold.9+0x22/0x2f [ 9272.177004] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9272.179531] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9272.183740] ? watchdog+0x40/0x40 [ 9272.185645] ? hrtimer_interrupt+0x128/0x2c0 [ 9272.188008] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9272.190801] ? apic_timer_interrupt+0xf/0x20 [ 9272.195426] [ 9272.196628] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9272.199800] do_raw_spin_lock+0x177/0x190 [ 9272.202711] _raw_spin_lock+0x21/0x30 [ 9272.204659] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9272.208179] ? do_raw_spin_unlock+0x75/0x190 [ 9272.210480] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9272.213005] ? do_wait_intr+0xf0/0xf0 [ 9272.214715] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9272.217705] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9272.220598] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9272.223875] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9272.226744] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9272.229682] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9272.231667] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9272.233950] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9272.236595] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9272.239582] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9272.242194] lmv_intent_open+0x338/0x1150 [lmv] [ 9272.247357] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9272.250205] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9272.253250] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9272.255470] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9272.258960] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9272.262939] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9272.266035] ? do_raw_spin_unlock+0x75/0x190 [ 9272.269581] ? _raw_spin_unlock+0x12/0x30 [ 9272.272639] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9272.275383] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9272.278076] ll_atomic_open+0x36a/0xd70 [lustre] [ 9272.282868] lookup_open+0x356/0xb40 [ 9272.284745] do_last+0x313/0xfc0 [ 9272.286767] ? nd_jump_root+0xe5/0x160 [ 9272.288956] ? path_init+0x437/0x520 [ 9272.291693] path_openat+0xf7/0x500 [ 9272.294278] ? do_raw_spin_unlock+0x75/0x190 [ 9272.297107] do_filp_open+0x99/0x140 [ 9272.298567] ? getname_flags+0x6e/0x330 [ 9272.301094] ? __check_object_size+0xff/0x256 [ 9272.304043] ? do_raw_spin_unlock+0x75/0x190 [ 9272.306914] ? _raw_spin_unlock+0x12/0x30 [ 9272.309858] do_sys_openat2+0x2b4/0x410 [ 9272.312063] do_sys_open+0x73/0xa0 [ 9272.314239] __x64_sys_openat+0x24/0x30 [ 9272.316618] do_syscall_64+0xc1/0x440 [ 9272.319015] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9272.323047] RIP: 0033:0x7ff54af45332 [ 9272.326708] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9272.339044] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9272.344629] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 9272.348893] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 9272.353242] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 9272.358706] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 9272.363236] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 9284.061380] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 9284.072397] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9284.110784] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9284.119649] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9284.125443] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9284.135763] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9284.151456] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9284.155970] RAX: 0000000000001895 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 9284.159602] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 9284.165617] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9284.170545] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 9284.174674] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 9284.178755] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 9284.181295] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9284.184590] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 9284.187867] Call Trace: [ 9284.188716] [ 9284.190516] ? show_regs.cold.9+0x22/0x2f [ 9284.193584] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9284.197606] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9284.206107] ? watchdog+0x40/0x40 [ 9284.207401] ? hrtimer_interrupt+0x128/0x2c0 [ 9284.208893] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9284.219593] ? apic_timer_interrupt+0xf/0x20 [ 9284.224113] [ 9284.225446] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9284.233953] do_raw_spin_lock+0x177/0x190 [ 9284.239635] _raw_spin_lock+0x21/0x30 [ 9284.242095] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9284.247772] ? do_raw_spin_unlock+0x75/0x190 [ 9284.252394] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9284.257740] ? do_wait_intr+0xf0/0xf0 [ 9284.262832] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9284.272639] mdc_close+0x53b/0xd70 [mdc] [ 9284.277647] lmv_close+0x18d/0x490 [lmv] [ 9284.283736] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 9284.295851] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9284.303644] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9284.307336] ll_md_close+0x274/0x800 [lustre] [ 9284.312977] ll_file_release+0x128/0xc50 [lustre] [ 9284.316566] __fput+0x10d/0x440 [ 9284.318543] ____fput+0x12/0x20 [ 9284.324626] task_work_run+0xc6/0x110 [ 9284.326520] exit_to_usermode_loop+0x1dd/0x1f0 [ 9284.331416] do_syscall_64+0x3d6/0x440 [ 9284.333513] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9284.338693] RIP: 0033:0x7ff54af45cd8 [ 9284.340453] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9284.360908] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9284.373832] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 9284.376726] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 9284.382066] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 9284.389977] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 9284.394381] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9288.055025] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [ 9288.059093] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 9288.062716] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) [ 9288.066220] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) [ 9288.081981] ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common [ 9288.093861] lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac [ 9288.101009] sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9288.101050] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9288.101058] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9288.101061] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9288.101091] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9288.101095] RSP: 0000:ffffaf9fc1b97bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9288.101101] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 9288.114197] rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9288.114237] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9288.114256] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9288.114265] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9288.129921] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 9288.136854] [ 9288.136864] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9288.136882] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9288.136887] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9288.136893] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 9288.136895] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 9288.136898] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 9288.136900] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 9288.136909] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 9288.136912] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 9288.143576] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 9288.144931] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9288.144938] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 9288.144946] Call Trace: [ 9288.153498] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 9288.153509] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9288.153512] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 9288.153514] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9288.153516] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 9288.153522] Call Trace: [ 9288.153528] [ 9288.153532] ? show_regs.cold.9+0x22/0x2f [ 9288.153543] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9288.153548] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9288.153551] ? watchdog+0x40/0x40 [ 9288.153554] ? hrtimer_interrupt+0x128/0x2c0 [ 9288.153560] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9288.153568] ? apic_timer_interrupt+0xf/0x20 [ 9288.153571] [ 9288.153573] ? native_safe_halt+0x2e/0x50 [ 9288.153576] kvm_wait+0xcd/0xe0 [ 9288.153583] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9288.153588] do_raw_spin_lock+0x177/0x190 [ 9288.153591] _raw_spin_lock+0x21/0x30 [ 9288.153596] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9288.153834] obd_mod_free+0x46/0x140 [ptlrpc] [ 9288.154257] mdc_commit_open+0x91/0xb0 [mdc] [ 9288.154286] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9288.154454] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9288.154584] after_reply+0xe2c/0x1780 [ptlrpc] [ 9288.154774] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9288.154904] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9288.155311] ? do_wait_intr+0xf0/0xf0 [ 9288.155321] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9288.155531] kthread+0x1d1/0x200 [ 9288.155539] ? set_kthread_struct+0x70/0x70 [ 9288.155542] ret_from_fork+0x1f/0x30 [ 9288.387365] [ 9288.389588] ? show_regs.cold.9+0x22/0x2f [ 9288.393905] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9288.399592] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9288.404692] ? watchdog+0x40/0x40 [ 9288.406422] ? hrtimer_interrupt+0x128/0x2c0 [ 9288.409686] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9288.414106] ? apic_timer_interrupt+0xf/0x20 [ 9288.419274] [ 9288.421954] ? native_safe_halt+0x2e/0x50 [ 9288.424308] kvm_wait+0xcd/0xe0 [ 9288.425882] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9288.433306] do_raw_spin_lock+0x177/0x190 [ 9288.437039] _raw_spin_lock+0x21/0x30 [ 9288.441585] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9288.447569] process_one_work+0x2c8/0x700 [ 9288.449353] worker_thread+0x296/0x6e0 [ 9288.451897] ? rescuer_thread+0x570/0x570 [ 9288.454978] kthread+0x1d1/0x200 [ 9288.459074] ? set_kthread_struct+0x70/0x70 [ 9288.461746] ret_from_fork+0x1f/0x30 [ 9292.255484] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7333s! [ 9292.269684] Showing busy workqueues and worker pools: [ 9292.276248] workqueue events: flags=0x0 [ 9292.281088] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9292.287984] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 9292.293822] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 9292.299002] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 9292.305562] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9292.309499] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 9292.313690] workqueue events_long: flags=0x0 [ 9292.316109] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9292.320496] pending: lu_objects_flush [obdclass] [ 9292.323171] workqueue events_unbound: flags=0x2 [ 9292.330284] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 9292.335512] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9292.343836] workqueue events_power_efficient: flags=0x80 [ 9292.348041] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9292.354086] pending: neigh_periodic_work [ 9292.359504] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9292.366792] pending: do_cache_clean, check_lifetime [ 9292.370025] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9292.376581] pending: neigh_periodic_work [ 9292.379882] workqueue writeback: flags=0x4a [ 9292.382853] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9292.386280] pending: wb_workfn, wb_workfn [ 9292.389548] workqueue xprtiod: flags=0xa [ 9292.393321] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9292.399586] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 9292.405256] workqueue ipv6_addrconf: flags=0x40008 [ 9292.408119] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9292.412939] pending: addrconf_verify_work [ 9292.416054] workqueue knbd0-recv: flags=0x1a [ 9292.422098] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9292.426794] in-flight: 657:recv_work [ 9292.431476] workqueue ptlrpc_pinger: flags=0x2 [ 9292.434519] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9292.440945] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 9292.446262] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7333s workers=4 idle: 20831 12642 28859 [ 9292.452276] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9282s workers=2 idle: 659 [ 9300.062226] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 9300.074541] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9300.130484] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9300.135920] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9300.143356] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9300.146934] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9300.169903] RSP: 0000:ffffaf9fc15234f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9300.178853] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 9300.183902] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 9300.188374] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9300.194702] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 9300.200958] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 9300.208521] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 9300.214840] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9300.221921] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 9300.227945] Call Trace: [ 9300.230640] [ 9300.231872] ? show_regs.cold.9+0x22/0x2f [ 9300.234845] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9300.237258] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9300.239556] ? watchdog+0x40/0x40 [ 9300.240989] ? hrtimer_interrupt+0x128/0x2c0 [ 9300.243906] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9300.249342] ? apic_timer_interrupt+0xf/0x20 [ 9300.251569] [ 9300.252813] ? native_safe_halt+0x2e/0x50 [ 9300.255772] kvm_wait+0xcd/0xe0 [ 9300.257352] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9300.263806] do_raw_spin_lock+0x177/0x190 [ 9300.267190] _raw_spin_lock+0x21/0x30 [ 9300.270413] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9300.277256] ? do_raw_spin_unlock+0x75/0x190 [ 9300.283186] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9300.285233] ? do_wait_intr+0xf0/0xf0 [ 9300.292639] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9300.299807] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9300.302939] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9300.305981] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9300.309818] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9300.317389] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9300.322627] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9300.327692] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9300.334012] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9300.338664] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9300.342849] lmv_intent_open+0x338/0x1150 [lmv] [ 9300.345938] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9300.353032] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9300.362449] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9300.364751] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9300.368640] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9300.372951] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9300.377789] ? do_raw_spin_unlock+0x75/0x190 [ 9300.381226] ? _raw_spin_unlock+0x12/0x30 [ 9300.383227] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9300.386277] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9300.391770] ll_atomic_open+0x36a/0xd70 [lustre] [ 9300.400493] lookup_open+0x356/0xb40 [ 9300.402736] do_last+0x313/0xfc0 [ 9300.406491] ? nd_jump_root+0xe5/0x160 [ 9300.409939] ? path_init+0x437/0x520 [ 9300.413353] path_openat+0xf7/0x500 [ 9300.415039] ? do_raw_spin_unlock+0x75/0x190 [ 9300.425249] do_filp_open+0x99/0x140 [ 9300.427000] ? getname_flags+0x6e/0x330 [ 9300.428158] ? __check_object_size+0xff/0x256 [ 9300.436325] ? do_raw_spin_unlock+0x75/0x190 [ 9300.443028] ? _raw_spin_unlock+0x12/0x30 [ 9300.451296] do_sys_openat2+0x2b4/0x410 [ 9300.455738] do_sys_open+0x73/0xa0 [ 9300.460354] __x64_sys_openat+0x24/0x30 [ 9300.461921] do_syscall_64+0xc1/0x440 [ 9300.468188] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9300.477718] RIP: 0033:0x7ff54af45332 [ 9300.485949] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9300.510001] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9300.522238] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 9300.528361] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 9300.535334] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 9300.544050] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 9300.551064] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 9312.064238] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 9312.082120] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9312.103957] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9312.110750] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9312.113511] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9312.114763] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9312.136598] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9312.140747] RAX: 0000000000005078 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 9312.150286] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 9312.152985] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9312.161719] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 9312.168375] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 9312.176378] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 9312.183577] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9312.185888] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 9312.189704] Call Trace: [ 9312.190629] [ 9312.191251] ? show_regs.cold.9+0x22/0x2f [ 9312.195172] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9312.197438] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9312.199746] ? watchdog+0x40/0x40 [ 9312.203788] ? hrtimer_interrupt+0x128/0x2c0 [ 9312.212332] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9312.216021] ? apic_timer_interrupt+0xf/0x20 [ 9312.220418] [ 9312.223489] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9312.233436] do_raw_spin_lock+0x177/0x190 [ 9312.245420] _raw_spin_lock+0x21/0x30 [ 9312.251177] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9312.262682] ? do_raw_spin_unlock+0x75/0x190 [ 9312.269899] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9312.280896] ? do_wait_intr+0xf0/0xf0 [ 9312.288989] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9312.293005] mdc_close+0x53b/0xd70 [mdc] [ 9312.303597] lmv_close+0x18d/0x490 [lmv] [ 9312.305127] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 9312.319389] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9312.335144] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9312.336528] ll_md_close+0x274/0x800 [lustre] [ 9312.337663] ll_file_release+0x128/0xc50 [lustre] [ 9312.349874] __fput+0x10d/0x440 [ 9312.350841] ____fput+0x12/0x20 [ 9312.372244] task_work_run+0xc6/0x110 [ 9312.373326] exit_to_usermode_loop+0x1dd/0x1f0 [ 9312.380673] do_syscall_64+0x3d6/0x440 [ 9312.384336] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9312.390748] RIP: 0033:0x7ff54af45cd8 [ 9312.393997] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9312.407393] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9312.417621] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 9312.420686] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 9312.426479] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 9312.432050] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 9312.438795] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9316.059025] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 9316.063390] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9316.119254] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9316.128907] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9316.141239] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9316.148855] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9316.152320] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9316.182712] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9316.186832] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 9316.196178] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 9316.204302] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 9316.208461] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 9316.210425] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 9316.220464] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 9316.224248] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9316.230041] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 9316.231686] Call Trace: [ 9316.232341] [ 9316.233005] ? show_regs.cold.9+0x22/0x2f [ 9316.236580] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9316.239849] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9316.242016] ? watchdog+0x40/0x40 [ 9316.244827] ? hrtimer_interrupt+0x128/0x2c0 [ 9316.248349] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9316.252350] ? apic_timer_interrupt+0xf/0x20 [ 9316.259463] [ 9316.262847] ? native_safe_halt+0x2e/0x50 [ 9316.264217] kvm_wait+0xcd/0xe0 [ 9316.267544] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9316.268804] do_raw_spin_lock+0x177/0x190 [ 9316.273815] _raw_spin_lock+0x21/0x30 [ 9316.275820] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9316.283593] process_one_work+0x2c8/0x700 [ 9316.294323] worker_thread+0x296/0x6e0 [ 9316.296155] ? rescuer_thread+0x570/0x570 [ 9316.297991] kthread+0x1d1/0x200 [ 9316.299865] ? set_kthread_struct+0x70/0x70 [ 9316.302671] ret_from_fork+0x1f/0x30 [ 9320.055140] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 9320.065462] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9320.104015] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9320.110361] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9320.115679] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9320.117740] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9320.131271] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9320.135780] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 9320.139748] RDX: 00000000000057a5 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 9320.147459] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 9320.153784] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 9320.156461] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9320.159587] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 9320.165166] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9320.173217] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 9320.177749] Call Trace: [ 9320.178712] [ 9320.179930] ? show_regs.cold.9+0x22/0x2f [ 9320.183432] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9320.187537] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9320.189970] ? watchdog+0x40/0x40 [ 9320.191257] ? hrtimer_interrupt+0x128/0x2c0 [ 9320.192955] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9320.195492] ? apic_timer_interrupt+0xf/0x20 [ 9320.197456] [ 9320.198408] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9320.201160] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9320.203823] do_raw_spin_lock+0x177/0x190 [ 9320.205626] _raw_spin_lock+0x21/0x30 [ 9320.207703] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9320.210408] obd_mod_free+0x46/0x140 [ptlrpc] [ 9320.212650] mdc_commit_open+0x91/0xb0 [mdc] [ 9320.215309] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9320.220685] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9320.223427] after_reply+0xe2c/0x1780 [ptlrpc] [ 9320.227249] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9320.232431] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9320.234796] ? do_wait_intr+0xf0/0xf0 [ 9320.236407] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9320.241996] kthread+0x1d1/0x200 [ 9320.243449] ? set_kthread_struct+0x70/0x70 [ 9320.245164] ret_from_fork+0x1f/0x30 [ 9322.464185] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7363s! [ 9322.477982] Showing busy workqueues and worker pools: [ 9322.484806] workqueue events: flags=0x0 [ 9322.491099] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9322.501658] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 9322.515349] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 9322.518937] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 9322.537203] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9322.540221] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 9322.543676] workqueue events_long: flags=0x0 [ 9322.547030] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9322.554513] pending: lu_objects_flush [obdclass] [ 9322.566159] workqueue events_unbound: flags=0x2 [ 9322.570274] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 9322.584427] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9322.595448] workqueue events_power_efficient: flags=0x80 [ 9322.599509] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9322.609755] pending: neigh_periodic_work [ 9322.617055] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9322.624751] pending: do_cache_clean, check_lifetime [ 9322.633304] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9322.643471] pending: neigh_periodic_work [ 9322.650170] workqueue writeback: flags=0x4a [ 9322.654360] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9322.658013] pending: wb_workfn, wb_workfn [ 9322.659864] workqueue xprtiod: flags=0xa [ 9322.665338] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9322.670413] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 9322.676228] workqueue ipv6_addrconf: flags=0x40008 [ 9322.681898] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9322.686311] pending: addrconf_verify_work [ 9322.688691] workqueue knbd0-recv: flags=0x1a [ 9322.691910] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9322.696639] in-flight: 657:recv_work [ 9322.698955] workqueue ptlrpc_pinger: flags=0x2 [ 9322.702225] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9322.709943] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 9322.714644] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7363s workers=4 idle: 20831 12642 28859 [ 9322.721251] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9312s workers=2 idle: 659 [ 9328.062846] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 9328.070471] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9328.104874] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9328.119329] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9328.128514] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9328.138600] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9328.157475] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9328.168730] RAX: 000000000000686f RBX: ffff96848811cae0 RCX: 0000000000000001 [ 9328.176676] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 9328.183295] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9328.189587] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 9328.195294] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 9328.206489] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 9328.213549] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9328.218468] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 9328.231770] Call Trace: [ 9328.233334] [ 9328.234561] ? show_regs.cold.9+0x22/0x2f [ 9328.238737] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9328.243357] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9328.245475] ? watchdog+0x40/0x40 [ 9328.251336] ? hrtimer_interrupt+0x128/0x2c0 [ 9328.257306] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9328.263733] ? apic_timer_interrupt+0xf/0x20 [ 9328.268411] [ 9328.271262] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9328.275564] do_raw_spin_lock+0x177/0x190 [ 9328.278162] _raw_spin_lock+0x21/0x30 [ 9328.280593] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9328.283866] ? do_raw_spin_unlock+0x75/0x190 [ 9328.287697] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9328.294583] ? do_wait_intr+0xf0/0xf0 [ 9328.296312] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9328.304516] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9328.308557] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9328.311688] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9328.315490] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9328.321046] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9328.325727] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9328.328350] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9328.335256] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9328.347454] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9328.352341] lmv_intent_open+0x338/0x1150 [lmv] [ 9328.355496] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9328.366716] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9328.371142] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9328.373261] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9328.378636] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9328.381165] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9328.387955] ? do_raw_spin_unlock+0x75/0x190 [ 9328.390165] ? _raw_spin_unlock+0x12/0x30 [ 9328.394385] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9328.398328] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9328.402617] ll_atomic_open+0x36a/0xd70 [lustre] [ 9328.408981] lookup_open+0x356/0xb40 [ 9328.410650] do_last+0x313/0xfc0 [ 9328.413098] ? nd_jump_root+0xe5/0x160 [ 9328.416442] ? path_init+0x437/0x520 [ 9328.418113] path_openat+0xf7/0x500 [ 9328.430138] ? do_raw_spin_unlock+0x75/0x190 [ 9328.432304] do_filp_open+0x99/0x140 [ 9328.438496] ? getname_flags+0x6e/0x330 [ 9328.442509] ? __check_object_size+0xff/0x256 [ 9328.447183] ? do_raw_spin_unlock+0x75/0x190 [ 9328.449147] ? _raw_spin_unlock+0x12/0x30 [ 9328.452860] do_sys_openat2+0x2b4/0x410 [ 9328.459842] do_sys_open+0x73/0xa0 [ 9328.461422] __x64_sys_openat+0x24/0x30 [ 9328.466340] do_syscall_64+0xc1/0x440 [ 9328.469583] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9328.477588] RIP: 0033:0x7ff54af45332 [ 9328.482653] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9328.500644] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9328.510749] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 9328.517129] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 9328.520918] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 9328.526346] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 9328.530089] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 9340.058107] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 9340.062520] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9340.091710] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9340.096917] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9340.102384] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9340.109338] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9340.121278] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9340.126004] RAX: 0000000000007f17 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 9340.135313] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 9340.138735] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9340.142125] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 9340.148980] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 9340.153097] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 9340.158348] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9340.163944] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 9340.167572] Call Trace: [ 9340.168593] [ 9340.169530] ? show_regs.cold.9+0x22/0x2f [ 9340.175232] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9340.183572] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9340.185070] ? watchdog+0x40/0x40 [ 9340.187870] ? hrtimer_interrupt+0x128/0x2c0 [ 9340.190581] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9340.194374] ? apic_timer_interrupt+0xf/0x20 [ 9340.197224] [ 9340.198272] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9340.200881] do_raw_spin_lock+0x177/0x190 [ 9340.202577] _raw_spin_lock+0x21/0x30 [ 9340.204580] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9340.210816] ? do_raw_spin_unlock+0x75/0x190 [ 9340.212895] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9340.216887] ? do_wait_intr+0xf0/0xf0 [ 9340.220269] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9340.223260] mdc_close+0x53b/0xd70 [mdc] [ 9340.225811] lmv_close+0x18d/0x490 [lmv] [ 9340.227331] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 9340.230389] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9340.238650] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9340.240330] ll_md_close+0x274/0x800 [lustre] [ 9340.249967] ll_file_release+0x128/0xc50 [lustre] [ 9340.251640] __fput+0x10d/0x440 [ 9340.258334] ____fput+0x12/0x20 [ 9340.259917] task_work_run+0xc6/0x110 [ 9340.262052] exit_to_usermode_loop+0x1dd/0x1f0 [ 9340.264352] do_syscall_64+0x3d6/0x440 [ 9340.268041] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9340.271631] RIP: 0033:0x7ff54af45cd8 [ 9340.273078] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9340.281994] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9340.285626] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 9340.291880] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 9340.298544] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 9340.301900] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 9340.310512] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9344.066056] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 9344.080898] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9344.125125] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9344.130878] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9344.135599] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9344.143492] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9344.148256] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9344.166502] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9344.172258] RAX: 0000000000006b0c RBX: ffff96848811cae0 RCX: 0000000000000001 [ 9344.179487] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 9344.184714] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 9344.190701] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 9344.197872] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 9344.202133] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 9344.208544] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9344.213550] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 9344.218674] Call Trace: [ 9344.220102] [ 9344.220889] ? show_regs.cold.9+0x22/0x2f [ 9344.223418] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9344.228498] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9344.231074] ? watchdog+0x40/0x40 [ 9344.234331] ? hrtimer_interrupt+0x128/0x2c0 [ 9344.236871] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9344.238701] ? apic_timer_interrupt+0xf/0x20 [ 9344.240422] [ 9344.243053] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9344.244758] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9344.246572] do_raw_spin_lock+0x177/0x190 [ 9344.247418] _raw_spin_lock+0x21/0x30 [ 9344.252298] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9344.255553] process_one_work+0x2c8/0x700 [ 9344.258394] worker_thread+0x296/0x6e0 [ 9344.264219] ? rescuer_thread+0x570/0x570 [ 9344.265527] kthread+0x1d1/0x200 [ 9344.266228] ? set_kthread_struct+0x70/0x70 [ 9344.267115] ret_from_fork+0x1f/0x30 [ 9348.055189] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [ 9348.064214] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9348.096834] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9348.109090] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9348.115264] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9348.125807] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9348.146638] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9348.157566] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 9348.162720] RDX: 00000000000003f7 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 9348.169048] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 9348.175446] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 9348.181484] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9348.186851] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 9348.191290] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9348.194426] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 9348.203231] Call Trace: [ 9348.204732] [ 9348.205712] ? show_regs.cold.9+0x22/0x2f [ 9348.207396] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9348.209762] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9348.211982] ? watchdog+0x40/0x40 [ 9348.213980] ? hrtimer_interrupt+0x128/0x2c0 [ 9348.216398] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9348.222457] ? apic_timer_interrupt+0xf/0x20 [ 9348.225928] [ 9348.227048] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9348.231048] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9348.234029] do_raw_spin_lock+0x177/0x190 [ 9348.236229] _raw_spin_lock+0x21/0x30 [ 9348.237948] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9348.243728] obd_mod_free+0x46/0x140 [ptlrpc] [ 9348.255052] mdc_commit_open+0x91/0xb0 [mdc] [ 9348.261840] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9348.265080] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9348.277234] after_reply+0xe2c/0x1780 [ptlrpc] [ 9348.284882] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9348.289134] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9348.292149] ? do_wait_intr+0xf0/0xf0 [ 9348.294120] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9348.296709] kthread+0x1d1/0x200 [ 9348.298006] ? set_kthread_struct+0x70/0x70 [ 9348.299767] ret_from_fork+0x1f/0x30 [ 9353.184209] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7394s! [ 9353.191051] Showing busy workqueues and worker pools: [ 9353.193892] workqueue events: flags=0x0 [ 9353.199575] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9353.206572] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 9353.213419] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 9353.216295] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 9353.221205] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9353.223925] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 9353.226893] workqueue events_long: flags=0x0 [ 9353.228851] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9353.232452] pending: lu_objects_flush [obdclass] [ 9353.236897] workqueue events_unbound: flags=0x2 [ 9353.239297] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 9353.242156] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9353.245877] workqueue events_power_efficient: flags=0x80 [ 9353.247858] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9353.249967] pending: neigh_periodic_work [ 9353.251907] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9353.258512] pending: do_cache_clean, check_lifetime [ 9353.265071] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9353.270829] pending: neigh_periodic_work [ 9353.276873] workqueue writeback: flags=0x4a [ 9353.278634] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9353.281491] pending: wb_workfn, wb_workfn [ 9353.283807] workqueue xprtiod: flags=0xa [ 9353.286148] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9353.295664] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 9353.304672] workqueue ipv6_addrconf: flags=0x40008 [ 9353.309547] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9353.315474] pending: addrconf_verify_work [ 9353.324866] workqueue knbd0-recv: flags=0x1a [ 9353.330592] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9353.332839] in-flight: 657:recv_work [ 9353.345379] workqueue ptlrpc_pinger: flags=0x2 [ 9353.349212] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9353.353235] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 9353.356831] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7394s workers=4 idle: 20831 12642 28859 [ 9353.360665] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9343s workers=2 idle: 659 [ 9356.067607] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 9356.080139] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9356.117285] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9356.123198] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9356.127108] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9356.131740] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9356.157372] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9356.162910] RAX: 0000000000007c94 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 9356.170098] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 9356.180116] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9356.187691] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 9356.195159] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 9356.203461] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 9356.211883] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9356.218020] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 9356.227081] Call Trace: [ 9356.230755] [ 9356.232115] ? show_regs.cold.9+0x22/0x2f [ 9356.237013] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9356.241598] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9356.247056] ? watchdog+0x40/0x40 [ 9356.250014] ? hrtimer_interrupt+0x128/0x2c0 [ 9356.254205] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9356.259160] ? apic_timer_interrupt+0xf/0x20 [ 9356.262531] [ 9356.263829] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9356.266660] do_raw_spin_lock+0x177/0x190 [ 9356.268250] _raw_spin_lock+0x21/0x30 [ 9356.270279] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9356.273437] ? do_raw_spin_unlock+0x75/0x190 [ 9356.275846] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9356.278745] ? do_wait_intr+0xf0/0xf0 [ 9356.280767] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9356.283822] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9356.286503] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9356.290414] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9356.294729] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9356.297035] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9356.305465] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9356.310481] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9356.315433] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9356.323698] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9356.329078] lmv_intent_open+0x338/0x1150 [lmv] [ 9356.334079] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9356.343873] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9356.352133] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9356.355658] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9356.360714] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9356.369178] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9356.377246] ? do_raw_spin_unlock+0x75/0x190 [ 9356.382743] ? _raw_spin_unlock+0x12/0x30 [ 9356.386859] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9356.394129] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9356.396256] ll_atomic_open+0x36a/0xd70 [lustre] [ 9356.402093] lookup_open+0x356/0xb40 [ 9356.405230] do_last+0x313/0xfc0 [ 9356.408978] ? nd_jump_root+0xe5/0x160 [ 9356.412519] ? path_init+0x437/0x520 [ 9356.416395] path_openat+0xf7/0x500 [ 9356.419194] ? do_raw_spin_unlock+0x75/0x190 [ 9356.423652] do_filp_open+0x99/0x140 [ 9356.425443] ? getname_flags+0x6e/0x330 [ 9356.430279] ? __check_object_size+0xff/0x256 [ 9356.435117] ? do_raw_spin_unlock+0x75/0x190 [ 9356.440800] ? _raw_spin_unlock+0x12/0x30 [ 9356.443153] do_sys_openat2+0x2b4/0x410 [ 9356.447042] do_sys_open+0x73/0xa0 [ 9356.450965] __x64_sys_openat+0x24/0x30 [ 9356.454864] do_syscall_64+0xc1/0x440 [ 9356.459618] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9356.463996] RIP: 0033:0x7ff54af45332 [ 9356.467878] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9356.489753] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9356.497485] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 9356.501976] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 9356.513331] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 9356.522626] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 9356.528995] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 9368.058367] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 9368.061942] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9368.088358] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9368.093423] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9368.097440] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9368.100028] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9368.108258] RSP: 0018:ffffaf9fc1543ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9368.111713] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 9368.114769] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 9368.118595] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9368.123166] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 9368.127033] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 9368.130368] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 9368.133840] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9368.136992] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 9368.140257] Call Trace: [ 9368.141754] [ 9368.142655] ? show_regs.cold.9+0x22/0x2f [ 9368.144379] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9368.146915] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9368.149432] ? watchdog+0x40/0x40 [ 9368.151078] ? hrtimer_interrupt+0x128/0x2c0 [ 9368.153980] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9368.156725] ? apic_timer_interrupt+0xf/0x20 [ 9368.158944] [ 9368.159938] ? native_safe_halt+0x2e/0x50 [ 9368.161760] kvm_wait+0xcd/0xe0 [ 9368.163123] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9368.165409] do_raw_spin_lock+0x177/0x190 [ 9368.167132] _raw_spin_lock+0x21/0x30 [ 9368.168824] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9368.171390] ? do_raw_spin_unlock+0x75/0x190 [ 9368.173637] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9368.176321] ? do_wait_intr+0xf0/0xf0 [ 9368.177897] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9368.180001] mdc_close+0x53b/0xd70 [mdc] [ 9368.181992] lmv_close+0x18d/0x490 [lmv] [ 9368.183770] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 9368.186417] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9368.189021] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9368.191437] ll_md_close+0x274/0x800 [lustre] [ 9368.193773] ll_file_release+0x128/0xc50 [lustre] [ 9368.196110] __fput+0x10d/0x440 [ 9368.197655] ____fput+0x12/0x20 [ 9368.199089] task_work_run+0xc6/0x110 [ 9368.200625] exit_to_usermode_loop+0x1dd/0x1f0 [ 9368.203010] do_syscall_64+0x3d6/0x440 [ 9368.204762] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9368.207607] RIP: 0033:0x7ff54af45cd8 [ 9368.209241] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9368.217111] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9368.220174] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 9368.223795] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 9368.229167] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 9368.233084] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 9368.237075] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9372.061360] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 9372.064662] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9372.111099] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9372.117491] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9372.121243] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9372.130391] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9372.134875] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9372.144461] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9372.151494] RAX: 000000000000178e RBX: ffff96848811cae0 RCX: 0000000000000001 [ 9372.155625] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 9372.159877] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 9372.173754] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 9372.177279] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 9372.182235] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 9372.186723] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9372.191258] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 9372.194306] Call Trace: [ 9372.195852] [ 9372.197015] ? show_regs.cold.9+0x22/0x2f [ 9372.199649] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9372.203670] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9372.205160] ? watchdog+0x40/0x40 [ 9372.206153] ? hrtimer_interrupt+0x128/0x2c0 [ 9372.209132] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9372.213888] ? apic_timer_interrupt+0xf/0x20 [ 9372.217790] [ 9372.219229] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9372.222521] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9372.227594] do_raw_spin_lock+0x177/0x190 [ 9372.229631] _raw_spin_lock+0x21/0x30 [ 9372.231677] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9372.238802] process_one_work+0x2c8/0x700 [ 9372.246522] worker_thread+0x296/0x6e0 [ 9372.248177] ? rescuer_thread+0x570/0x570 [ 9372.250189] kthread+0x1d1/0x200 [ 9372.251776] ? set_kthread_struct+0x70/0x70 [ 9372.255632] ret_from_fork+0x1f/0x30 [ 9380.058551] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 9380.070018] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9380.131102] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9380.143068] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9380.162990] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9380.173118] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9380.185377] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9380.188744] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 9380.191911] RDX: 0000000000000af9 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 9380.196032] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 9380.199129] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 9380.203176] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9380.208121] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 9380.211946] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9380.217036] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 9380.220938] Call Trace: [ 9380.222607] [ 9380.223891] ? show_regs.cold.9+0x22/0x2f [ 9380.229029] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9380.235026] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9380.240742] ? watchdog+0x40/0x40 [ 9380.244027] ? hrtimer_interrupt+0x128/0x2c0 [ 9380.254027] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9380.257399] ? apic_timer_interrupt+0xf/0x20 [ 9380.258984] [ 9380.270752] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9380.272623] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9380.283689] do_raw_spin_lock+0x177/0x190 [ 9380.285091] _raw_spin_lock+0x21/0x30 [ 9380.294951] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9380.296950] obd_mod_free+0x46/0x140 [ptlrpc] [ 9380.313980] mdc_commit_open+0x91/0xb0 [mdc] [ 9380.322125] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9380.328218] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9380.336669] after_reply+0xe2c/0x1780 [ptlrpc] [ 9380.343010] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9380.347837] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9380.361458] ? do_wait_intr+0xf0/0xf0 [ 9380.362866] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9380.375747] kthread+0x1d1/0x200 [ 9380.376962] ? set_kthread_struct+0x70/0x70 [ 9380.383987] ret_from_fork+0x1f/0x30 [ 9383.397117] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7424s! [ 9383.406103] Showing busy workqueues and worker pools: [ 9383.408610] workqueue events: flags=0x0 [ 9383.410523] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9383.418370] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 9383.424718] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 9383.427094] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 9383.441007] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9383.446862] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 9383.452055] workqueue events_long: flags=0x0 [ 9383.454106] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9383.457688] pending: lu_objects_flush [obdclass] [ 9383.460400] workqueue events_unbound: flags=0x2 [ 9383.463149] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 9383.469045] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9383.476977] workqueue events_power_efficient: flags=0x80 [ 9383.483030] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9383.488645] pending: neigh_periodic_work [ 9383.492030] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9383.497164] pending: do_cache_clean, check_lifetime [ 9383.502316] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9383.508006] pending: neigh_periodic_work [ 9383.514195] workqueue writeback: flags=0x4a [ 9383.515793] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9383.525085] pending: wb_workfn, wb_workfn [ 9383.533484] workqueue xprtiod: flags=0xa [ 9383.538065] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9383.543610] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 9383.548473] workqueue ipv6_addrconf: flags=0x40008 [ 9383.551640] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9383.555153] pending: addrconf_verify_work [ 9383.557157] workqueue knbd0-recv: flags=0x1a [ 9383.559175] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9383.568985] in-flight: 657:recv_work [ 9383.571592] workqueue ptlrpc_pinger: flags=0x2 [ 9383.574482] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9383.577689] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 9383.579720] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7424s workers=4 idle: 20831 12642 28859 [ 9383.585631] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9373s workers=2 idle: 659 [ 9384.062270] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 9384.069721] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9384.122904] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9384.130529] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9384.140816] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9384.148821] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9384.163558] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9384.173908] RAX: 0000000000003235 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 9384.181466] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 9384.190058] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9384.204530] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 9384.215311] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 9384.218764] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 9384.223616] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9384.226652] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 9384.234439] Call Trace: [ 9384.235295] [ 9384.236728] ? show_regs.cold.9+0x22/0x2f [ 9384.240731] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9384.245574] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9384.250676] ? watchdog+0x40/0x40 [ 9384.255666] ? hrtimer_interrupt+0x128/0x2c0 [ 9384.262724] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9384.269454] ? apic_timer_interrupt+0xf/0x20 [ 9384.273922] [ 9384.277087] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9384.283978] do_raw_spin_lock+0x177/0x190 [ 9384.292407] _raw_spin_lock+0x21/0x30 [ 9384.293937] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9384.298363] ? do_raw_spin_unlock+0x75/0x190 [ 9384.301857] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9384.305002] ? do_wait_intr+0xf0/0xf0 [ 9384.311434] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9384.313461] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9384.326550] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9384.331713] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9384.338881] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9384.345051] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9384.350246] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9384.351961] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9384.358909] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9384.364761] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9384.369995] lmv_intent_open+0x338/0x1150 [lmv] [ 9384.372774] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9384.380807] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9384.385814] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9384.391989] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9384.397393] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9384.399858] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9384.401849] ? do_raw_spin_unlock+0x75/0x190 [ 9384.403837] ? _raw_spin_unlock+0x12/0x30 [ 9384.405790] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9384.408344] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9384.410778] ll_atomic_open+0x36a/0xd70 [lustre] [ 9384.414474] lookup_open+0x356/0xb40 [ 9384.418866] do_last+0x313/0xfc0 [ 9384.421590] ? nd_jump_root+0xe5/0x160 [ 9384.426300] ? path_init+0x437/0x520 [ 9384.430661] path_openat+0xf7/0x500 [ 9384.432917] ? do_raw_spin_unlock+0x75/0x190 [ 9384.434519] do_filp_open+0x99/0x140 [ 9384.437624] ? getname_flags+0x6e/0x330 [ 9384.446371] ? __check_object_size+0xff/0x256 [ 9384.453488] ? do_raw_spin_unlock+0x75/0x190 [ 9384.456422] ? _raw_spin_unlock+0x12/0x30 [ 9384.462723] do_sys_openat2+0x2b4/0x410 [ 9384.467632] do_sys_open+0x73/0xa0 [ 9384.469737] __x64_sys_openat+0x24/0x30 [ 9384.480657] do_syscall_64+0xc1/0x440 [ 9384.485777] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9384.492729] RIP: 0033:0x7ff54af45332 [ 9384.496464] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9384.518542] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9384.524477] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 9384.535389] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 9384.541640] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 9384.551599] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 9384.557299] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 9394.060029] rcu: INFO: rcu_sched self-detected stall on CPU [ 9394.063426] rcu: 0-...!: (3870738 ticks this GP) idle=98a/1/0x4000000000000002 softirq=125154/125154 fqs=959 [ 9394.068387] (t=7440220 jiffies g=253585 q=5341) [ 9394.070569] rcu: rcu_sched kthread starved for 7435180 jiffies! g253585 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [ 9394.075192] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 9394.079292] rcu: RCU grace-period kthread stack dump: [ 9394.082015] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 9394.086396] Call Trace: [ 9394.087496] __schedule+0x351/0xcb0 [ 9394.089049] schedule+0xc0/0x180 [ 9394.090667] schedule_timeout+0xb4/0x190 [ 9394.094443] ? do_raw_spin_unlock+0x75/0x190 [ 9394.096700] ? __next_timer_interrupt+0x160/0x160 [ 9394.099642] ? prepare_to_swait_event+0xf3/0x140 [ 9394.101834] rcu_gp_kthread+0x8dd/0x15b0 [ 9394.103864] ? force_qs_rnp+0x2c0/0x2c0 [ 9394.106478] kthread+0x1d1/0x200 [ 9394.108372] ? set_kthread_struct+0x70/0x70 [ 9394.110468] ret_from_fork+0x1f/0x30 [ 9394.112502] rcu: Stack dump where RCU GP kthread last ran: [ 9394.114826] NMI backtrace for cpu 0 [ 9394.116485] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9394.121545] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9394.125961] Call Trace: [ 9394.127119] [ 9394.128048] ? dump_stack+0xbb/0x10e [ 9394.129786] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 9394.132730] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 9394.135298] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 9394.138012] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 9394.140627] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 9394.142011] ? rcu_check_gp_kthread_starvation+0x16f/0x188 [ 9394.144117] ? rcu_sched_clock_irq.cold.51+0xce/0x307 [ 9394.146226] ? update_process_times+0x6d/0xc0 [ 9394.148095] ? tick_sched_handle.isra.18+0x27/0x90 [ 9394.150990] ? tick_sched_timer+0x53/0xb0 [ 9394.153532] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9394.159681] ? tick_sched_handle.isra.18+0x90/0x90 [ 9394.161529] ? hrtimer_interrupt+0x128/0x2c0 [ 9394.163402] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9394.166929] ? apic_timer_interrupt+0xf/0x20 [ 9394.168709] [ 9394.169493] ? native_safe_halt+0x2e/0x50 [ 9394.171208] ? kvm_wait+0xcd/0xe0 [ 9394.172539] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9394.175821] ? do_raw_spin_lock+0x177/0x190 [ 9394.177486] ? _raw_spin_lock+0x21/0x30 [ 9394.178936] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9394.181208] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 9394.184455] ? mdc_commit_open+0x91/0xb0 [mdc] [ 9394.187953] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9394.190398] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9394.194142] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 9394.201628] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9394.204042] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9394.210943] ? do_wait_intr+0xf0/0xf0 [ 9394.213432] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9394.217924] ? kthread+0x1d1/0x200 [ 9394.219533] ? set_kthread_struct+0x70/0x70 [ 9394.222161] ? ret_from_fork+0x1f/0x30 [ 9394.225728] NMI backtrace for cpu 0 [ 9394.227068] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9394.234752] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9394.241507] Call Trace: [ 9394.242755] [ 9394.243714] ? dump_stack+0xbb/0x10e [ 9394.246109] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 9394.250292] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 9394.252151] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 9394.255390] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 9394.258274] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 9394.262545] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 9394.264394] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 9394.267482] ? update_process_times+0x6d/0xc0 [ 9394.270358] ? tick_sched_handle.isra.18+0x27/0x90 [ 9394.272626] ? tick_sched_timer+0x53/0xb0 [ 9394.274686] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9394.282835] ? tick_sched_handle.isra.18+0x90/0x90 [ 9394.286991] ? hrtimer_interrupt+0x128/0x2c0 [ 9394.289854] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9394.292587] ? apic_timer_interrupt+0xf/0x20 [ 9394.294945] [ 9394.296083] ? native_safe_halt+0x2e/0x50 [ 9394.297897] ? kvm_wait+0xcd/0xe0 [ 9394.298933] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9394.302123] ? do_raw_spin_lock+0x177/0x190 [ 9394.304723] ? _raw_spin_lock+0x21/0x30 [ 9394.306102] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9394.311562] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 9394.314678] ? mdc_commit_open+0x91/0xb0 [mdc] [ 9394.317433] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9394.323839] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9394.325749] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 9394.327249] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9394.333558] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9394.335378] ? do_wait_intr+0xf0/0xf0 [ 9394.340457] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9394.343074] ? kthread+0x1d1/0x200 [ 9394.344284] ? set_kthread_struct+0x70/0x70 [ 9394.346556] ? ret_from_fork+0x1f/0x30 [ 9396.058014] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 9396.066182] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9396.109731] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9396.139586] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9396.147043] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9396.160670] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9396.203877] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9396.217218] RAX: 0000000000001c7e RBX: ffff96848811cae0 RCX: 0000000000000001 [ 9396.229037] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 9396.235676] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9396.246308] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 9396.256066] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 9396.260702] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 9396.272033] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9396.275390] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 9396.284708] Call Trace: [ 9396.288854] [ 9396.289242] ? show_regs.cold.9+0x22/0x2f [ 9396.290266] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9396.298354] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9396.306465] ? watchdog+0x40/0x40 [ 9396.308234] ? hrtimer_interrupt+0x128/0x2c0 [ 9396.309812] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9396.327560] ? apic_timer_interrupt+0xf/0x20 [ 9396.336261] [ 9396.337029] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9396.338936] do_raw_spin_lock+0x177/0x190 [ 9396.349562] _raw_spin_lock+0x21/0x30 [ 9396.351993] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9396.360874] ? do_raw_spin_unlock+0x75/0x190 [ 9396.363614] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9396.369853] ? do_wait_intr+0xf0/0xf0 [ 9396.374760] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9396.380768] mdc_close+0x53b/0xd70 [mdc] [ 9396.386610] lmv_close+0x18d/0x490 [lmv] [ 9396.390725] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 9396.392867] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9396.399711] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9396.402699] ll_md_close+0x274/0x800 [lustre] [ 9396.409605] ll_file_release+0x128/0xc50 [lustre] [ 9396.417332] __fput+0x10d/0x440 [ 9396.418770] ____fput+0x12/0x20 [ 9396.425561] task_work_run+0xc6/0x110 [ 9396.430071] exit_to_usermode_loop+0x1dd/0x1f0 [ 9396.438914] do_syscall_64+0x3d6/0x440 [ 9396.445148] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9396.450071] RIP: 0033:0x7ff54af45cd8 [ 9396.451573] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9396.477640] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9396.484907] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 9396.487267] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 9396.514490] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 9396.522755] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 9396.533386] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9400.059098] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 9400.062961] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9400.078550] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9400.086206] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9400.090815] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9400.095615] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9400.097404] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9400.113152] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9400.119588] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 9400.122716] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 9400.129361] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 9400.136295] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 9400.143548] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 9400.150886] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 9400.156483] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9400.159568] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 9400.163281] Call Trace: [ 9400.164436] [ 9400.165782] ? show_regs.cold.9+0x22/0x2f [ 9400.167732] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9400.171989] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9400.176713] ? watchdog+0x40/0x40 [ 9400.184823] ? hrtimer_interrupt+0x128/0x2c0 [ 9400.186639] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9400.191196] ? apic_timer_interrupt+0xf/0x20 [ 9400.194965] [ 9400.196486] ? native_safe_halt+0x2e/0x50 [ 9400.199315] kvm_wait+0xcd/0xe0 [ 9400.200826] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9400.206202] do_raw_spin_lock+0x177/0x190 [ 9400.211694] _raw_spin_lock+0x21/0x30 [ 9400.212976] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9400.217612] process_one_work+0x2c8/0x700 [ 9400.219568] worker_thread+0x296/0x6e0 [ 9400.221215] ? rescuer_thread+0x570/0x570 [ 9400.223061] kthread+0x1d1/0x200 [ 9400.224342] ? set_kthread_struct+0x70/0x70 [ 9400.225565] ret_from_fork+0x1f/0x30 [ 9412.062829] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 9412.079336] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9412.139095] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9412.152573] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9412.163433] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9412.167888] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9412.182459] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9412.190380] RAX: 0000000000007f87 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 9412.208012] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 9412.221008] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9412.238285] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 9412.250615] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 9412.257942] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 9412.263443] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9412.267983] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 9412.272069] Call Trace: [ 9412.272811] [ 9412.273471] ? show_regs.cold.9+0x22/0x2f [ 9412.284566] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9412.290053] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9412.293226] ? watchdog+0x40/0x40 [ 9412.296648] ? hrtimer_interrupt+0x128/0x2c0 [ 9412.298804] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9412.303197] ? apic_timer_interrupt+0xf/0x20 [ 9412.305276] [ 9412.306443] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9412.314230] do_raw_spin_lock+0x177/0x190 [ 9412.316419] _raw_spin_lock+0x21/0x30 [ 9412.321203] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9412.324598] ? do_raw_spin_unlock+0x75/0x190 [ 9412.327222] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9412.335524] ? do_wait_intr+0xf0/0xf0 [ 9412.339890] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9412.347207] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9412.353564] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9412.364473] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9412.368873] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9412.381423] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9412.388204] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9412.391659] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9412.395624] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9412.397772] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9412.412863] lmv_intent_open+0x338/0x1150 [lmv] [ 9412.420772] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9412.431283] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9412.435896] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9412.439664] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9412.441614] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9412.444334] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9412.447048] ? do_raw_spin_unlock+0x75/0x190 [ 9412.448731] ? _raw_spin_unlock+0x12/0x30 [ 9412.456382] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9412.460675] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9412.468920] ll_atomic_open+0x36a/0xd70 [lustre] [ 9412.474120] lookup_open+0x356/0xb40 [ 9412.477697] do_last+0x313/0xfc0 [ 9412.479179] ? nd_jump_root+0xe5/0x160 [ 9412.480767] ? path_init+0x437/0x520 [ 9412.483403] path_openat+0xf7/0x500 [ 9412.484944] ? do_raw_spin_unlock+0x75/0x190 [ 9412.487384] do_filp_open+0x99/0x140 [ 9412.490779] ? getname_flags+0x6e/0x330 [ 9412.493677] ? __check_object_size+0xff/0x256 [ 9412.499777] ? do_raw_spin_unlock+0x75/0x190 [ 9412.501380] ? _raw_spin_unlock+0x12/0x30 [ 9412.502845] do_sys_openat2+0x2b4/0x410 [ 9412.511842] do_sys_open+0x73/0xa0 [ 9412.514547] __x64_sys_openat+0x24/0x30 [ 9412.519707] do_syscall_64+0xc1/0x440 [ 9412.522354] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9412.530545] RIP: 0033:0x7ff54af45332 [ 9412.533235] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9412.543337] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9412.546545] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 9412.549596] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 9412.554224] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 9412.560374] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 9412.568660] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 9413.599754] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7454s! [ 9413.604635] Showing busy workqueues and worker pools: [ 9413.608882] workqueue events: flags=0x0 [ 9413.611380] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9413.619005] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 9413.624821] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 9413.628269] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 9413.635000] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9413.638790] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 9413.643186] workqueue events_long: flags=0x0 [ 9413.645852] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9413.649657] pending: lu_objects_flush [obdclass] [ 9413.652617] workqueue events_unbound: flags=0x2 [ 9413.655406] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 9413.658925] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9413.663274] workqueue events_power_efficient: flags=0x80 [ 9413.666032] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9413.669834] pending: neigh_periodic_work [ 9413.672157] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9413.675263] pending: do_cache_clean, check_lifetime [ 9413.677675] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9413.680992] pending: neigh_periodic_work [ 9413.682861] workqueue writeback: flags=0x4a [ 9413.684670] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9413.687708] pending: wb_workfn, wb_workfn [ 9413.690917] workqueue xprtiod: flags=0xa [ 9413.693152] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9413.696030] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 9413.699594] workqueue ipv6_addrconf: flags=0x40008 [ 9413.701715] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9413.704096] pending: addrconf_verify_work [ 9413.706666] workqueue knbd0-recv: flags=0x1a [ 9413.708419] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9413.711032] in-flight: 657:recv_work [ 9413.712390] workqueue ptlrpc_pinger: flags=0x2 [ 9413.714198] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9413.717063] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 9413.720025] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7454s workers=4 idle: 20831 12642 28859 [ 9413.723808] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9403s workers=2 idle: 659 [ 9424.058026] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 9424.061122] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9424.105525] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9424.113586] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9424.125386] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9424.133695] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9424.155140] RSP: 0018:ffffaf9fc1543ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9424.169621] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 9424.178484] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 9424.185592] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9424.195703] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 9424.204749] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 9424.213254] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 9424.223902] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9424.225793] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 9424.237605] Call Trace: [ 9424.241858] [ 9424.243583] ? show_regs.cold.9+0x22/0x2f [ 9424.251246] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9424.254580] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9424.259351] ? watchdog+0x40/0x40 [ 9424.261202] ? hrtimer_interrupt+0x128/0x2c0 [ 9424.263780] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9424.269876] ? apic_timer_interrupt+0xf/0x20 [ 9424.275514] [ 9424.276670] ? native_safe_halt+0x2e/0x50 [ 9424.278687] kvm_wait+0xcd/0xe0 [ 9424.281681] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9424.284813] do_raw_spin_lock+0x177/0x190 [ 9424.287621] _raw_spin_lock+0x21/0x30 [ 9424.290593] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9424.296140] ? do_raw_spin_unlock+0x75/0x190 [ 9424.298650] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9424.306180] ? do_wait_intr+0xf0/0xf0 [ 9424.310493] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9424.317352] mdc_close+0x53b/0xd70 [mdc] [ 9424.322846] lmv_close+0x18d/0x490 [lmv] [ 9424.325634] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 9424.336811] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9424.342024] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9424.349643] ll_md_close+0x274/0x800 [lustre] [ 9424.353829] ll_file_release+0x128/0xc50 [lustre] [ 9424.365550] __fput+0x10d/0x440 [ 9424.367820] ____fput+0x12/0x20 [ 9424.371309] task_work_run+0xc6/0x110 [ 9424.374084] exit_to_usermode_loop+0x1dd/0x1f0 [ 9424.378369] do_syscall_64+0x3d6/0x440 [ 9424.381525] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9424.385925] RIP: 0033:0x7ff54af45cd8 [ 9424.387758] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9424.405469] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9424.413925] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 9424.417208] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 9424.426328] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 9424.432589] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 9424.435711] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9428.063792] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 9428.073845] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9428.130466] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9428.150401] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9428.159940] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9428.166522] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9428.172672] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9428.208911] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9428.213933] RAX: 00000000000011c1 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 9428.215705] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 9428.225991] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 9428.228707] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 9428.231023] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 9428.232730] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 9428.235692] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9428.238771] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 9428.242885] Call Trace: [ 9428.244406] [ 9428.245225] ? show_regs.cold.9+0x22/0x2f [ 9428.247141] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9428.249535] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9428.251659] ? watchdog+0x40/0x40 [ 9428.253230] ? hrtimer_interrupt+0x128/0x2c0 [ 9428.255187] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9428.257409] ? apic_timer_interrupt+0xf/0x20 [ 9428.262739] [ 9428.263852] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9428.266760] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9428.270217] do_raw_spin_lock+0x177/0x190 [ 9428.272187] _raw_spin_lock+0x21/0x30 [ 9428.277978] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9428.283140] process_one_work+0x2c8/0x700 [ 9428.284492] worker_thread+0x296/0x6e0 [ 9428.285776] ? rescuer_thread+0x570/0x570 [ 9428.291679] kthread+0x1d1/0x200 [ 9428.292823] ? set_kthread_struct+0x70/0x70 [ 9428.295693] ret_from_fork+0x1f/0x30 [ 9440.056519] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 9440.061046] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 9440.074304] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9440.074386] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9440.074394] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9440.074397] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9440.074414] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9440.074418] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9440.074423] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 9440.074425] RDX: 0000000000002f2c RSI: 0000000000000003 RDI: ffff96848811cae0 [ 9440.074428] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 9440.074430] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 9440.074433] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9440.074436] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 9440.074439] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9440.074442] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 9440.074451] Call Trace: [ 9440.074458] [ 9440.074478] ? show_regs.cold.9+0x22/0x2f [ 9440.074490] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9440.074497] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9440.074502] ? watchdog+0x40/0x40 [ 9440.074506] ? hrtimer_interrupt+0x128/0x2c0 [ 9440.074511] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9440.074519] ? apic_timer_interrupt+0xf/0x20 [ 9440.074524] [ 9440.074527] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9440.074532] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9440.074536] do_raw_spin_lock+0x177/0x190 [ 9440.074540] _raw_spin_lock+0x21/0x30 [ 9440.074546] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9440.083949] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9440.084037] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9440.084044] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9440.084047] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9440.084060] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9440.084064] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 9440.084069] RAX: 0000000000003584 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 9440.084071] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 9440.084074] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9440.084077] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 9440.084079] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 9440.084083] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 9440.084086] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9440.084089] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 9440.084097] Call Trace: [ 9440.084102] [ 9440.084107] ? show_regs.cold.9+0x22/0x2f [ 9440.084116] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9440.084123] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9440.084132] ? watchdog+0x40/0x40 [ 9440.084136] ? hrtimer_interrupt+0x128/0x2c0 [ 9440.084140] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9440.084148] ? apic_timer_interrupt+0xf/0x20 [ 9440.084152] [ 9440.084154] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9440.084161] do_raw_spin_lock+0x177/0x190 [ 9440.084177] _raw_spin_lock+0x21/0x30 [ 9440.084183] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9440.084578] ? do_raw_spin_unlock+0x75/0x190 [ 9440.084584] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9440.085008] ? do_wait_intr+0xf0/0xf0 [ 9440.085016] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9440.139995] obd_mod_free+0x46/0x140 [ptlrpc] [ 9440.140373] mdc_commit_open+0x91/0xb0 [mdc] [ 9440.140411] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9440.140603] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9440.141009] after_reply+0xe2c/0x1780 [ptlrpc] [ 9440.141273] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9440.141478] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9440.141748] ? do_wait_intr+0xf0/0xf0 [ 9440.141760] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9440.142151] kthread+0x1d1/0x200 [ 9440.142157] ? set_kthread_struct+0x70/0x70 [ 9440.142161] ret_from_fork+0x1f/0x30 [ 9440.549644] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9440.554572] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9440.563150] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9440.571837] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9440.579876] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9440.590102] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9440.596597] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9440.605088] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9440.611435] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9440.617839] lmv_intent_open+0x338/0x1150 [lmv] [ 9440.620948] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9440.624500] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9440.628178] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9440.631542] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9440.634603] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9440.638197] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9440.641325] ? do_raw_spin_unlock+0x75/0x190 [ 9440.646538] ? _raw_spin_unlock+0x12/0x30 [ 9440.649265] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9440.652913] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9440.657812] ll_atomic_open+0x36a/0xd70 [lustre] [ 9440.661351] lookup_open+0x356/0xb40 [ 9440.665615] do_last+0x313/0xfc0 [ 9440.667369] ? nd_jump_root+0xe5/0x160 [ 9440.679151] ? path_init+0x437/0x520 [ 9440.684374] path_openat+0xf7/0x500 [ 9440.690468] ? do_raw_spin_unlock+0x75/0x190 [ 9440.693600] do_filp_open+0x99/0x140 [ 9440.697286] ? getname_flags+0x6e/0x330 [ 9440.701169] ? __check_object_size+0xff/0x256 [ 9440.704071] ? do_raw_spin_unlock+0x75/0x190 [ 9440.710186] ? _raw_spin_unlock+0x12/0x30 [ 9440.712599] do_sys_openat2+0x2b4/0x410 [ 9440.718210] do_sys_open+0x73/0xa0 [ 9440.721565] __x64_sys_openat+0x24/0x30 [ 9440.726724] do_syscall_64+0xc1/0x440 [ 9440.730562] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9440.738116] RIP: 0033:0x7ff54af45332 [ 9440.742889] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9440.767245] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9440.776208] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 9440.784823] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 9440.795442] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 9440.804996] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 9440.814635] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 9443.807386] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7484s! [ 9443.811850] Showing busy workqueues and worker pools: [ 9443.814826] workqueue events: flags=0x0 [ 9443.817174] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9443.821507] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 9443.828394] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 9443.832542] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 9443.843381] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9443.848547] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 9443.853211] workqueue events_long: flags=0x0 [ 9443.856282] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9443.859395] pending: lu_objects_flush [obdclass] [ 9443.864505] workqueue events_unbound: flags=0x2 [ 9443.869941] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 9443.877373] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9443.884517] workqueue events_power_efficient: flags=0x80 [ 9443.888114] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9443.892662] pending: neigh_periodic_work [ 9443.896722] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9443.900801] pending: do_cache_clean, check_lifetime [ 9443.906221] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9443.913153] pending: neigh_periodic_work [ 9443.917874] workqueue writeback: flags=0x4a [ 9443.920403] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9443.924298] pending: wb_workfn, wb_workfn [ 9443.928185] workqueue xprtiod: flags=0xa [ 9443.930464] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9443.935765] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 9443.939684] workqueue ipv6_addrconf: flags=0x40008 [ 9443.942884] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9443.946941] pending: addrconf_verify_work [ 9443.949632] workqueue knbd0-recv: flags=0x1a [ 9443.953368] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9443.957449] in-flight: 657:recv_work [ 9443.960468] workqueue ptlrpc_pinger: flags=0x2 [ 9443.964410] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9443.969382] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 9443.973165] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7485s workers=4 idle: 20831 12642 28859 [ 9443.978131] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9434s workers=2 idle: 659 [ 9452.058637] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 9452.064112] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9452.118647] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9452.131522] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9452.137736] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9452.140973] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9452.163014] RSP: 0018:ffffaf9fc1543ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9452.171933] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 9452.177801] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 9452.182541] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9452.190798] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 9452.196512] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 9452.198928] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 9452.207042] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9452.211703] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 9452.226313] Call Trace: [ 9452.228835] [ 9452.233818] ? show_regs.cold.9+0x22/0x2f [ 9452.239380] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9452.245638] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9452.247668] ? watchdog+0x40/0x40 [ 9452.251955] ? hrtimer_interrupt+0x128/0x2c0 [ 9452.257748] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9452.270023] ? apic_timer_interrupt+0xf/0x20 [ 9452.271613] [ 9452.278667] ? native_safe_halt+0x2e/0x50 [ 9452.285937] kvm_wait+0xcd/0xe0 [ 9452.290259] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9452.305356] do_raw_spin_lock+0x177/0x190 [ 9452.308844] _raw_spin_lock+0x21/0x30 [ 9452.310060] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9452.313544] ? do_raw_spin_unlock+0x75/0x190 [ 9452.317955] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9452.326276] ? do_wait_intr+0xf0/0xf0 [ 9452.328462] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9452.334468] mdc_close+0x53b/0xd70 [mdc] [ 9452.339780] lmv_close+0x18d/0x490 [lmv] [ 9452.341309] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 9452.346263] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9452.352028] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9452.356845] ll_md_close+0x274/0x800 [lustre] [ 9452.359749] ll_file_release+0x128/0xc50 [lustre] [ 9452.367999] __fput+0x10d/0x440 [ 9452.370360] ____fput+0x12/0x20 [ 9452.375254] task_work_run+0xc6/0x110 [ 9452.377732] exit_to_usermode_loop+0x1dd/0x1f0 [ 9452.386021] do_syscall_64+0x3d6/0x440 [ 9452.394520] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9452.402060] RIP: 0033:0x7ff54af45cd8 [ 9452.403367] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9452.443410] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9452.464015] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 9452.487500] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 9452.497059] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 9452.501713] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 9452.508860] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9456.059023] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 9456.061608] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9456.078854] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9456.083099] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9456.086769] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9456.089363] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9456.091109] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9456.098089] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9456.101162] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 9456.103681] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 9456.106124] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 9456.108661] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 9456.111161] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 9456.113640] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 9456.116583] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9456.118467] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 9456.120833] Call Trace: [ 9456.121660] [ 9456.122361] ? show_regs.cold.9+0x22/0x2f [ 9456.123867] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9456.126633] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9456.128351] ? watchdog+0x40/0x40 [ 9456.129518] ? hrtimer_interrupt+0x128/0x2c0 [ 9456.131280] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9456.132942] ? apic_timer_interrupt+0xf/0x20 [ 9456.134588] [ 9456.135327] ? native_safe_halt+0x2e/0x50 [ 9456.136797] kvm_wait+0xcd/0xe0 [ 9456.137872] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9456.140207] do_raw_spin_lock+0x177/0x190 [ 9456.141735] _raw_spin_lock+0x21/0x30 [ 9456.143065] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9456.145205] process_one_work+0x2c8/0x700 [ 9456.146641] worker_thread+0x296/0x6e0 [ 9456.148128] ? rescuer_thread+0x570/0x570 [ 9456.149633] kthread+0x1d1/0x200 [ 9456.151138] ? set_kthread_struct+0x70/0x70 [ 9456.152623] ret_from_fork+0x1f/0x30 [ 9468.055026] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 9468.057222] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9468.060030] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 9468.096318] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9468.103808] Modules linked in: lustre(O) [ 9468.121488] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9468.121500] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9468.121515] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9468.121519] RSP: 0000:ffffaf9fc1b97bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9468.121525] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 9468.121527] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 9468.121529] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 9468.121532] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 9468.121535] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9468.121538] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 9468.121541] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9468.121544] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 9468.121552] Call Trace: [ 9468.121558] [ 9468.121564] ? show_regs.cold.9+0x22/0x2f [ 9468.121579] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9468.121617] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9468.121622] ? watchdog+0x40/0x40 [ 9468.121627] ? hrtimer_interrupt+0x128/0x2c0 [ 9468.121631] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9468.121639] ? apic_timer_interrupt+0xf/0x20 [ 9468.121644] [ 9468.121647] ? native_safe_halt+0x2e/0x50 [ 9468.121650] kvm_wait+0xcd/0xe0 [ 9468.121658] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9468.121665] do_raw_spin_lock+0x177/0x190 [ 9468.121669] _raw_spin_lock+0x21/0x30 [ 9468.121676] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9468.122232] obd_mod_free+0x46/0x140 [ptlrpc] [ 9468.123496] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9468.123567] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9468.123573] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9468.123576] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9468.123595] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9468.123599] RSP: 0000:ffffaf9fc15234f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9468.123604] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 9468.123606] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 9468.123609] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9468.123611] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 9468.123614] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 9468.123617] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 9468.123620] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9468.123623] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 9468.123630] Call Trace: [ 9468.123633] [ 9468.123636] ? show_regs.cold.9+0x22/0x2f [ 9468.123646] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9468.123652] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9468.123656] ? watchdog+0x40/0x40 [ 9468.123661] ? hrtimer_interrupt+0x128/0x2c0 [ 9468.123665] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9468.123672] ? apic_timer_interrupt+0xf/0x20 [ 9468.123676] [ 9468.123679] ? native_safe_halt+0x2e/0x50 [ 9468.123683] kvm_wait+0xcd/0xe0 [ 9468.123689] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9468.123696] do_raw_spin_lock+0x177/0x190 [ 9468.123700] _raw_spin_lock+0x21/0x30 [ 9468.123704] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9468.123991] ? do_raw_spin_unlock+0x75/0x190 [ 9468.123996] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9468.124208] ? do_wait_intr+0xf0/0xf0 [ 9468.124216] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9468.124750] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9468.132710] mdc_commit_open+0x91/0xb0 [mdc] [ 9468.134283] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9468.145872] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9468.146281] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9468.153644] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9468.153779] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9468.153821] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9468.153849] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9468.153873] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9468.153950] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9468.154258] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9468.154286] lmv_intent_open+0x338/0x1150 [lmv] [ 9468.154317] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9468.154402] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9468.159933] after_reply+0xe2c/0x1780 [ptlrpc] [ 9468.166734] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9468.172272] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9468.172865] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9468.177664] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9468.181615] ? do_wait_intr+0xf0/0xf0 [ 9468.181642] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9468.182171] kthread+0x1d1/0x200 [ 9468.182179] ? set_kthread_struct+0x70/0x70 [ 9468.182184] ret_from_fork+0x1f/0x30 [ 9468.472720] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9468.475040] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9468.477719] ? do_raw_spin_unlock+0x75/0x190 [ 9468.479814] ? _raw_spin_unlock+0x12/0x30 [ 9468.483988] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9468.488926] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9468.491265] ll_atomic_open+0x36a/0xd70 [lustre] [ 9468.498052] lookup_open+0x356/0xb40 [ 9468.499657] do_last+0x313/0xfc0 [ 9468.502218] ? nd_jump_root+0xe5/0x160 [ 9468.505096] ? path_init+0x437/0x520 [ 9468.507865] path_openat+0xf7/0x500 [ 9468.509761] ? do_raw_spin_unlock+0x75/0x190 [ 9468.512365] do_filp_open+0x99/0x140 [ 9468.514202] ? getname_flags+0x6e/0x330 [ 9468.516325] ? __check_object_size+0xff/0x256 [ 9468.517752] ? do_raw_spin_unlock+0x75/0x190 [ 9468.520085] ? _raw_spin_unlock+0x12/0x30 [ 9468.523704] do_sys_openat2+0x2b4/0x410 [ 9468.527209] do_sys_open+0x73/0xa0 [ 9468.530336] __x64_sys_openat+0x24/0x30 [ 9468.531983] do_syscall_64+0xc1/0x440 [ 9468.533829] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9468.537834] RIP: 0033:0x7ff54af45332 [ 9468.540825] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9468.554094] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9468.561858] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 9468.566976] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 9468.572907] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 9468.576436] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 9468.580168] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 9474.015034] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7515s! [ 9474.021066] Showing busy workqueues and worker pools: [ 9474.025995] workqueue events: flags=0x0 [ 9474.028906] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9474.031896] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 9474.040286] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 9474.047313] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 9474.057859] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9474.064169] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 9474.067296] workqueue events_long: flags=0x0 [ 9474.069584] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9474.072978] pending: lu_objects_flush [obdclass] [ 9474.076808] workqueue events_unbound: flags=0x2 [ 9474.081505] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 9474.088256] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9474.096567] workqueue events_power_efficient: flags=0x80 [ 9474.101758] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9474.109502] pending: neigh_periodic_work [ 9474.112362] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9474.116936] pending: do_cache_clean, check_lifetime [ 9474.119983] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9474.131165] pending: neigh_periodic_work [ 9474.134515] workqueue writeback: flags=0x4a [ 9474.141343] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9474.146929] pending: wb_workfn, wb_workfn [ 9474.149391] workqueue xprtiod: flags=0xa [ 9474.152165] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9474.154709] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 9474.164178] workqueue ipv6_addrconf: flags=0x40008 [ 9474.167848] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9474.174895] pending: addrconf_verify_work [ 9474.177458] workqueue knbd0-recv: flags=0x1a [ 9474.179696] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9474.183121] in-flight: 657:recv_work [ 9474.187867] workqueue ptlrpc_pinger: flags=0x2 [ 9474.189199] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9474.194284] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 9474.197219] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7515s workers=4 idle: 20831 12642 28859 [ 9474.208198] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9464s workers=2 idle: 659 [ 9480.060816] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 9480.063865] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9480.092921] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9480.098792] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9480.103855] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9480.107734] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9480.122156] RSP: 0018:ffffaf9fc1543ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9480.125603] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 9480.133784] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 9480.138247] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9480.141919] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 9480.145728] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 9480.149155] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 9480.155494] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9480.158822] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 9480.162879] Call Trace: [ 9480.164092] [ 9480.165018] ? show_regs.cold.9+0x22/0x2f [ 9480.167109] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9480.169509] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9480.171912] ? watchdog+0x40/0x40 [ 9480.173491] ? hrtimer_interrupt+0x128/0x2c0 [ 9480.175794] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9480.178860] ? apic_timer_interrupt+0xf/0x20 [ 9480.181517] [ 9480.182494] ? native_safe_halt+0x2e/0x50 [ 9480.184431] kvm_wait+0xcd/0xe0 [ 9480.185782] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9480.188840] do_raw_spin_lock+0x177/0x190 [ 9480.190711] _raw_spin_lock+0x21/0x30 [ 9480.191802] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9480.193726] ? do_raw_spin_unlock+0x75/0x190 [ 9480.195187] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9480.197952] ? do_wait_intr+0xf0/0xf0 [ 9480.199142] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9480.200949] mdc_close+0x53b/0xd70 [mdc] [ 9480.204040] lmv_close+0x18d/0x490 [lmv] [ 9480.205780] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 9480.208243] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9480.210452] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9480.212543] ll_md_close+0x274/0x800 [lustre] [ 9480.214168] ll_file_release+0x128/0xc50 [lustre] [ 9480.215824] __fput+0x10d/0x440 [ 9480.216674] ____fput+0x12/0x20 [ 9480.217759] task_work_run+0xc6/0x110 [ 9480.219252] exit_to_usermode_loop+0x1dd/0x1f0 [ 9480.220946] do_syscall_64+0x3d6/0x440 [ 9480.222340] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9480.224150] RIP: 0033:0x7ff54af45cd8 [ 9480.225667] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9480.231389] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9480.234518] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 9480.237597] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 9480.240283] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 9480.242933] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 9480.245578] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9484.059014] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 9484.062275] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9484.085599] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9484.092152] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9484.096845] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9484.100729] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9484.103278] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9484.111889] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9484.115152] RAX: 00000000000063d1 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 9484.117845] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 9484.125790] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 9484.134420] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 9484.138523] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 9484.144144] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 9484.150484] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9484.153307] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 9484.156734] Call Trace: [ 9484.158543] [ 9484.159576] ? show_regs.cold.9+0x22/0x2f [ 9484.161480] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9484.164034] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9484.166128] ? watchdog+0x40/0x40 [ 9484.167647] ? hrtimer_interrupt+0x128/0x2c0 [ 9484.170078] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9484.176513] ? apic_timer_interrupt+0xf/0x20 [ 9484.179069] [ 9484.180223] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9484.183202] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9484.186173] do_raw_spin_lock+0x177/0x190 [ 9484.188477] _raw_spin_lock+0x21/0x30 [ 9484.190571] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9484.193680] process_one_work+0x2c8/0x700 [ 9484.195972] worker_thread+0x296/0x6e0 [ 9484.198094] ? rescuer_thread+0x570/0x570 [ 9484.200198] kthread+0x1d1/0x200 [ 9484.201902] ? set_kthread_struct+0x70/0x70 [ 9484.204238] ret_from_fork+0x1f/0x30 [ 9496.060794] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 9496.064851] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9496.104738] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9496.110907] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9496.117750] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9496.122976] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9496.134838] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9496.141959] RAX: 0000000000004107 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 9496.146153] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 9496.151571] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9496.160975] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 9496.164632] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 9496.168787] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 9496.172906] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9496.175461] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 9496.179868] Call Trace: [ 9496.184049] [ 9496.187234] ? show_regs.cold.9+0x22/0x2f [ 9496.196855] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9496.198679] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9496.205274] ? watchdog+0x40/0x40 [ 9496.207249] ? hrtimer_interrupt+0x128/0x2c0 [ 9496.212211] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9496.221896] ? apic_timer_interrupt+0xf/0x20 [ 9496.223450] [ 9496.224129] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9496.230210] do_raw_spin_lock+0x177/0x190 [ 9496.237907] _raw_spin_lock+0x21/0x30 [ 9496.239485] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9496.251486] ? do_raw_spin_unlock+0x75/0x190 [ 9496.259564] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9496.266397] ? do_wait_intr+0xf0/0xf0 [ 9496.270859] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9496.272817] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9496.293105] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9496.301992] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9496.311954] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9496.314042] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9496.324124] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9496.328103] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9496.335592] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9496.340014] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9496.348659] lmv_intent_open+0x338/0x1150 [lmv] [ 9496.365915] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9496.371546] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9496.380038] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9496.384806] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9496.396942] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9496.413762] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9496.424101] ? do_raw_spin_unlock+0x75/0x190 [ 9496.427657] ? _raw_spin_unlock+0x12/0x30 [ 9496.429847] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9496.439913] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9496.446101] ll_atomic_open+0x36a/0xd70 [lustre] [ 9496.454687] lookup_open+0x356/0xb40 [ 9496.460917] do_last+0x313/0xfc0 [ 9496.469568] ? nd_jump_root+0xe5/0x160 [ 9496.473445] ? path_init+0x437/0x520 [ 9496.477827] path_openat+0xf7/0x500 [ 9496.481415] ? do_raw_spin_unlock+0x75/0x190 [ 9496.490365] do_filp_open+0x99/0x140 [ 9496.493985] ? getname_flags+0x6e/0x330 [ 9496.503605] ? __check_object_size+0xff/0x256 [ 9496.514889] ? do_raw_spin_unlock+0x75/0x190 [ 9496.518478] ? _raw_spin_unlock+0x12/0x30 [ 9496.524979] do_sys_openat2+0x2b4/0x410 [ 9496.533525] do_sys_open+0x73/0xa0 [ 9496.537736] __x64_sys_openat+0x24/0x30 [ 9496.545001] do_syscall_64+0xc1/0x440 [ 9496.549921] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9496.567691] RIP: 0033:0x7ff54af45332 [ 9496.571115] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9496.593022] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9496.596097] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 9496.604994] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 9496.613010] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 9496.616346] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 9496.620935] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 9500.055013] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 9500.058904] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9500.081448] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9500.086805] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9500.091050] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9500.094433] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9500.103773] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9500.109634] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 9500.114584] RDX: 0000000000002a79 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 9500.118989] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 9500.123021] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 9500.137648] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9500.154787] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 9500.161703] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9500.177607] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 9500.184854] Call Trace: [ 9500.187363] [ 9500.189595] ? show_regs.cold.9+0x22/0x2f [ 9500.190737] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9500.193536] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9500.196826] ? watchdog+0x40/0x40 [ 9500.197652] ? hrtimer_interrupt+0x128/0x2c0 [ 9500.203983] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9500.206488] ? apic_timer_interrupt+0xf/0x20 [ 9500.208544] [ 9500.215679] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9500.217883] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9500.226393] do_raw_spin_lock+0x177/0x190 [ 9500.229971] _raw_spin_lock+0x21/0x30 [ 9500.231542] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9500.233805] obd_mod_free+0x46/0x140 [ptlrpc] [ 9500.235898] mdc_commit_open+0x91/0xb0 [mdc] [ 9500.253950] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9500.261612] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9500.271498] after_reply+0xe2c/0x1780 [ptlrpc] [ 9500.284110] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9500.285526] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9500.302060] ? do_wait_intr+0xf0/0xf0 [ 9500.305776] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9500.315675] kthread+0x1d1/0x200 [ 9500.318544] ? set_kthread_struct+0x70/0x70 [ 9500.327766] ret_from_fork+0x1f/0x30 [ 9504.227470] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7545s! [ 9504.232812] Showing busy workqueues and worker pools: [ 9504.234844] workqueue events: flags=0x0 [ 9504.238837] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9504.245023] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 9504.254890] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 9504.261697] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 9504.274992] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9504.283115] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 9504.288687] workqueue events_long: flags=0x0 [ 9504.294190] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9504.299913] pending: lu_objects_flush [obdclass] [ 9504.304711] workqueue events_unbound: flags=0x2 [ 9504.310457] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 9504.315527] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9504.320966] workqueue events_power_efficient: flags=0x80 [ 9504.323446] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9504.326042] pending: neigh_periodic_work [ 9504.327891] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9504.330712] pending: do_cache_clean, check_lifetime [ 9504.337403] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9504.343450] pending: neigh_periodic_work [ 9504.345511] workqueue writeback: flags=0x4a [ 9504.347012] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9504.350028] pending: wb_workfn, wb_workfn [ 9504.357593] workqueue xprtiod: flags=0xa [ 9504.362354] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9504.369622] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 9504.378119] workqueue ipv6_addrconf: flags=0x40008 [ 9504.381146] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9504.385482] pending: addrconf_verify_work [ 9504.388635] workqueue knbd0-recv: flags=0x1a [ 9504.393693] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9504.399534] in-flight: 657:recv_work [ 9504.404044] workqueue ptlrpc_pinger: flags=0x2 [ 9504.407513] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9504.411917] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 9504.421890] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7545s workers=4 idle: 20831 12642 28859 [ 9504.440651] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9494s workers=2 idle: 659 [ 9508.058089] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [ 9508.061765] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9508.093828] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9508.096569] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9508.098596] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9508.100152] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9508.109521] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9508.112247] RAX: 000000000000538c RBX: ffff96848811cae0 RCX: 0000000000000001 [ 9508.116061] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 9508.119881] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9508.124778] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 9508.128470] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 9508.132771] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 9508.137184] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9508.139904] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 9508.143462] Call Trace: [ 9508.145690] [ 9508.146496] ? show_regs.cold.9+0x22/0x2f [ 9508.148346] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9508.150482] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9508.152860] ? watchdog+0x40/0x40 [ 9508.154715] ? hrtimer_interrupt+0x128/0x2c0 [ 9508.156655] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9508.159117] ? apic_timer_interrupt+0xf/0x20 [ 9508.161397] [ 9508.162548] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9508.164638] do_raw_spin_lock+0x177/0x190 [ 9508.166432] _raw_spin_lock+0x21/0x30 [ 9508.167653] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9508.169942] ? do_raw_spin_unlock+0x75/0x190 [ 9508.171527] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9508.173878] ? do_wait_intr+0xf0/0xf0 [ 9508.175546] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9508.178533] mdc_close+0x53b/0xd70 [mdc] [ 9508.180573] lmv_close+0x18d/0x490 [lmv] [ 9508.182310] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 9508.185148] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9508.187649] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9508.189851] ll_md_close+0x274/0x800 [lustre] [ 9508.191833] ll_file_release+0x128/0xc50 [lustre] [ 9508.195780] __fput+0x10d/0x440 [ 9508.197233] ____fput+0x12/0x20 [ 9508.198787] task_work_run+0xc6/0x110 [ 9508.200580] exit_to_usermode_loop+0x1dd/0x1f0 [ 9508.204170] do_syscall_64+0x3d6/0x440 [ 9508.205879] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9508.209389] RIP: 0033:0x7ff54af45cd8 [ 9508.210954] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9508.217894] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9508.223494] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 9508.225664] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 9508.227953] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 9508.230611] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 9508.232276] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9512.059739] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 9512.071819] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9512.136642] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9512.160547] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9512.178126] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9512.191907] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9512.203104] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9512.235720] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9512.238659] RAX: 0000000000006986 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 9512.248658] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 9512.259254] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 9512.265895] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 9512.274270] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 9512.280107] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 9512.290102] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9512.292077] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 9512.297429] Call Trace: [ 9512.301661] [ 9512.302837] ? show_regs.cold.9+0x22/0x2f [ 9512.308003] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9512.309367] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9512.315804] ? watchdog+0x40/0x40 [ 9512.318353] ? hrtimer_interrupt+0x128/0x2c0 [ 9512.321396] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9512.325237] ? apic_timer_interrupt+0xf/0x20 [ 9512.328746] [ 9512.331111] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9512.335738] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9512.341920] do_raw_spin_lock+0x177/0x190 [ 9512.343671] _raw_spin_lock+0x21/0x30 [ 9512.345951] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9512.348544] process_one_work+0x2c8/0x700 [ 9512.350374] worker_thread+0x296/0x6e0 [ 9512.351674] ? rescuer_thread+0x570/0x570 [ 9512.353182] kthread+0x1d1/0x200 [ 9512.356857] ? set_kthread_struct+0x70/0x70 [ 9512.360460] ret_from_fork+0x1f/0x30 [ 9524.061532] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 9524.067567] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9524.116694] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9524.126305] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9524.131220] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9524.135705] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9524.157274] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9524.168019] RAX: 0000000000005f87 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 9524.173523] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 9524.179614] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9524.184821] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 9524.198563] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 9524.213857] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 9524.217509] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9524.222544] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 9524.231691] Call Trace: [ 9524.237350] [ 9524.242748] ? show_regs.cold.9+0x22/0x2f [ 9524.247590] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9524.249745] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9524.253219] ? watchdog+0x40/0x40 [ 9524.259432] ? hrtimer_interrupt+0x128/0x2c0 [ 9524.266494] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9524.271158] ? apic_timer_interrupt+0xf/0x20 [ 9524.276744] [ 9524.277333] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9524.279089] do_raw_spin_lock+0x177/0x190 [ 9524.283468] _raw_spin_lock+0x21/0x30 [ 9524.287629] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9524.294773] ? do_raw_spin_unlock+0x75/0x190 [ 9524.296100] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9524.299692] ? do_wait_intr+0xf0/0xf0 [ 9524.309496] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9524.314259] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9524.319568] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9524.331819] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9524.333782] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9524.342427] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9524.347344] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9524.354489] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9524.359907] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9524.363487] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9524.369293] lmv_intent_open+0x338/0x1150 [lmv] [ 9524.379407] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9524.384466] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9524.389362] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9524.390891] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9524.395647] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9524.398151] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9524.401111] ? do_raw_spin_unlock+0x75/0x190 [ 9524.403907] ? _raw_spin_unlock+0x12/0x30 [ 9524.406622] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9524.411468] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9524.414138] ll_atomic_open+0x36a/0xd70 [lustre] [ 9524.421703] lookup_open+0x356/0xb40 [ 9524.427673] do_last+0x313/0xfc0 [ 9524.433863] ? nd_jump_root+0xe5/0x160 [ 9524.439018] ? path_init+0x437/0x520 [ 9524.442983] path_openat+0xf7/0x500 [ 9524.444033] ? do_raw_spin_unlock+0x75/0x190 [ 9524.444972] do_filp_open+0x99/0x140 [ 9524.445757] ? getname_flags+0x6e/0x330 [ 9524.452236] ? __check_object_size+0xff/0x256 [ 9524.456011] ? do_raw_spin_unlock+0x75/0x190 [ 9524.459619] ? _raw_spin_unlock+0x12/0x30 [ 9524.467249] do_sys_openat2+0x2b4/0x410 [ 9524.477625] do_sys_open+0x73/0xa0 [ 9524.482595] __x64_sys_openat+0x24/0x30 [ 9524.490115] do_syscall_64+0xc1/0x440 [ 9524.497652] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9524.502837] RIP: 0033:0x7ff54af45332 [ 9524.505267] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9524.526494] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9524.540218] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 9524.553200] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 9524.563656] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 9524.567830] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 9524.571591] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 9532.060255] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 9532.065213] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9532.106409] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9532.113702] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9532.125447] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9532.129706] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9532.147551] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9532.157382] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 9532.165678] RDX: 0000000000000ce6 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 9532.180362] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 9532.188271] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 9532.201711] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9532.212520] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 9532.225146] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9532.235223] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 9532.237601] Call Trace: [ 9532.250117] [ 9532.253411] ? show_regs.cold.9+0x22/0x2f [ 9532.259516] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9532.271771] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9532.274570] ? watchdog+0x40/0x40 [ 9532.276504] ? hrtimer_interrupt+0x128/0x2c0 [ 9532.279062] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9532.281357] ? apic_timer_interrupt+0xf/0x20 [ 9532.283539] [ 9532.284654] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9532.293217] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9532.301613] do_raw_spin_lock+0x177/0x190 [ 9532.305669] _raw_spin_lock+0x21/0x30 [ 9532.308798] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9532.314146] obd_mod_free+0x46/0x140 [ptlrpc] [ 9532.318414] mdc_commit_open+0x91/0xb0 [mdc] [ 9532.323570] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9532.331321] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9532.342267] after_reply+0xe2c/0x1780 [ptlrpc] [ 9532.347477] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9532.352121] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9532.353947] ? do_wait_intr+0xf0/0xf0 [ 9532.362399] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9532.370250] kthread+0x1d1/0x200 [ 9532.372642] ? set_kthread_struct+0x70/0x70 [ 9532.375512] ret_from_fork+0x1f/0x30 [ 9534.943377] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7576s! [ 9534.947393] Showing busy workqueues and worker pools: [ 9534.953176] workqueue events: flags=0x0 [ 9534.954564] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9534.961565] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 9534.969456] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 9534.974701] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 9534.982379] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9534.986151] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 9534.988143] workqueue events_long: flags=0x0 [ 9534.998946] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9535.011095] pending: lu_objects_flush [obdclass] [ 9535.016293] workqueue events_unbound: flags=0x2 [ 9535.022293] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 9535.024451] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9535.037743] workqueue events_power_efficient: flags=0x80 [ 9535.042657] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9535.047498] pending: neigh_periodic_work [ 9535.049288] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9535.052466] pending: do_cache_clean, check_lifetime [ 9535.054705] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9535.056798] pending: neigh_periodic_work [ 9535.062477] workqueue writeback: flags=0x4a [ 9535.063589] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9535.066111] pending: wb_workfn, wb_workfn [ 9535.067300] workqueue xprtiod: flags=0xa [ 9535.069087] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9535.071376] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 9535.074085] workqueue ipv6_addrconf: flags=0x40008 [ 9535.078571] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9535.080605] pending: addrconf_verify_work [ 9535.084732] workqueue knbd0-recv: flags=0x1a [ 9535.086215] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9535.090272] in-flight: 657:recv_work [ 9535.091560] workqueue ptlrpc_pinger: flags=0x2 [ 9535.093290] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9535.095648] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 9535.098323] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7576s workers=4 idle: 20831 12642 28859 [ 9535.102527] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9525s workers=2 idle: 659 [ 9536.062545] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [ 9536.071347] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9536.110558] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9536.117331] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9536.121938] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9536.125009] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9536.135139] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9536.139003] RAX: 0000000000000312 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 9536.142603] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 9536.146361] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9536.150140] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 9536.154191] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 9536.157785] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 9536.162365] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9536.165814] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 9536.170226] Call Trace: [ 9536.171642] [ 9536.172995] ? show_regs.cold.9+0x22/0x2f [ 9536.175298] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9536.177997] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9536.180566] ? watchdog+0x40/0x40 [ 9536.182200] ? hrtimer_interrupt+0x128/0x2c0 [ 9536.185278] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9536.187555] ? apic_timer_interrupt+0xf/0x20 [ 9536.189291] [ 9536.190096] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9536.192585] do_raw_spin_lock+0x177/0x190 [ 9536.194763] _raw_spin_lock+0x21/0x30 [ 9536.197133] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9536.200451] ? do_raw_spin_unlock+0x75/0x190 [ 9536.203492] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9536.206501] ? do_wait_intr+0xf0/0xf0 [ 9536.208236] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9536.210961] mdc_close+0x53b/0xd70 [mdc] [ 9536.212464] lmv_close+0x18d/0x490 [lmv] [ 9536.214121] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 9536.216585] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9536.219284] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9536.221288] ll_md_close+0x274/0x800 [lustre] [ 9536.223267] ll_file_release+0x128/0xc50 [lustre] [ 9536.225506] __fput+0x10d/0x440 [ 9536.226986] ____fput+0x12/0x20 [ 9536.228302] task_work_run+0xc6/0x110 [ 9536.229725] exit_to_usermode_loop+0x1dd/0x1f0 [ 9536.231424] do_syscall_64+0x3d6/0x440 [ 9536.232704] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9536.235351] RIP: 0033:0x7ff54af45cd8 [ 9536.236724] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9536.245014] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9536.248155] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 9536.251073] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 9536.254112] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 9536.257032] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 9536.260447] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9540.060337] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 9540.066439] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9540.117357] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9540.127169] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9540.134370] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9540.138354] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9540.142980] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9540.156086] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9540.163664] RAX: 0000000000007591 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 9540.169571] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 9540.179713] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 9540.181642] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 9540.185757] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 9540.194692] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 9540.208415] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9540.212634] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 9540.225311] Call Trace: [ 9540.227465] [ 9540.228451] ? show_regs.cold.9+0x22/0x2f [ 9540.233787] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9540.237662] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9540.242459] ? watchdog+0x40/0x40 [ 9540.247960] ? hrtimer_interrupt+0x128/0x2c0 [ 9540.251591] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9540.259668] ? apic_timer_interrupt+0xf/0x20 [ 9540.263434] [ 9540.265748] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9540.273306] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9540.274675] do_raw_spin_lock+0x177/0x190 [ 9540.275514] _raw_spin_lock+0x21/0x30 [ 9540.282449] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9540.286213] process_one_work+0x2c8/0x700 [ 9540.290371] worker_thread+0x296/0x6e0 [ 9540.297413] ? rescuer_thread+0x570/0x570 [ 9540.299881] kthread+0x1d1/0x200 [ 9540.303666] ? set_kthread_struct+0x70/0x70 [ 9540.305133] ret_from_fork+0x1f/0x30 [ 9552.060027] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [ 9552.063504] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9552.090487] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9552.095394] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9552.103924] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9552.107042] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9552.116984] RSP: 0000:ffffaf9fc15234f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9552.121347] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 9552.124384] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 9552.128326] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9552.131449] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 9552.134507] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 9552.140426] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 9552.147371] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9552.149726] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 9552.154972] Call Trace: [ 9552.155583] [ 9552.156940] ? show_regs.cold.9+0x22/0x2f [ 9552.159341] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9552.162365] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9552.165234] ? watchdog+0x40/0x40 [ 9552.167388] ? hrtimer_interrupt+0x128/0x2c0 [ 9552.172263] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9552.174267] ? apic_timer_interrupt+0xf/0x20 [ 9552.176991] [ 9552.180174] ? native_safe_halt+0x2e/0x50 [ 9552.183815] kvm_wait+0xcd/0xe0 [ 9552.186271] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9552.188492] do_raw_spin_lock+0x177/0x190 [ 9552.191378] _raw_spin_lock+0x21/0x30 [ 9552.197911] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9552.200484] ? do_raw_spin_unlock+0x75/0x190 [ 9552.203498] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9552.205877] ? do_wait_intr+0xf0/0xf0 [ 9552.207499] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9552.210444] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9552.213161] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9552.216344] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9552.219075] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9552.223396] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9552.225401] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9552.227852] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9552.230352] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9552.234128] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9552.236012] lmv_intent_open+0x338/0x1150 [lmv] [ 9552.238342] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9552.241508] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9552.243839] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9552.245418] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9552.247448] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9552.249469] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9552.252956] ? do_raw_spin_unlock+0x75/0x190 [ 9552.254958] ? _raw_spin_unlock+0x12/0x30 [ 9552.256739] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9552.260517] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9552.264446] ll_atomic_open+0x36a/0xd70 [lustre] [ 9552.266353] lookup_open+0x356/0xb40 [ 9552.268535] do_last+0x313/0xfc0 [ 9552.269740] ? nd_jump_root+0xe5/0x160 [ 9552.271375] ? path_init+0x437/0x520 [ 9552.273913] path_openat+0xf7/0x500 [ 9552.276031] ? do_raw_spin_unlock+0x75/0x190 [ 9552.277501] do_filp_open+0x99/0x140 [ 9552.279505] ? getname_flags+0x6e/0x330 [ 9552.281113] ? __check_object_size+0xff/0x256 [ 9552.283243] ? do_raw_spin_unlock+0x75/0x190 [ 9552.285318] ? _raw_spin_unlock+0x12/0x30 [ 9552.286663] do_sys_openat2+0x2b4/0x410 [ 9552.288301] do_sys_open+0x73/0xa0 [ 9552.289502] __x64_sys_openat+0x24/0x30 [ 9552.291431] do_syscall_64+0xc1/0x440 [ 9552.293626] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9552.295403] RIP: 0033:0x7ff54af45332 [ 9552.296598] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9552.304991] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9552.308420] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 9552.311572] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 9552.314568] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 9552.317468] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 9552.321478] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 9560.056921] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 9560.060060] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9560.084061] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9560.089204] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9560.093819] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9560.096797] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9560.107164] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9560.112923] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 9560.116863] RDX: 0000000000007f8d RSI: 0000000000000003 RDI: ffff96848811cae0 [ 9560.120699] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 9560.123194] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 9560.126670] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9560.129790] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 9560.133300] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9560.135753] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 9560.138190] Call Trace: [ 9560.140126] [ 9560.141107] ? show_regs.cold.9+0x22/0x2f [ 9560.142729] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9560.146052] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9560.147931] ? watchdog+0x40/0x40 [ 9560.150906] ? hrtimer_interrupt+0x128/0x2c0 [ 9560.153384] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9560.155326] ? apic_timer_interrupt+0xf/0x20 [ 9560.157474] [ 9560.159718] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9560.161879] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9560.164437] do_raw_spin_lock+0x177/0x190 [ 9560.166712] _raw_spin_lock+0x21/0x30 [ 9560.168182] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9560.170592] obd_mod_free+0x46/0x140 [ptlrpc] [ 9560.174800] mdc_commit_open+0x91/0xb0 [mdc] [ 9560.177148] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9560.179748] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9560.182669] after_reply+0xe2c/0x1780 [ptlrpc] [ 9560.185856] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9560.188827] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9560.191304] ? do_wait_intr+0xf0/0xf0 [ 9560.193048] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9560.195472] kthread+0x1d1/0x200 [ 9560.196895] ? set_kthread_struct+0x70/0x70 [ 9560.198937] ret_from_fork+0x1f/0x30 [ 9564.061855] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [ 9564.068230] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9564.102323] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9564.112406] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9564.119435] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9564.122139] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9564.130118] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9564.138850] RAX: 0000000000006a09 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 9564.151945] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 9564.157555] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9564.163056] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 9564.165913] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 9564.170267] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 9564.173913] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9564.176029] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 9564.179106] Call Trace: [ 9564.179982] [ 9564.180976] ? show_regs.cold.9+0x22/0x2f [ 9564.182640] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9564.186422] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9564.188495] ? watchdog+0x40/0x40 [ 9564.189853] ? hrtimer_interrupt+0x128/0x2c0 [ 9564.191533] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9564.193130] ? apic_timer_interrupt+0xf/0x20 [ 9564.194836] [ 9564.195748] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9564.198035] do_raw_spin_lock+0x177/0x190 [ 9564.199582] _raw_spin_lock+0x21/0x30 [ 9564.200804] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9564.203096] ? do_raw_spin_unlock+0x75/0x190 [ 9564.204901] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9564.207231] ? do_wait_intr+0xf0/0xf0 [ 9564.208813] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9564.211103] mdc_close+0x53b/0xd70 [mdc] [ 9564.212802] lmv_close+0x18d/0x490 [lmv] [ 9564.214385] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 9564.216898] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9564.219378] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9564.221351] ll_md_close+0x274/0x800 [lustre] [ 9564.223273] ll_file_release+0x128/0xc50 [lustre] [ 9564.224972] __fput+0x10d/0x440 [ 9564.226998] ____fput+0x12/0x20 [ 9564.228401] task_work_run+0xc6/0x110 [ 9564.230227] exit_to_usermode_loop+0x1dd/0x1f0 [ 9564.232603] do_syscall_64+0x3d6/0x440 [ 9564.234372] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9564.236816] RIP: 0033:0x7ff54af45cd8 [ 9564.238739] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9564.247205] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9564.251140] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 9564.254646] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 9564.258264] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 9564.262412] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 9564.266421] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9565.151030] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7606s! [ 9565.157959] Showing busy workqueues and worker pools: [ 9565.163073] workqueue events: flags=0x0 [ 9565.164900] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9565.175997] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 9565.188576] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 9565.194135] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 9565.200928] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9565.203201] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 9565.207090] workqueue events_long: flags=0x0 [ 9565.209681] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9565.212399] pending: lu_objects_flush [obdclass] [ 9565.217714] workqueue events_unbound: flags=0x2 [ 9565.220398] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 9565.223979] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9565.228041] workqueue events_power_efficient: flags=0x80 [ 9565.230445] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9565.236388] pending: neigh_periodic_work [ 9565.240820] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9565.247280] pending: do_cache_clean, check_lifetime [ 9565.251842] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9565.254044] pending: neigh_periodic_work [ 9565.255378] workqueue writeback: flags=0x4a [ 9565.257347] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9565.260372] pending: wb_workfn, wb_workfn [ 9565.264010] workqueue xprtiod: flags=0xa [ 9565.268177] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9565.271377] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 9565.281018] workqueue ipv6_addrconf: flags=0x40008 [ 9565.283413] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9565.287795] pending: addrconf_verify_work [ 9565.290981] workqueue knbd0-recv: flags=0x1a [ 9565.293292] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9565.296024] in-flight: 657:recv_work [ 9565.298186] workqueue ptlrpc_pinger: flags=0x2 [ 9565.301138] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9565.304050] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 9565.306772] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7606s workers=4 idle: 20831 12642 28859 [ 9565.308872] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9555s workers=2 idle: 659 [ 9568.061970] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 9568.065431] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9568.086152] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9568.090221] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9568.093274] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9568.095335] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9568.097236] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9568.104685] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9568.107520] RAX: 000000000000780d RBX: ffff96848811cae0 RCX: 0000000000000001 [ 9568.109927] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 9568.112562] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 9568.115073] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 9568.118280] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 9568.121212] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 9568.124311] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9568.126368] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 9568.128571] Call Trace: [ 9568.129289] [ 9568.129881] ? show_regs.cold.9+0x22/0x2f [ 9568.130895] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9568.132331] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9568.133827] ? watchdog+0x40/0x40 [ 9568.135059] ? hrtimer_interrupt+0x128/0x2c0 [ 9568.136558] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9568.138972] ? apic_timer_interrupt+0xf/0x20 [ 9568.140857] [ 9568.142066] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9568.144341] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9568.146199] do_raw_spin_lock+0x177/0x190 [ 9568.147872] _raw_spin_lock+0x21/0x30 [ 9568.148778] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9568.150600] process_one_work+0x2c8/0x700 [ 9568.151949] worker_thread+0x296/0x6e0 [ 9568.153754] ? rescuer_thread+0x570/0x570 [ 9568.154979] kthread+0x1d1/0x200 [ 9568.156051] ? set_kthread_struct+0x70/0x70 [ 9568.157679] ret_from_fork+0x1f/0x30 [ 9574.063067] rcu: INFO: rcu_sched self-detected stall on CPU [ 9574.070882] rcu: 0-...!: (3967984 ticks this GP) idle=98a/1/0x4000000000000002 softirq=125154/125154 fqs=959 [ 9574.080325] (t=7620223 jiffies g=253585 q=5394) [ 9574.082933] rcu: rcu_sched kthread starved for 7615183 jiffies! g253585 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [ 9574.088635] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 9574.097373] rcu: RCU grace-period kthread stack dump: [ 9574.099928] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 9574.104921] Call Trace: [ 9574.106821] __schedule+0x351/0xcb0 [ 9574.108441] schedule+0xc0/0x180 [ 9574.109799] schedule_timeout+0xb4/0x190 [ 9574.111944] ? do_raw_spin_unlock+0x75/0x190 [ 9574.114103] ? __next_timer_interrupt+0x160/0x160 [ 9574.115946] ? prepare_to_swait_event+0xf3/0x140 [ 9574.118259] rcu_gp_kthread+0x8dd/0x15b0 [ 9574.120186] ? force_qs_rnp+0x2c0/0x2c0 [ 9574.122131] kthread+0x1d1/0x200 [ 9574.123683] ? set_kthread_struct+0x70/0x70 [ 9574.125706] ret_from_fork+0x1f/0x30 [ 9574.127549] rcu: Stack dump where RCU GP kthread last ran: [ 9574.130361] Sending NMI from CPU 0 to CPUs 1: [ 9574.132334] NMI backtrace for cpu 1 skipped: idling at native_safe_halt+0x2e/0x50 [ 9574.133295] NMI backtrace for cpu 0 [ 9574.142802] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9574.147968] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9574.152917] Call Trace: [ 9574.154151] [ 9574.155131] ? dump_stack+0xbb/0x10e [ 9574.156995] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 9574.159807] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 9574.162857] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 9574.165631] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 9574.168464] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 9574.171518] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 9574.173715] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 9574.175573] ? update_process_times+0x6d/0xc0 [ 9574.177917] ? tick_sched_handle.isra.18+0x27/0x90 [ 9574.186952] ? tick_sched_timer+0x53/0xb0 [ 9574.188876] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9574.191646] ? tick_sched_handle.isra.18+0x90/0x90 [ 9574.193987] ? hrtimer_interrupt+0x128/0x2c0 [ 9574.195928] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9574.198188] ? apic_timer_interrupt+0xf/0x20 [ 9574.200778] [ 9574.201824] ? native_safe_halt+0x2e/0x50 [ 9574.203829] ? kvm_wait+0xcd/0xe0 [ 9574.205715] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9574.208520] ? do_raw_spin_lock+0x177/0x190 [ 9574.210769] ? _raw_spin_lock+0x21/0x30 [ 9574.212900] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9574.215853] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 9574.218717] ? mdc_commit_open+0x91/0xb0 [mdc] [ 9574.221124] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9574.224177] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9574.227360] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 9574.230115] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9574.233064] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9574.235685] ? do_wait_intr+0xf0/0xf0 [ 9574.237896] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9574.240884] ? kthread+0x1d1/0x200 [ 9574.242578] ? set_kthread_struct+0x70/0x70 [ 9574.244674] ? ret_from_fork+0x1f/0x30 [ 9580.061865] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [ 9580.073846] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9580.104994] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9580.111128] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9580.117088] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9580.120634] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9580.130038] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9580.134689] RAX: 0000000000001ac6 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 9580.138945] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 9580.142322] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9580.144874] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 9580.148602] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 9580.151861] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 9580.155722] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9580.158633] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 9580.162851] Call Trace: [ 9580.164860] [ 9580.166000] ? show_regs.cold.9+0x22/0x2f [ 9580.167992] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9580.171248] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9580.177280] ? watchdog+0x40/0x40 [ 9580.179273] ? hrtimer_interrupt+0x128/0x2c0 [ 9580.181867] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9580.184635] ? apic_timer_interrupt+0xf/0x20 [ 9580.189718] [ 9580.191527] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9580.193330] do_raw_spin_lock+0x177/0x190 [ 9580.197970] _raw_spin_lock+0x21/0x30 [ 9580.201307] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9580.209732] ? do_raw_spin_unlock+0x75/0x190 [ 9580.215657] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9580.225163] ? do_wait_intr+0xf0/0xf0 [ 9580.230726] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9580.236316] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9580.239871] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9580.246180] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9580.248855] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9580.251170] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9580.253116] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9580.255020] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9580.257478] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9580.259288] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9580.265817] lmv_intent_open+0x338/0x1150 [lmv] [ 9580.269022] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9580.274809] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9580.277175] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9580.279680] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9580.285489] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9580.290656] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9580.294794] ? do_raw_spin_unlock+0x75/0x190 [ 9580.296595] ? _raw_spin_unlock+0x12/0x30 [ 9580.298766] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9580.300988] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9580.303310] ll_atomic_open+0x36a/0xd70 [lustre] [ 9580.305832] lookup_open+0x356/0xb40 [ 9580.307286] do_last+0x313/0xfc0 [ 9580.309567] ? nd_jump_root+0xe5/0x160 [ 9580.311205] ? path_init+0x437/0x520 [ 9580.312973] path_openat+0xf7/0x500 [ 9580.316397] ? do_raw_spin_unlock+0x75/0x190 [ 9580.318627] do_filp_open+0x99/0x140 [ 9580.319970] ? getname_flags+0x6e/0x330 [ 9580.321594] ? __check_object_size+0xff/0x256 [ 9580.324934] ? do_raw_spin_unlock+0x75/0x190 [ 9580.326242] ? _raw_spin_unlock+0x12/0x30 [ 9580.329601] do_sys_openat2+0x2b4/0x410 [ 9580.331625] do_sys_open+0x73/0xa0 [ 9580.333894] __x64_sys_openat+0x24/0x30 [ 9580.335678] do_syscall_64+0xc1/0x440 [ 9580.336934] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9580.342659] RIP: 0033:0x7ff54af45332 [ 9580.343859] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9580.357800] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9580.360738] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 9580.365105] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 9580.368665] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 9580.371370] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 9580.374893] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 9595.360426] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7636s! [ 9595.368015] Showing busy workqueues and worker pools: [ 9595.373360] workqueue events: flags=0x0 [ 9595.378204] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9595.383780] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 9595.392089] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 9595.398702] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 9595.409800] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9595.414810] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 9595.419566] workqueue events_long: flags=0x0 [ 9595.424027] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9595.435764] pending: lu_objects_flush [obdclass] [ 9595.442059] workqueue events_unbound: flags=0x2 [ 9595.446401] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 9595.450004] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9595.456886] workqueue events_power_efficient: flags=0x80 [ 9595.459074] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9595.461960] pending: neigh_periodic_work [ 9595.467922] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9595.475495] pending: do_cache_clean, check_lifetime [ 9595.480622] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9595.484953] pending: neigh_periodic_work [ 9595.490913] workqueue writeback: flags=0x4a [ 9595.494410] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9595.503450] pending: wb_workfn, wb_workfn [ 9595.504731] workqueue xprtiod: flags=0xa [ 9595.506918] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9595.510739] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 9595.514610] workqueue ipv6_addrconf: flags=0x40008 [ 9595.520458] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9595.522180] pending: addrconf_verify_work [ 9595.524571] workqueue knbd0-recv: flags=0x1a [ 9595.526511] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9595.528653] in-flight: 657:recv_work [ 9595.530022] workqueue ptlrpc_pinger: flags=0x2 [ 9595.532823] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9595.537756] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 9595.543035] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7636s workers=4 idle: 20831 12642 28859 [ 9595.546140] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9585s workers=2 idle: 659 [ 9596.061023] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 9596.064445] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9596.107843] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9596.122095] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9596.127748] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9596.130408] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9596.132155] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9596.140490] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9596.143853] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 9596.146455] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 9596.149831] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 9596.153047] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 9596.156984] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 9596.160681] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 9596.164415] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9596.167249] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 9596.170336] Call Trace: [ 9596.172237] [ 9596.173299] ? show_regs.cold.9+0x22/0x2f [ 9596.175894] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9596.178283] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9596.181175] ? watchdog+0x40/0x40 [ 9596.182478] ? hrtimer_interrupt+0x128/0x2c0 [ 9596.185881] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9596.187922] ? apic_timer_interrupt+0xf/0x20 [ 9596.190324] [ 9596.191971] ? native_safe_halt+0x2e/0x50 [ 9596.194195] kvm_wait+0xcd/0xe0 [ 9596.195694] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9596.198161] do_raw_spin_lock+0x177/0x190 [ 9596.200242] _raw_spin_lock+0x21/0x30 [ 9596.202124] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9596.204840] process_one_work+0x2c8/0x700 [ 9596.206815] worker_thread+0x296/0x6e0 [ 9596.209004] ? rescuer_thread+0x570/0x570 [ 9596.211476] kthread+0x1d1/0x200 [ 9596.213165] ? set_kthread_struct+0x70/0x70 [ 9596.215154] ret_from_fork+0x1f/0x30 [ 9600.058013] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 9600.062658] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9600.089617] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9600.099980] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9600.112676] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9600.118958] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9600.132145] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9600.136823] RAX: 0000000000000071 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 9600.140663] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 9600.144739] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9600.148774] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 9600.152781] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 9600.156700] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 9600.160979] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9600.164072] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 9600.168190] Call Trace: [ 9600.169751] [ 9600.170917] ? show_regs.cold.9+0x22/0x2f [ 9600.173041] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9600.175505] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9600.178016] ? watchdog+0x40/0x40 [ 9600.179862] ? hrtimer_interrupt+0x128/0x2c0 [ 9600.182244] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9600.184892] ? apic_timer_interrupt+0xf/0x20 [ 9600.187478] [ 9600.188604] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9600.191261] do_raw_spin_lock+0x177/0x190 [ 9600.193833] _raw_spin_lock+0x21/0x30 [ 9600.196003] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9600.199269] ? do_raw_spin_unlock+0x75/0x190 [ 9600.201783] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9600.204788] ? do_wait_intr+0xf0/0xf0 [ 9600.206813] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9600.210577] mdc_close+0x53b/0xd70 [mdc] [ 9600.213277] lmv_close+0x18d/0x490 [lmv] [ 9600.216452] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 9600.219489] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9600.221246] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9600.222857] ll_md_close+0x274/0x800 [lustre] [ 9600.224843] ll_file_release+0x128/0xc50 [lustre] [ 9600.226468] __fput+0x10d/0x440 [ 9600.228159] ____fput+0x12/0x20 [ 9600.230533] task_work_run+0xc6/0x110 [ 9600.233379] exit_to_usermode_loop+0x1dd/0x1f0 [ 9600.235545] do_syscall_64+0x3d6/0x440 [ 9600.238158] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9600.241003] RIP: 0033:0x7ff54af45cd8 [ 9600.242757] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9600.252992] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9600.259785] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 9600.266679] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 9600.274419] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 9600.282269] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 9600.288790] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9608.061285] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [ 9608.064434] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9608.085256] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9608.098954] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9608.105173] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9608.111461] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9608.125355] RSP: 0000:ffffaf9fc15234f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9608.130758] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 9608.136279] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 9608.142717] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9608.150573] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 9608.157015] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 9608.160170] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 9608.166027] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9608.170008] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 9608.174395] Call Trace: [ 9608.175797] [ 9608.176844] ? show_regs.cold.9+0x22/0x2f [ 9608.178918] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9608.182682] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9608.185257] ? watchdog+0x40/0x40 [ 9608.186798] ? hrtimer_interrupt+0x128/0x2c0 [ 9608.188589] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9608.193323] ? apic_timer_interrupt+0xf/0x20 [ 9608.195292] [ 9608.196293] ? native_safe_halt+0x2e/0x50 [ 9608.198063] kvm_wait+0xcd/0xe0 [ 9608.199806] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9608.204777] do_raw_spin_lock+0x177/0x190 [ 9608.209812] _raw_spin_lock+0x21/0x30 [ 9608.211947] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9608.215163] ? do_raw_spin_unlock+0x75/0x190 [ 9608.217591] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9608.219939] ? do_wait_intr+0xf0/0xf0 [ 9608.221877] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9608.225833] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9608.231019] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9608.240075] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9608.242874] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9608.245799] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9608.251020] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9608.253379] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9608.259013] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9608.263873] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9608.267307] lmv_intent_open+0x338/0x1150 [lmv] [ 9608.270377] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9608.273165] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9608.275808] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9608.277792] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9608.279963] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9608.282673] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9608.284918] ? do_raw_spin_unlock+0x75/0x190 [ 9608.287405] ? _raw_spin_unlock+0x12/0x30 [ 9608.289381] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9608.291582] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9608.293517] ll_atomic_open+0x36a/0xd70 [lustre] [ 9608.295768] lookup_open+0x356/0xb40 [ 9608.297612] do_last+0x313/0xfc0 [ 9608.298938] ? nd_jump_root+0xe5/0x160 [ 9608.300590] ? path_init+0x437/0x520 [ 9608.301976] path_openat+0xf7/0x500 [ 9608.303398] ? do_raw_spin_unlock+0x75/0x190 [ 9608.305312] do_filp_open+0x99/0x140 [ 9608.306838] ? getname_flags+0x6e/0x330 [ 9608.308668] ? __check_object_size+0xff/0x256 [ 9608.310100] ? do_raw_spin_unlock+0x75/0x190 [ 9608.311819] ? _raw_spin_unlock+0x12/0x30 [ 9608.313646] do_sys_openat2+0x2b4/0x410 [ 9608.315408] do_sys_open+0x73/0xa0 [ 9608.316869] __x64_sys_openat+0x24/0x30 [ 9608.318240] do_syscall_64+0xc1/0x440 [ 9608.319564] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9608.321007] RIP: 0033:0x7ff54af45332 [ 9608.322597] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9608.329396] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9608.333245] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 9608.336045] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 9608.339141] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 9608.341996] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 9608.344942] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 9620.055034] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 9620.058500] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9620.095788] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9620.108175] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9620.117675] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9620.122082] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9620.133096] RSP: 0000:ffffaf9fc1b97bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9620.139306] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 9620.144350] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 9620.148322] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 9620.152699] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 9620.157376] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9620.167217] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 9620.174267] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9620.177703] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 9620.184182] Call Trace: [ 9620.186754] [ 9620.188717] ? show_regs.cold.9+0x22/0x2f [ 9620.192917] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9620.196326] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9620.199994] ? watchdog+0x40/0x40 [ 9620.202979] ? hrtimer_interrupt+0x128/0x2c0 [ 9620.205395] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9620.210191] ? apic_timer_interrupt+0xf/0x20 [ 9620.213136] [ 9620.214235] ? native_safe_halt+0x2e/0x50 [ 9620.216488] kvm_wait+0xcd/0xe0 [ 9620.218325] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9620.220496] do_raw_spin_lock+0x177/0x190 [ 9620.222371] _raw_spin_lock+0x21/0x30 [ 9620.224206] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9620.228981] obd_mod_free+0x46/0x140 [ptlrpc] [ 9620.232136] mdc_commit_open+0x91/0xb0 [mdc] [ 9620.235051] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9620.238230] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9620.241330] after_reply+0xe2c/0x1780 [ptlrpc] [ 9620.244294] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9620.250823] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9620.253702] ? do_wait_intr+0xf0/0xf0 [ 9620.254998] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9620.260179] kthread+0x1d1/0x200 [ 9620.265772] ? set_kthread_struct+0x70/0x70 [ 9620.269281] ret_from_fork+0x1f/0x30 [ 9624.059149] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 9624.061873] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9624.079089] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9624.083544] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9624.086789] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9624.089617] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9624.091303] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9624.098421] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9624.101442] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 9624.104676] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 9624.107239] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 9624.110165] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 9624.113288] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 9624.115711] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 9624.119368] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9624.121533] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 9624.124435] Call Trace: [ 9624.125493] [ 9624.126332] ? show_regs.cold.9+0x22/0x2f [ 9624.127619] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9624.129136] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9624.131003] ? watchdog+0x40/0x40 [ 9624.132578] ? hrtimer_interrupt+0x128/0x2c0 [ 9624.134688] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9624.136911] ? apic_timer_interrupt+0xf/0x20 [ 9624.139091] [ 9624.139844] ? native_safe_halt+0x2e/0x50 [ 9624.141204] kvm_wait+0xcd/0xe0 [ 9624.144354] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9624.148339] do_raw_spin_lock+0x177/0x190 [ 9624.151638] _raw_spin_lock+0x21/0x30 [ 9624.153383] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9624.155579] process_one_work+0x2c8/0x700 [ 9624.157179] worker_thread+0x296/0x6e0 [ 9624.158688] ? rescuer_thread+0x570/0x570 [ 9624.159890] kthread+0x1d1/0x200 [ 9624.160829] ? set_kthread_struct+0x70/0x70 [ 9624.162079] ret_from_fork+0x1f/0x30 [ 9625.569842] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7666s! [ 9625.572675] Showing busy workqueues and worker pools: [ 9625.575651] workqueue events: flags=0x0 [ 9625.577984] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9625.581499] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 9625.588928] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 9625.593159] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 9625.598725] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9625.601699] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 9625.608402] workqueue events_long: flags=0x0 [ 9625.612530] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9625.618936] pending: lu_objects_flush [obdclass] [ 9625.626222] workqueue events_unbound: flags=0x2 [ 9625.628194] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 9625.633102] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9625.640663] workqueue events_power_efficient: flags=0x80 [ 9625.644112] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9625.646697] pending: neigh_periodic_work [ 9625.648185] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9625.651895] pending: do_cache_clean, check_lifetime [ 9625.660353] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9625.665224] pending: neigh_periodic_work [ 9625.671218] workqueue writeback: flags=0x4a [ 9625.676171] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9625.677932] pending: wb_workfn, wb_workfn [ 9625.679733] workqueue xprtiod: flags=0xa [ 9625.681167] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9625.686103] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 9625.692219] workqueue ipv6_addrconf: flags=0x40008 [ 9625.695427] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9625.708915] pending: addrconf_verify_work [ 9625.714788] workqueue knbd0-recv: flags=0x1a [ 9625.718440] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9625.725112] in-flight: 657:recv_work [ 9625.726410] workqueue ptlrpc_pinger: flags=0x2 [ 9625.729479] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9625.734774] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 9625.739257] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7666s workers=4 idle: 20831 12642 28859 [ 9625.752137] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9615s workers=2 idle: 659 [ 9628.064384] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 9628.071507] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9628.121051] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9628.135957] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9628.146970] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9628.154666] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9628.185107] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9628.192508] RAX: 0000000000007e97 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 9628.196649] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 9628.209462] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9628.220635] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 9628.229625] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 9628.237841] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 9628.244558] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9628.251074] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 9628.255746] Call Trace: [ 9628.257481] [ 9628.260448] ? show_regs.cold.9+0x22/0x2f [ 9628.261918] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9628.270170] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9628.271846] ? watchdog+0x40/0x40 [ 9628.273325] ? hrtimer_interrupt+0x128/0x2c0 [ 9628.276833] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9628.284451] ? apic_timer_interrupt+0xf/0x20 [ 9628.285752] [ 9628.286658] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9628.289642] do_raw_spin_lock+0x177/0x190 [ 9628.290595] _raw_spin_lock+0x21/0x30 [ 9628.291518] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9628.297339] ? do_raw_spin_unlock+0x75/0x190 [ 9628.299756] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9628.304739] ? do_wait_intr+0xf0/0xf0 [ 9628.308625] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9628.313734] mdc_close+0x53b/0xd70 [mdc] [ 9628.317352] lmv_close+0x18d/0x490 [lmv] [ 9628.319152] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 9628.320703] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9628.328799] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9628.331331] ll_md_close+0x274/0x800 [lustre] [ 9628.333728] ll_file_release+0x128/0xc50 [lustre] [ 9628.335805] __fput+0x10d/0x440 [ 9628.338445] ____fput+0x12/0x20 [ 9628.339553] task_work_run+0xc6/0x110 [ 9628.342709] exit_to_usermode_loop+0x1dd/0x1f0 [ 9628.346387] do_syscall_64+0x3d6/0x440 [ 9628.347882] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9628.351664] RIP: 0033:0x7ff54af45cd8 [ 9628.353598] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9628.379154] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9628.381640] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 9628.386137] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 9628.388989] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 9628.391243] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 9628.393306] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9636.060070] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [ 9636.063742] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9636.080903] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9636.084775] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9636.087739] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9636.090027] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9636.097205] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9636.099927] RAX: 000000000000649c RBX: ffff96848811cae0 RCX: 0000000000000001 [ 9636.102545] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 9636.106138] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9636.110075] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 9636.114947] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 9636.117220] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 9636.122556] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9636.124424] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 9636.127938] Call Trace: [ 9636.129049] [ 9636.130019] ? show_regs.cold.9+0x22/0x2f [ 9636.132113] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9636.134186] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9636.136409] ? watchdog+0x40/0x40 [ 9636.138155] ? hrtimer_interrupt+0x128/0x2c0 [ 9636.140237] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9636.142454] ? apic_timer_interrupt+0xf/0x20 [ 9636.144322] [ 9636.145187] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9636.147792] do_raw_spin_lock+0x177/0x190 [ 9636.149451] _raw_spin_lock+0x21/0x30 [ 9636.152167] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9636.161097] ? do_raw_spin_unlock+0x75/0x190 [ 9636.165414] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9636.174058] ? do_wait_intr+0xf0/0xf0 [ 9636.182337] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9636.186762] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9636.192437] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9636.197214] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9636.200687] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9636.203759] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9636.207050] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9636.211090] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9636.215149] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9636.219575] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9636.223209] lmv_intent_open+0x338/0x1150 [lmv] [ 9636.225598] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9636.228011] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9636.234192] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9636.237655] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9636.240231] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9636.243199] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9636.250153] ? do_raw_spin_unlock+0x75/0x190 [ 9636.254439] ? _raw_spin_unlock+0x12/0x30 [ 9636.258216] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9636.262521] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9636.265179] ll_atomic_open+0x36a/0xd70 [lustre] [ 9636.268096] lookup_open+0x356/0xb40 [ 9636.270115] do_last+0x313/0xfc0 [ 9636.272517] ? nd_jump_root+0xe5/0x160 [ 9636.274758] ? path_init+0x437/0x520 [ 9636.276595] path_openat+0xf7/0x500 [ 9636.278392] ? do_raw_spin_unlock+0x75/0x190 [ 9636.281086] do_filp_open+0x99/0x140 [ 9636.283141] ? getname_flags+0x6e/0x330 [ 9636.285714] ? __check_object_size+0xff/0x256 [ 9636.288243] ? do_raw_spin_unlock+0x75/0x190 [ 9636.292531] ? _raw_spin_unlock+0x12/0x30 [ 9636.294470] do_sys_openat2+0x2b4/0x410 [ 9636.299074] do_sys_open+0x73/0xa0 [ 9636.301387] __x64_sys_openat+0x24/0x30 [ 9636.303205] do_syscall_64+0xc1/0x440 [ 9636.305348] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9636.310199] RIP: 0033:0x7ff54af45332 [ 9636.312051] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9636.322663] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9636.326138] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 9636.330548] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 9636.342934] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 9636.352432] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 9636.357328] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 9652.060432] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [ 9652.064362] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 9652.066620] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9652.066834] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9652.066842] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9652.066845] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9652.066860] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9652.066865] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9652.066870] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 9652.066872] RDX: 0000000000002aef RSI: 0000000000000003 RDI: ffff96848811cae0 [ 9652.066875] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 9652.066878] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 9652.066881] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9652.066884] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 9652.066888] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9652.066890] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 9652.066898] Call Trace: [ 9652.066903] [ 9652.066909] ? show_regs.cold.9+0x22/0x2f [ 9652.066919] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9652.066925] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9652.066929] ? watchdog+0x40/0x40 [ 9652.066934] ? hrtimer_interrupt+0x128/0x2c0 [ 9652.066938] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9652.066947] ? apic_timer_interrupt+0xf/0x20 [ 9652.066952] [ 9652.066956] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9652.066961] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9652.066966] do_raw_spin_lock+0x177/0x190 [ 9652.066971] _raw_spin_lock+0x21/0x30 [ 9652.066976] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9652.068967] obd_mod_free+0x46/0x140 [ptlrpc] [ 9652.076832] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9652.077761] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9652.077768] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9652.077775] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9652.078294] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9652.078307] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9652.078312] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9652.078317] RAX: 000000000000368b RBX: ffff96848811cae0 RCX: 0000000000000001 [ 9652.078319] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 9652.078321] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 9652.078323] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 9652.078326] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 9652.078330] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 9652.078333] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9652.078336] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 9652.078347] Call Trace: [ 9652.078353] [ 9652.078358] ? show_regs.cold.9+0x22/0x2f [ 9652.078370] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9652.079736] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9652.079741] ? watchdog+0x40/0x40 [ 9652.079745] ? hrtimer_interrupt+0x128/0x2c0 [ 9652.079749] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9652.079757] ? apic_timer_interrupt+0xf/0x20 [ 9652.079762] [ 9652.079766] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9652.079772] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9652.079779] do_raw_spin_lock+0x177/0x190 [ 9652.079785] _raw_spin_lock+0x21/0x30 [ 9652.079789] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9652.080093] process_one_work+0x2c8/0x700 [ 9652.080100] worker_thread+0x296/0x6e0 [ 9652.080105] ? rescuer_thread+0x570/0x570 [ 9652.080109] kthread+0x1d1/0x200 [ 9652.080114] ? set_kthread_struct+0x70/0x70 [ 9652.080117] ret_from_fork+0x1f/0x30 [ 9652.567397] mdc_commit_open+0x91/0xb0 [mdc] [ 9652.569551] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9652.572611] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9652.575204] after_reply+0xe2c/0x1780 [ptlrpc] [ 9652.578001] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9652.580298] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9652.582717] ? do_wait_intr+0xf0/0xf0 [ 9652.584915] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9652.587371] kthread+0x1d1/0x200 [ 9652.589385] ? set_kthread_struct+0x70/0x70 [ 9652.591544] ret_from_fork+0x1f/0x30 [ 9655.775072] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7696s! [ 9655.779379] Showing busy workqueues and worker pools: [ 9655.782562] workqueue events: flags=0x0 [ 9655.784663] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9655.788179] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 9655.793285] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 9655.796752] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 9655.809183] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9655.813632] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 9655.823167] workqueue events_long: flags=0x0 [ 9655.827090] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9655.837973] pending: lu_objects_flush [obdclass] [ 9655.842159] workqueue events_unbound: flags=0x2 [ 9655.849518] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 9655.853910] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9655.867730] workqueue events_power_efficient: flags=0x80 [ 9655.870934] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9655.874516] pending: neigh_periodic_work [ 9655.877917] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9655.881961] pending: do_cache_clean, check_lifetime [ 9655.884206] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9655.886377] pending: neigh_periodic_work [ 9655.894528] workqueue writeback: flags=0x4a [ 9655.899527] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9655.908450] pending: wb_workfn, wb_workfn [ 9655.913096] workqueue xprtiod: flags=0xa [ 9655.914953] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9655.922372] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 9655.927459] workqueue ipv6_addrconf: flags=0x40008 [ 9655.930037] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9655.934887] pending: addrconf_verify_work [ 9655.940253] workqueue knbd0-recv: flags=0x1a [ 9655.949058] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9655.958319] in-flight: 657:recv_work [ 9655.963777] workqueue ptlrpc_pinger: flags=0x2 [ 9655.964983] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9655.981414] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 9655.985245] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7697s workers=4 idle: 20831 12642 28859 [ 9655.994209] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9646s workers=2 idle: 659 [ 9656.058778] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 9656.061782] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9656.110844] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9656.118273] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9656.125963] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9656.129930] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9656.167033] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9656.176855] RAX: 000000000000750a RBX: ffff96848811cae0 RCX: 0000000000000001 [ 9656.185151] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 9656.191112] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9656.195279] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 9656.200878] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 9656.215328] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 9656.222444] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9656.225879] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 9656.229915] Call Trace: [ 9656.231447] [ 9656.232857] ? show_regs.cold.9+0x22/0x2f [ 9656.235986] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9656.242860] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9656.247814] ? watchdog+0x40/0x40 [ 9656.252726] ? hrtimer_interrupt+0x128/0x2c0 [ 9656.258779] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9656.262583] ? apic_timer_interrupt+0xf/0x20 [ 9656.267027] [ 9656.267607] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9656.272945] do_raw_spin_lock+0x177/0x190 [ 9656.274270] _raw_spin_lock+0x21/0x30 [ 9656.280244] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9656.288600] ? do_raw_spin_unlock+0x75/0x190 [ 9656.296459] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9656.305292] ? do_wait_intr+0xf0/0xf0 [ 9656.307532] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9656.312481] mdc_close+0x53b/0xd70 [mdc] [ 9656.318333] lmv_close+0x18d/0x490 [lmv] [ 9656.320691] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 9656.324957] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9656.330439] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9656.333619] ll_md_close+0x274/0x800 [lustre] [ 9656.336655] ll_file_release+0x128/0xc50 [lustre] [ 9656.338212] __fput+0x10d/0x440 [ 9656.342265] ____fput+0x12/0x20 [ 9656.345387] task_work_run+0xc6/0x110 [ 9656.354289] exit_to_usermode_loop+0x1dd/0x1f0 [ 9656.357509] do_syscall_64+0x3d6/0x440 [ 9656.363723] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9656.368827] RIP: 0033:0x7ff54af45cd8 [ 9656.371909] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9656.383452] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9656.387567] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 9656.393254] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 9656.398476] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 9656.404902] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 9656.409517] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9664.060047] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 9664.064630] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9664.101406] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9664.125663] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9664.129880] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9664.131445] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9664.155644] RSP: 0000:ffffaf9fc15234f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9664.159809] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 9664.163936] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 9664.167716] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9664.172905] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 9664.177183] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 9664.181082] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 9664.185433] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9664.187844] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 9664.190595] Call Trace: [ 9664.191439] [ 9664.192584] ? show_regs.cold.9+0x22/0x2f [ 9664.194130] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9664.196121] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9664.198049] ? watchdog+0x40/0x40 [ 9664.199276] ? hrtimer_interrupt+0x128/0x2c0 [ 9664.201082] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9664.203972] ? apic_timer_interrupt+0xf/0x20 [ 9664.205759] [ 9664.206501] ? native_safe_halt+0x2e/0x50 [ 9664.208749] kvm_wait+0xcd/0xe0 [ 9664.210451] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9664.212982] do_raw_spin_lock+0x177/0x190 [ 9664.215481] _raw_spin_lock+0x21/0x30 [ 9664.217193] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9664.220467] ? do_raw_spin_unlock+0x75/0x190 [ 9664.222225] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9664.225114] ? do_wait_intr+0xf0/0xf0 [ 9664.228727] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9664.230625] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9664.234534] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9664.239819] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9664.246426] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9664.249839] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9664.252179] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9664.255164] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9664.257613] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9664.261064] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9664.263618] lmv_intent_open+0x338/0x1150 [lmv] [ 9664.266085] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9664.268937] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9664.271945] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9664.274363] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9664.277050] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9664.280847] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9664.284002] ? do_raw_spin_unlock+0x75/0x190 [ 9664.285651] ? _raw_spin_unlock+0x12/0x30 [ 9664.288018] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9664.290195] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9664.293122] ll_atomic_open+0x36a/0xd70 [lustre] [ 9664.297165] lookup_open+0x356/0xb40 [ 9664.298327] do_last+0x313/0xfc0 [ 9664.299440] ? nd_jump_root+0xe5/0x160 [ 9664.301466] ? path_init+0x437/0x520 [ 9664.303203] path_openat+0xf7/0x500 [ 9664.306083] ? do_raw_spin_unlock+0x75/0x190 [ 9664.309991] do_filp_open+0x99/0x140 [ 9664.312940] ? getname_flags+0x6e/0x330 [ 9664.314486] ? __check_object_size+0xff/0x256 [ 9664.319354] ? do_raw_spin_unlock+0x75/0x190 [ 9664.326207] ? _raw_spin_unlock+0x12/0x30 [ 9664.334199] do_sys_openat2+0x2b4/0x410 [ 9664.338158] do_sys_open+0x73/0xa0 [ 9664.340649] __x64_sys_openat+0x24/0x30 [ 9664.346067] do_syscall_64+0xc1/0x440 [ 9664.350102] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9664.354790] RIP: 0033:0x7ff54af45332 [ 9664.361105] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9664.378443] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9664.382276] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 9664.386033] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 9664.389986] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 9664.393277] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 9664.397202] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 9680.055012] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [ 9680.058621] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9680.059066] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 9680.079744] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9680.079764] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9680.079767] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9680.079786] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9680.079791] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9680.079868] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 9680.083914] Modules linked in: lustre(O) osp(O) [ 9680.089125] RDX: 00000000000019e4 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 9680.089138] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 9680.089141] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 9680.089144] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9680.089147] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 9680.089149] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9680.089152] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 9680.089161] Call Trace: [ 9680.089171] [ 9680.089180] ? show_regs.cold.9+0x22/0x2f [ 9680.089198] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9680.089206] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9680.089209] ? watchdog+0x40/0x40 [ 9680.089229] ? hrtimer_interrupt+0x128/0x2c0 [ 9680.089233] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9680.089245] ? apic_timer_interrupt+0xf/0x20 [ 9680.089250] [ 9680.089253] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9680.089261] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9680.089266] do_raw_spin_lock+0x177/0x190 [ 9680.089272] _raw_spin_lock+0x21/0x30 [ 9680.089280] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9680.089635] obd_mod_free+0x46/0x140 [ptlrpc] [ 9680.093861] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) [ 9680.096541] mdc_commit_open+0x91/0xb0 [mdc] [ 9680.096601] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9680.105065] lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) [ 9680.108284] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9680.110723] libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9680.110769] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9680.112915] after_reply+0xe2c/0x1780 [ptlrpc] [ 9680.116587] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9680.121928] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9680.125697] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9680.128820] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9680.133065] [ 9680.133080] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9680.135522] ? do_wait_intr+0xf0/0xf0 [ 9680.135559] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9680.138038] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9680.138052] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9680.138058] RAX: 0000000000000a99 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 9680.138060] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 9680.138061] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 9680.138063] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 9680.138065] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 9680.138069] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 9680.138071] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9680.138072] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 9680.138080] Call Trace: [ 9680.138088] [ 9680.138095] ? show_regs.cold.9+0x22/0x2f [ 9680.138109] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9680.138117] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9680.138123] ? watchdog+0x40/0x40 [ 9680.138130] ? hrtimer_interrupt+0x128/0x2c0 [ 9680.139220] kthread+0x1d1/0x200 [ 9680.139934] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9680.142074] ? set_kthread_struct+0x70/0x70 [ 9680.142095] ret_from_fork+0x1f/0x30 [ 9680.247707] ? apic_timer_interrupt+0xf/0x20 [ 9680.249422] [ 9680.250338] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9680.252437] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9680.254643] do_raw_spin_lock+0x177/0x190 [ 9680.256296] _raw_spin_lock+0x21/0x30 [ 9680.257705] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9680.260111] process_one_work+0x2c8/0x700 [ 9680.261831] worker_thread+0x296/0x6e0 [ 9680.263478] ? rescuer_thread+0x570/0x570 [ 9680.265081] kthread+0x1d1/0x200 [ 9680.266744] ? set_kthread_struct+0x70/0x70 [ 9680.268318] ret_from_fork+0x1f/0x30 [ 9684.058058] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 9684.064197] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9684.094839] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9684.100812] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9684.105708] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9684.109581] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9684.119087] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9684.123268] RAX: 000000000000390d RBX: ffff96848811cae0 RCX: 0000000000000001 [ 9684.126845] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 9684.130427] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9684.134156] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 9684.137868] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 9684.141274] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 9684.145599] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9684.148744] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 9684.152244] Call Trace: [ 9684.153810] [ 9684.154884] ? show_regs.cold.9+0x22/0x2f [ 9684.157119] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9684.159832] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9684.163496] ? watchdog+0x40/0x40 [ 9684.165666] ? hrtimer_interrupt+0x128/0x2c0 [ 9684.167663] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9684.170843] ? apic_timer_interrupt+0xf/0x20 [ 9684.172828] [ 9684.173873] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9684.176772] do_raw_spin_lock+0x177/0x190 [ 9684.179373] _raw_spin_lock+0x21/0x30 [ 9684.181767] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9684.185623] ? do_raw_spin_unlock+0x75/0x190 [ 9684.187673] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9684.191118] ? do_wait_intr+0xf0/0xf0 [ 9684.193168] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9684.196322] mdc_close+0x53b/0xd70 [mdc] [ 9684.199422] lmv_close+0x18d/0x490 [lmv] [ 9684.201594] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 9684.204497] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9684.207652] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9684.210689] ll_md_close+0x274/0x800 [lustre] [ 9684.212822] ll_file_release+0x128/0xc50 [lustre] [ 9684.215921] __fput+0x10d/0x440 [ 9684.217278] ____fput+0x12/0x20 [ 9684.219015] task_work_run+0xc6/0x110 [ 9684.220929] exit_to_usermode_loop+0x1dd/0x1f0 [ 9684.222863] do_syscall_64+0x3d6/0x440 [ 9684.225006] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9684.227297] RIP: 0033:0x7ff54af45cd8 [ 9684.229150] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9684.238203] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9684.242276] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 9684.245655] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 9684.248904] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 9684.252089] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 9684.255641] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9686.495092] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7727s! [ 9686.499510] Showing busy workqueues and worker pools: [ 9686.502669] workqueue events: flags=0x0 [ 9686.504644] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9686.508243] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 9686.514170] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 9686.517775] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 9686.524840] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9686.528652] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 9686.532643] workqueue events_long: flags=0x0 [ 9686.535460] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9686.540090] pending: lu_objects_flush [obdclass] [ 9686.543334] workqueue events_unbound: flags=0x2 [ 9686.545629] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 9686.549340] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9686.555516] workqueue events_power_efficient: flags=0x80 [ 9686.558609] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9686.563179] pending: neigh_periodic_work [ 9686.565597] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9686.569849] pending: do_cache_clean, check_lifetime [ 9686.572624] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9686.577180] pending: neigh_periodic_work [ 9686.579502] workqueue writeback: flags=0x4a [ 9686.582033] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9686.586383] pending: wb_workfn, wb_workfn [ 9686.589184] workqueue xprtiod: flags=0xa [ 9686.591307] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9686.594654] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 9686.599311] workqueue ipv6_addrconf: flags=0x40008 [ 9686.601951] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9686.605684] pending: addrconf_verify_work [ 9686.608250] workqueue knbd0-recv: flags=0x1a [ 9686.610577] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9686.614414] in-flight: 657:recv_work [ 9686.616503] workqueue ptlrpc_pinger: flags=0x2 [ 9686.619238] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9686.622487] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 9686.626536] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7727s workers=4 idle: 20831 12642 28859 [ 9686.631499] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9676s workers=2 idle: 659 [ 9692.065513] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 9692.069829] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9692.101923] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9692.104181] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9692.109399] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9692.112087] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9692.124565] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9692.129047] RAX: 0000000000007d8a RBX: ffff96848811cae0 RCX: 0000000000000001 [ 9692.139768] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 9692.148718] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9692.157851] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 9692.169212] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 9692.176033] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 9692.181802] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9692.187943] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 9692.192515] Call Trace: [ 9692.193286] [ 9692.193998] ? show_regs.cold.9+0x22/0x2f [ 9692.206855] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9692.210957] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9692.216787] ? watchdog+0x40/0x40 [ 9692.219598] ? hrtimer_interrupt+0x128/0x2c0 [ 9692.225735] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9692.227617] ? apic_timer_interrupt+0xf/0x20 [ 9692.236865] [ 9692.237659] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9692.239269] do_raw_spin_lock+0x177/0x190 [ 9692.244136] _raw_spin_lock+0x21/0x30 [ 9692.245180] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9692.258132] ? do_raw_spin_unlock+0x75/0x190 [ 9692.262788] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9692.271793] ? do_wait_intr+0xf0/0xf0 [ 9692.273142] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9692.283861] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9692.293057] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9692.295181] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9692.303515] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9692.310682] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9692.315882] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9692.319903] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9692.322057] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9692.327418] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9692.333093] lmv_intent_open+0x338/0x1150 [lmv] [ 9692.339115] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9692.343565] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9692.357419] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9692.359020] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9692.367983] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9692.374593] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9692.377855] ? do_raw_spin_unlock+0x75/0x190 [ 9692.383395] ? _raw_spin_unlock+0x12/0x30 [ 9692.388544] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9692.394545] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9692.402044] ll_atomic_open+0x36a/0xd70 [lustre] [ 9692.404451] lookup_open+0x356/0xb40 [ 9692.407050] do_last+0x313/0xfc0 [ 9692.409159] ? nd_jump_root+0xe5/0x160 [ 9692.415597] ? path_init+0x437/0x520 [ 9692.416769] path_openat+0xf7/0x500 [ 9692.419563] ? do_raw_spin_unlock+0x75/0x190 [ 9692.424205] do_filp_open+0x99/0x140 [ 9692.427794] ? getname_flags+0x6e/0x330 [ 9692.434178] ? __check_object_size+0xff/0x256 [ 9692.437084] ? do_raw_spin_unlock+0x75/0x190 [ 9692.440046] ? _raw_spin_unlock+0x12/0x30 [ 9692.441427] do_sys_openat2+0x2b4/0x410 [ 9692.442178] do_sys_open+0x73/0xa0 [ 9692.444550] __x64_sys_openat+0x24/0x30 [ 9692.448730] do_syscall_64+0xc1/0x440 [ 9692.452743] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9692.457951] RIP: 0033:0x7ff54af45332 [ 9692.461879] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9692.480520] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9692.490193] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 9692.497120] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 9692.504032] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 9692.508085] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 9692.516311] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 9708.066155] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 9708.076929] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9708.114349] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9708.125313] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9708.136275] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9708.141163] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9708.147527] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9708.162792] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9708.169401] RAX: 0000000000007c09 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 9708.177001] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 9708.184037] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 9708.193274] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 9708.206676] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 9708.214894] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 9708.220035] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9708.228568] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 9708.235004] Call Trace: [ 9708.235888] [ 9708.243312] ? show_regs.cold.9+0x22/0x2f [ 9708.249048] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9708.262633] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9708.274383] ? watchdog+0x40/0x40 [ 9708.276522] ? hrtimer_interrupt+0x128/0x2c0 [ 9708.278516] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9708.282389] ? apic_timer_interrupt+0xf/0x20 [ 9708.292221] [ 9708.293080] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9708.298090] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9708.309544] do_raw_spin_lock+0x177/0x190 [ 9708.311159] _raw_spin_lock+0x21/0x30 [ 9708.319960] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9708.331952] process_one_work+0x2c8/0x700 [ 9708.338382] worker_thread+0x296/0x6e0 [ 9708.346272] ? rescuer_thread+0x570/0x570 [ 9708.351690] kthread+0x1d1/0x200 [ 9708.352867] ? set_kthread_struct+0x70/0x70 [ 9708.356148] ret_from_fork+0x1f/0x30 [ 9712.058581] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [ 9712.060427] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [ 9712.064534] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9712.064865] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9712.064873] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9712.064876] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9712.064894] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9712.064899] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9712.064903] RAX: 0000000000002a84 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 9712.064906] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 9712.064908] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9712.064910] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 9712.064913] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 9712.064916] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 9712.064919] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9712.064921] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 9712.064930] Call Trace: [ 9712.064936] [ 9712.064944] ? show_regs.cold.9+0x22/0x2f [ 9712.064955] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9712.064962] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9712.064967] ? watchdog+0x40/0x40 [ 9712.064971] ? hrtimer_interrupt+0x128/0x2c0 [ 9712.064975] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9712.064984] ? apic_timer_interrupt+0xf/0x20 [ 9712.064989] [ 9712.064992] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9712.065000] do_raw_spin_lock+0x177/0x190 [ 9712.065005] _raw_spin_lock+0x21/0x30 [ 9712.065112] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9712.065972] ? do_raw_spin_unlock+0x75/0x190 [ 9712.065979] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9712.066452] ? do_wait_intr+0xf0/0xf0 [ 9712.066461] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9712.071700] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9712.071789] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9712.071800] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9712.119788] mdc_close+0x53b/0xd70 [mdc] [ 9712.119858] lmv_close+0x18d/0x490 [lmv] [ 9712.119912] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 9712.120383] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9712.130942] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9712.130980] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9712.130986] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9712.130992] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 9712.130995] RDX: 00000000000030b1 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 9712.130997] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 9712.131000] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 9712.131003] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9712.131007] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 9712.131010] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9712.131012] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 9712.131208] Call Trace: [ 9712.131225] [ 9712.131231] ? show_regs.cold.9+0x22/0x2f [ 9712.131243] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9712.131250] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9712.131255] ? watchdog+0x40/0x40 [ 9712.131260] ? hrtimer_interrupt+0x128/0x2c0 [ 9712.131264] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9712.131274] ? apic_timer_interrupt+0xf/0x20 [ 9712.131279] [ 9712.131282] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9712.131287] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9712.131292] do_raw_spin_lock+0x177/0x190 [ 9712.131296] _raw_spin_lock+0x21/0x30 [ 9712.131303] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9712.131736] obd_mod_free+0x46/0x140 [ptlrpc] [ 9712.132207] mdc_commit_open+0x91/0xb0 [mdc] [ 9712.132258] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9712.132461] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9712.132722] after_reply+0xe2c/0x1780 [ptlrpc] [ 9712.133343] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9712.134132] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9712.134365] ? do_wait_intr+0xf0/0xf0 [ 9712.134376] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9712.135612] kthread+0x1d1/0x200 [ 9712.135620] ? set_kthread_struct+0x70/0x70 [ 9712.135624] ret_from_fork+0x1f/0x30 [ 9712.545790] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9712.553499] ll_md_close+0x274/0x800 [lustre] [ 9712.556375] ll_file_release+0x128/0xc50 [lustre] [ 9712.559859] __fput+0x10d/0x440 [ 9712.562406] ____fput+0x12/0x20 [ 9712.564501] task_work_run+0xc6/0x110 [ 9712.566497] exit_to_usermode_loop+0x1dd/0x1f0 [ 9712.569497] do_syscall_64+0x3d6/0x440 [ 9712.572303] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9712.574472] RIP: 0033:0x7ff54af45cd8 [ 9712.577792] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9712.590644] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9712.599488] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 9712.604308] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 9712.610221] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 9712.618521] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 9712.628660] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9716.705807] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7757s! [ 9716.710573] Showing busy workqueues and worker pools: [ 9716.716449] workqueue events: flags=0x0 [ 9716.718445] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9716.721482] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 9716.725962] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 9716.732157] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 9716.740952] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9716.761486] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 9716.776453] workqueue events_long: flags=0x0 [ 9716.783074] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9716.791624] pending: lu_objects_flush [obdclass] [ 9716.793456] workqueue events_unbound: flags=0x2 [ 9716.809296] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 9716.823967] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9716.841053] workqueue events_power_efficient: flags=0x80 [ 9716.846493] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9716.849734] pending: neigh_periodic_work [ 9716.853608] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9716.860559] pending: do_cache_clean, check_lifetime [ 9716.866498] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9716.889455] pending: neigh_periodic_work [ 9716.892735] workqueue writeback: flags=0x4a [ 9716.895574] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9716.904591] pending: wb_workfn, wb_workfn [ 9716.909345] workqueue xprtiod: flags=0xa [ 9716.913676] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9716.918509] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 9716.927528] workqueue ipv6_addrconf: flags=0x40008 [ 9716.930968] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9716.937589] pending: addrconf_verify_work [ 9716.944718] workqueue knbd0-recv: flags=0x1a [ 9716.946632] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9716.955469] in-flight: 657:recv_work [ 9716.960875] workqueue ptlrpc_pinger: flags=0x2 [ 9716.962941] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9716.968260] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 9716.972687] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7758s workers=4 idle: 20831 12642 28859 [ 9716.981204] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9707s workers=2 idle: 659 [ 9720.060555] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 9720.064950] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9720.084120] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9720.088650] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9720.092429] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9720.095099] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9720.105386] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9720.108428] RAX: 000000000000520f RBX: ffff96848811cae0 RCX: 0000000000000001 [ 9720.111328] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 9720.115132] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9720.118259] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 9720.121311] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 9720.124396] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 9720.127815] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9720.130224] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 9720.132975] Call Trace: [ 9720.134079] [ 9720.134946] ? show_regs.cold.9+0x22/0x2f [ 9720.136417] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9720.138822] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9720.140857] ? watchdog+0x40/0x40 [ 9720.142309] ? hrtimer_interrupt+0x128/0x2c0 [ 9720.145248] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9720.147960] ? apic_timer_interrupt+0xf/0x20 [ 9720.149810] [ 9720.150352] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9720.152255] do_raw_spin_lock+0x177/0x190 [ 9720.154322] _raw_spin_lock+0x21/0x30 [ 9720.155576] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9720.158263] ? do_raw_spin_unlock+0x75/0x190 [ 9720.160442] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9720.162947] ? do_wait_intr+0xf0/0xf0 [ 9720.164512] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9720.167176] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9720.169892] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9720.172516] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9720.175873] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9720.178080] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9720.179694] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9720.182935] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9720.186449] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9720.189547] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9720.193453] lmv_intent_open+0x338/0x1150 [lmv] [ 9720.197043] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9720.199534] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9720.202455] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9720.205064] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9720.207871] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9720.210900] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9720.213545] ? do_raw_spin_unlock+0x75/0x190 [ 9720.215847] ? _raw_spin_unlock+0x12/0x30 [ 9720.218189] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9720.220006] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9720.222524] ll_atomic_open+0x36a/0xd70 [lustre] [ 9720.224290] lookup_open+0x356/0xb40 [ 9720.228612] do_last+0x313/0xfc0 [ 9720.233772] ? nd_jump_root+0xe5/0x160 [ 9720.236229] ? path_init+0x437/0x520 [ 9720.240934] path_openat+0xf7/0x500 [ 9720.247662] ? do_raw_spin_unlock+0x75/0x190 [ 9720.249016] do_filp_open+0x99/0x140 [ 9720.255428] ? getname_flags+0x6e/0x330 [ 9720.260380] ? __check_object_size+0xff/0x256 [ 9720.261743] ? do_raw_spin_unlock+0x75/0x190 [ 9720.267389] ? _raw_spin_unlock+0x12/0x30 [ 9720.269363] do_sys_openat2+0x2b4/0x410 [ 9720.270825] do_sys_open+0x73/0xa0 [ 9720.272509] __x64_sys_openat+0x24/0x30 [ 9720.274832] do_syscall_64+0xc1/0x440 [ 9720.276509] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9720.281525] RIP: 0033:0x7ff54af45332 [ 9720.289583] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9720.310511] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9720.315958] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 9720.324183] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 9720.326370] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 9720.348843] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 9720.354189] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 9736.059908] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 9736.071979] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9736.118499] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9736.129294] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9736.134290] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9736.141114] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9736.147712] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9736.160793] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9736.164963] RAX: 0000000000007d41 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 9736.170001] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 9736.175348] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 9736.181779] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 9736.188319] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 9736.190753] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 9736.203153] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9736.207900] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 9736.213501] Call Trace: [ 9736.214911] [ 9736.217929] ? show_regs.cold.9+0x22/0x2f [ 9736.219384] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9736.223498] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9736.225658] ? watchdog+0x40/0x40 [ 9736.227954] ? hrtimer_interrupt+0x128/0x2c0 [ 9736.232775] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9736.235208] ? apic_timer_interrupt+0xf/0x20 [ 9736.237386] [ 9736.239392] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9736.242537] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9736.246527] do_raw_spin_lock+0x177/0x190 [ 9736.250521] _raw_spin_lock+0x21/0x30 [ 9736.252444] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9736.257102] process_one_work+0x2c8/0x700 [ 9736.259743] worker_thread+0x296/0x6e0 [ 9736.265049] ? rescuer_thread+0x570/0x570 [ 9736.267181] kthread+0x1d1/0x200 [ 9736.270562] ? set_kthread_struct+0x70/0x70 [ 9736.273819] ret_from_fork+0x1f/0x30 [ 9740.058087] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [ 9740.061465] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9740.081941] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9740.087459] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9740.091985] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9740.094039] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9740.102703] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9740.106723] RAX: 00000000000008a0 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 9740.114056] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 9740.119195] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9740.122855] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 9740.126491] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 9740.129676] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 9740.133202] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9740.136163] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 9740.142539] Call Trace: [ 9740.144112] [ 9740.144909] ? show_regs.cold.9+0x22/0x2f [ 9740.147052] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9740.148970] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9740.150582] ? watchdog+0x40/0x40 [ 9740.154509] ? hrtimer_interrupt+0x128/0x2c0 [ 9740.156446] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9740.159149] ? apic_timer_interrupt+0xf/0x20 [ 9740.161497] [ 9740.162466] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9740.164613] do_raw_spin_lock+0x177/0x190 [ 9740.166536] _raw_spin_lock+0x21/0x30 [ 9740.168098] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9740.169895] ? do_raw_spin_unlock+0x75/0x190 [ 9740.171650] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9740.174669] ? do_wait_intr+0xf0/0xf0 [ 9740.176630] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9740.179831] mdc_close+0x53b/0xd70 [mdc] [ 9740.181591] lmv_close+0x18d/0x490 [lmv] [ 9740.185968] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 9740.191553] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9740.198230] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9740.200705] ll_md_close+0x274/0x800 [lustre] [ 9740.204537] ll_file_release+0x128/0xc50 [lustre] [ 9740.208394] __fput+0x10d/0x440 [ 9740.213275] ____fput+0x12/0x20 [ 9740.214684] task_work_run+0xc6/0x110 [ 9740.217668] exit_to_usermode_loop+0x1dd/0x1f0 [ 9740.223133] do_syscall_64+0x3d6/0x440 [ 9740.224478] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9740.227642] RIP: 0033:0x7ff54af45cd8 [ 9740.229969] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9740.239006] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9740.244589] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 9740.254474] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 9740.260757] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 9740.265429] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 9740.269312] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9744.055005] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 9744.059385] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9744.081798] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9744.088664] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9744.094235] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9744.097083] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9744.119025] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9744.123398] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 9744.127117] RDX: 0000000000007f8e RSI: 0000000000000003 RDI: ffff96848811cae0 [ 9744.130701] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 9744.133456] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 9744.139611] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9744.145269] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 9744.150192] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9744.153421] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 9744.155524] Call Trace: [ 9744.156340] [ 9744.158172] ? show_regs.cold.9+0x22/0x2f [ 9744.159237] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9744.161725] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9744.164508] ? watchdog+0x40/0x40 [ 9744.166529] ? hrtimer_interrupt+0x128/0x2c0 [ 9744.172632] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9744.176110] ? apic_timer_interrupt+0xf/0x20 [ 9744.177203] [ 9744.177649] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9744.180273] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9744.182951] do_raw_spin_lock+0x177/0x190 [ 9744.184882] _raw_spin_lock+0x21/0x30 [ 9744.186481] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9744.189253] obd_mod_free+0x46/0x140 [ptlrpc] [ 9744.191564] mdc_commit_open+0x91/0xb0 [mdc] [ 9744.193486] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9744.198014] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9744.204075] after_reply+0xe2c/0x1780 [ptlrpc] [ 9744.211080] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9744.213186] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9744.216379] ? do_wait_intr+0xf0/0xf0 [ 9744.217442] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9744.219851] kthread+0x1d1/0x200 [ 9744.220638] ? set_kthread_struct+0x70/0x70 [ 9744.222181] ret_from_fork+0x1f/0x30 [ 9747.423071] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7788s! [ 9747.427261] Showing busy workqueues and worker pools: [ 9747.438095] workqueue events: flags=0x0 [ 9747.439809] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9747.443515] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 9747.448966] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 9747.456179] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 9747.470517] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9747.478986] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 9747.486324] workqueue events_long: flags=0x0 [ 9747.491983] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9747.504809] pending: lu_objects_flush [obdclass] [ 9747.508637] workqueue events_unbound: flags=0x2 [ 9747.512488] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 9747.518486] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9747.537243] workqueue events_power_efficient: flags=0x80 [ 9747.543054] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9747.559535] pending: neigh_periodic_work [ 9747.566989] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9747.577470] pending: do_cache_clean, check_lifetime [ 9747.584963] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9747.590784] pending: neigh_periodic_work [ 9747.597186] workqueue writeback: flags=0x4a [ 9747.604016] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9747.608097] pending: wb_workfn, wb_workfn [ 9747.610666] workqueue xprtiod: flags=0xa [ 9747.612563] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9747.619570] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 9747.623244] workqueue ipv6_addrconf: flags=0x40008 [ 9747.626484] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9747.630643] pending: addrconf_verify_work [ 9747.633649] workqueue knbd0-recv: flags=0x1a [ 9747.636154] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9747.640240] in-flight: 657:recv_work [ 9747.642410] workqueue ptlrpc_pinger: flags=0x2 [ 9747.644961] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9747.652289] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 9747.656600] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7788s workers=4 idle: 20831 12642 28859 [ 9747.662134] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9737s workers=2 idle: 659 [ 9748.060025] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 9748.063194] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9748.087903] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9748.099167] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9748.103451] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9748.106361] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9748.119935] RSP: 0000:ffffaf9fc15234f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9748.123575] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 9748.127140] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 9748.130453] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9748.135375] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 9748.138632] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 9748.143716] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 9748.150673] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9748.154015] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 9748.159098] Call Trace: [ 9748.161175] [ 9748.162942] ? show_regs.cold.9+0x22/0x2f [ 9748.166531] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9748.169548] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9748.172142] ? watchdog+0x40/0x40 [ 9748.173688] ? hrtimer_interrupt+0x128/0x2c0 [ 9748.175891] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9748.177216] ? apic_timer_interrupt+0xf/0x20 [ 9748.180454] [ 9748.181921] ? native_safe_halt+0x2e/0x50 [ 9748.186258] kvm_wait+0xcd/0xe0 [ 9748.187739] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9748.190630] do_raw_spin_lock+0x177/0x190 [ 9748.200883] _raw_spin_lock+0x21/0x30 [ 9748.203677] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9748.212185] ? do_raw_spin_unlock+0x75/0x190 [ 9748.214901] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9748.220108] ? do_wait_intr+0xf0/0xf0 [ 9748.222417] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9748.234523] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9748.242086] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9748.249522] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9748.253450] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9748.258523] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9748.263446] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9748.265976] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9748.271949] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9748.276740] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9748.282263] lmv_intent_open+0x338/0x1150 [lmv] [ 9748.288570] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9748.294044] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9748.300246] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9748.302605] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9748.308297] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9748.311505] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9748.319910] ? do_raw_spin_unlock+0x75/0x190 [ 9748.322522] ? _raw_spin_unlock+0x12/0x30 [ 9748.328474] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9748.333257] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9748.339470] ll_atomic_open+0x36a/0xd70 [lustre] [ 9748.344029] lookup_open+0x356/0xb40 [ 9748.349943] do_last+0x313/0xfc0 [ 9748.351619] ? nd_jump_root+0xe5/0x160 [ 9748.355470] ? path_init+0x437/0x520 [ 9748.357619] path_openat+0xf7/0x500 [ 9748.362632] ? do_raw_spin_unlock+0x75/0x190 [ 9748.366671] do_filp_open+0x99/0x140 [ 9748.374630] ? getname_flags+0x6e/0x330 [ 9748.376681] ? __check_object_size+0xff/0x256 [ 9748.383483] ? do_raw_spin_unlock+0x75/0x190 [ 9748.384803] ? _raw_spin_unlock+0x12/0x30 [ 9748.388148] do_sys_openat2+0x2b4/0x410 [ 9748.390493] do_sys_open+0x73/0xa0 [ 9748.393035] __x64_sys_openat+0x24/0x30 [ 9748.395512] do_syscall_64+0xc1/0x440 [ 9748.399138] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9748.402581] RIP: 0033:0x7ff54af45332 [ 9748.404907] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9748.423130] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9748.429391] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 9748.433026] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 9748.439679] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 9748.442131] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 9748.455501] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 9754.068278] rcu: INFO: rcu_sched self-detected stall on CPU [ 9754.075626] rcu: 0-...!: (4070970 ticks this GP) idle=98a/1/0x4000000000000002 softirq=125154/125154 fqs=959 [ 9754.088284] (t=7800228 jiffies g=253585 q=5417) [ 9754.095643] rcu: rcu_sched kthread starved for 7795188 jiffies! g253585 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=3 [ 9754.101933] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 9754.113012] rcu: RCU grace-period kthread stack dump: [ 9754.119363] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 9754.124734] Call Trace: [ 9754.128912] __schedule+0x351/0xcb0 [ 9754.132046] schedule+0xc0/0x180 [ 9754.134228] schedule_timeout+0xb4/0x190 [ 9754.135619] ? do_raw_spin_unlock+0x75/0x190 [ 9754.138312] ? __next_timer_interrupt+0x160/0x160 [ 9754.142091] ? prepare_to_swait_event+0xf3/0x140 [ 9754.145984] rcu_gp_kthread+0x8dd/0x15b0 [ 9754.148562] ? force_qs_rnp+0x2c0/0x2c0 [ 9754.152398] kthread+0x1d1/0x200 [ 9754.155876] ? set_kthread_struct+0x70/0x70 [ 9754.158600] ret_from_fork+0x1f/0x30 [ 9754.163149] rcu: Stack dump where RCU GP kthread last ran: [ 9754.167213] Sending NMI from CPU 0 to CPUs 3: [ 9754.176517] NMI backtrace for cpu 3 [ 9754.176527] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9754.176530] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9754.176532] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9754.176534] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9754.176536] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 [ 9754.176540] RAX: 0000000000002e1e RBX: ffff96848811cae0 RCX: 0000000000000001 [ 9754.176542] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 9754.176543] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9754.176545] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 9754.176623] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 9754.176793] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 9754.176795] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9754.176796] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 9754.176797] Call Trace: [ 9754.176798] [ 9754.176799] ? show_regs.cold.9+0x22/0x2f [ 9754.176800] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 9754.176801] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 9754.176803] ? nmi_handle+0xbd/0x210 [ 9754.176804] ? default_do_nmi+0x4b/0x1b0 [ 9754.176805] ? do_nmi+0x280/0x380 [ 9754.176816] ? end_repeat_nmi+0x16/0x1f [ 9754.176817] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9754.176818] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9754.176820] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9754.176821] [ 9754.176822] do_raw_spin_lock+0x177/0x190 [ 9754.176823] _raw_spin_lock+0x21/0x30 [ 9754.176824] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9754.176826] ? do_raw_spin_unlock+0x75/0x190 [ 9754.176827] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9754.176828] ? do_wait_intr+0xf0/0xf0 [ 9754.176830] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9754.176831] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9754.176833] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9754.176834] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9754.176835] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9754.176837] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9754.176838] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9754.176839] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9754.176841] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9754.176842] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9754.176844] lmv_intent_open+0x338/0x1150 [lmv] [ 9754.176845] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9754.176847] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9754.176848] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9754.176849] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9754.176850] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9754.176852] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9754.176853] ? do_raw_spin_unlock+0x75/0x190 [ 9754.176854] ? _raw_spin_unlock+0x12/0x30 [ 9754.176856] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9754.176858] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9754.176859] ll_atomic_open+0x36a/0xd70 [lustre] [ 9754.176860] lookup_open+0x356/0xb40 [ 9754.176862] do_last+0x313/0xfc0 [ 9754.176863] ? nd_jump_root+0xe5/0x160 [ 9754.176864] ? path_init+0x437/0x520 [ 9754.176865] path_openat+0xf7/0x500 [ 9754.176867] ? do_raw_spin_unlock+0x75/0x190 [ 9754.176868] do_filp_open+0x99/0x140 [ 9754.176869] ? getname_flags+0x6e/0x330 [ 9754.176871] ? __check_object_size+0xff/0x256 [ 9754.176872] ? do_raw_spin_unlock+0x75/0x190 [ 9754.176873] ? _raw_spin_unlock+0x12/0x30 [ 9754.176875] do_sys_openat2+0x2b4/0x410 [ 9754.176875] do_sys_open+0x73/0xa0 [ 9754.176877] __x64_sys_openat+0x24/0x30 [ 9754.176878] do_syscall_64+0xc1/0x440 [ 9754.176879] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9754.176881] RIP: 0033:0x7ff54af45332 [ 9754.176882] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9754.176884] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9754.176888] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 9754.176890] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 9754.176891] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 9754.176893] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 9754.176894] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 9754.178685] NMI backtrace for cpu 0 [ 9754.517202] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9754.522891] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9754.526931] Call Trace: [ 9754.528041] [ 9754.528990] ? dump_stack+0xbb/0x10e [ 9754.530875] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 9754.532688] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 9754.537878] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 9754.539925] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 9754.542368] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 9754.545124] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 9754.547143] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 9754.550045] ? update_process_times+0x6d/0xc0 [ 9754.552001] ? tick_sched_handle.isra.18+0x27/0x90 [ 9754.554620] ? tick_sched_timer+0x53/0xb0 [ 9754.557837] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9754.560331] ? tick_sched_handle.isra.18+0x90/0x90 [ 9754.562964] ? hrtimer_interrupt+0x128/0x2c0 [ 9754.565168] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9754.567416] ? apic_timer_interrupt+0xf/0x20 [ 9754.569536] [ 9754.570542] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9754.575067] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9754.578170] ? do_raw_spin_lock+0x177/0x190 [ 9754.580026] ? _raw_spin_lock+0x21/0x30 [ 9754.581863] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9754.584667] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 9754.588368] ? mdc_commit_open+0x91/0xb0 [mdc] [ 9754.591419] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9754.595944] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9754.602446] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 9754.607450] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9754.610364] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9754.612858] ? do_wait_intr+0xf0/0xf0 [ 9754.615883] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9754.618790] ? kthread+0x1d1/0x200 [ 9754.620559] ? set_kthread_struct+0x70/0x70 [ 9754.622977] ? ret_from_fork+0x1f/0x30 [ 9764.064236] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 9764.076891] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9764.127241] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9764.140095] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9764.147415] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9764.154311] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9764.161199] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9764.190503] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9764.201597] RAX: 0000000000004483 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 9764.214340] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 9764.226466] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 9764.237285] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 9764.247455] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 9764.263200] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 9764.276502] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9764.284014] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 9764.295264] Call Trace: [ 9764.298522] [ 9764.300973] ? show_regs.cold.9+0x22/0x2f [ 9764.308479] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9764.313905] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9764.317936] ? watchdog+0x40/0x40 [ 9764.318996] ? hrtimer_interrupt+0x128/0x2c0 [ 9764.334980] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9764.337371] ? apic_timer_interrupt+0xf/0x20 [ 9764.351995] [ 9764.352796] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9764.362355] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9764.371435] do_raw_spin_lock+0x177/0x190 [ 9764.377519] _raw_spin_lock+0x21/0x30 [ 9764.391237] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9764.396381] process_one_work+0x2c8/0x700 [ 9764.397405] worker_thread+0x296/0x6e0 [ 9764.410667] ? rescuer_thread+0x570/0x570 [ 9764.411836] kthread+0x1d1/0x200 [ 9764.412638] ? set_kthread_struct+0x70/0x70 [ 9764.425807] ret_from_fork+0x1f/0x30 [ 9768.060579] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [ 9768.069094] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9768.105905] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9768.110964] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9768.116167] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9768.118996] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9768.125947] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9768.129150] RAX: 0000000000006d36 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 9768.134234] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 9768.138041] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9768.141564] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 9768.144828] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 9768.147649] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 9768.150934] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9768.153164] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 9768.155981] Call Trace: [ 9768.156990] [ 9768.157855] ? show_regs.cold.9+0x22/0x2f [ 9768.159717] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9768.161579] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9768.163352] ? watchdog+0x40/0x40 [ 9768.164742] ? hrtimer_interrupt+0x128/0x2c0 [ 9768.166064] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9768.167565] ? apic_timer_interrupt+0xf/0x20 [ 9768.168878] [ 9768.169547] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9768.171624] do_raw_spin_lock+0x177/0x190 [ 9768.173224] _raw_spin_lock+0x21/0x30 [ 9768.174668] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9768.176882] ? do_raw_spin_unlock+0x75/0x190 [ 9768.178610] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9768.180758] ? do_wait_intr+0xf0/0xf0 [ 9768.182173] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9768.184267] mdc_close+0x53b/0xd70 [mdc] [ 9768.185858] lmv_close+0x18d/0x490 [lmv] [ 9768.187370] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 9768.189727] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9768.192173] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9768.194092] ll_md_close+0x274/0x800 [lustre] [ 9768.195949] ll_file_release+0x128/0xc50 [lustre] [ 9768.197887] __fput+0x10d/0x440 [ 9768.198983] ____fput+0x12/0x20 [ 9768.203369] task_work_run+0xc6/0x110 [ 9768.205783] exit_to_usermode_loop+0x1dd/0x1f0 [ 9768.207379] do_syscall_64+0x3d6/0x440 [ 9768.213776] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9768.217705] RIP: 0033:0x7ff54af45cd8 [ 9768.219402] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9768.233364] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9768.240724] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 9768.246357] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 9768.254755] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 9768.261606] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 9768.266856] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9778.143058] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7819s! [ 9778.147432] Showing busy workqueues and worker pools: [ 9778.152989] workqueue events: flags=0x0 [ 9778.154884] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9778.160794] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 9778.166042] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 9778.168938] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 9778.179226] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9778.186212] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 9778.188559] workqueue events_long: flags=0x0 [ 9778.191511] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9778.198770] pending: lu_objects_flush [obdclass] [ 9778.200798] workqueue events_unbound: flags=0x2 [ 9778.206830] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 9778.210786] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9778.222228] workqueue events_power_efficient: flags=0x80 [ 9778.229030] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9778.236146] pending: neigh_periodic_work [ 9778.239382] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9778.245489] pending: do_cache_clean, check_lifetime [ 9778.249748] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9778.257323] pending: neigh_periodic_work [ 9778.264020] workqueue writeback: flags=0x4a [ 9778.266197] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9778.273264] pending: wb_workfn, wb_workfn [ 9778.281218] workqueue xprtiod: flags=0xa [ 9778.287305] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9778.294230] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 9778.307185] workqueue ipv6_addrconf: flags=0x40008 [ 9778.315245] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9778.321385] pending: addrconf_verify_work [ 9778.324167] workqueue knbd0-recv: flags=0x1a [ 9778.329290] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9778.333676] in-flight: 657:recv_work [ 9778.339096] workqueue ptlrpc_pinger: flags=0x2 [ 9778.344954] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9778.353501] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 9778.360156] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7819s workers=4 idle: 20831 12642 28859 [ 9778.369568] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9768s workers=2 idle: 659 [ 9780.060065] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 9780.065855] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9780.095254] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9780.101054] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9780.107449] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9780.111703] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9780.126482] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9780.131641] RAX: 0000000000005f18 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 9780.136408] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 9780.142444] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9780.148141] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 9780.150738] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 9780.157280] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 9780.165183] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9780.169184] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 9780.174153] Call Trace: [ 9780.176758] [ 9780.178056] ? show_regs.cold.9+0x22/0x2f [ 9780.180729] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9780.184045] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9780.187176] ? watchdog+0x40/0x40 [ 9780.188983] ? hrtimer_interrupt+0x128/0x2c0 [ 9780.191152] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9780.194083] ? apic_timer_interrupt+0xf/0x20 [ 9780.196379] [ 9780.197536] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9780.200474] do_raw_spin_lock+0x177/0x190 [ 9780.202795] _raw_spin_lock+0x21/0x30 [ 9780.205298] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9780.209440] ? do_raw_spin_unlock+0x75/0x190 [ 9780.211977] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9780.214906] ? do_wait_intr+0xf0/0xf0 [ 9780.217076] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9780.224667] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9780.231186] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9780.248308] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9780.257983] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9780.266571] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9780.268658] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9780.271607] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9780.274824] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9780.281284] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9780.285923] lmv_intent_open+0x338/0x1150 [lmv] [ 9780.288747] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9780.292442] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9780.295462] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9780.298374] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9780.301394] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9780.304369] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9780.307704] ? do_raw_spin_unlock+0x75/0x190 [ 9780.310326] ? _raw_spin_unlock+0x12/0x30 [ 9780.313304] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9780.317370] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9780.320036] ll_atomic_open+0x36a/0xd70 [lustre] [ 9780.322169] lookup_open+0x356/0xb40 [ 9780.324263] do_last+0x313/0xfc0 [ 9780.326470] ? nd_jump_root+0xe5/0x160 [ 9780.328615] ? path_init+0x437/0x520 [ 9780.330541] path_openat+0xf7/0x500 [ 9780.332508] ? do_raw_spin_unlock+0x75/0x190 [ 9780.335059] do_filp_open+0x99/0x140 [ 9780.337837] ? getname_flags+0x6e/0x330 [ 9780.340769] ? __check_object_size+0xff/0x256 [ 9780.343075] ? do_raw_spin_unlock+0x75/0x190 [ 9780.345630] ? _raw_spin_unlock+0x12/0x30 [ 9780.348055] do_sys_openat2+0x2b4/0x410 [ 9780.349770] do_sys_open+0x73/0xa0 [ 9780.351803] __x64_sys_openat+0x24/0x30 [ 9780.353988] do_syscall_64+0xc1/0x440 [ 9780.356738] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9780.359692] RIP: 0033:0x7ff54af45332 [ 9780.361637] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9780.374078] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9780.378737] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 9780.383083] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 9780.387508] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 9780.391912] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 9780.396291] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 9792.059031] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 9792.065854] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9792.113098] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9792.123750] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9792.132202] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9792.140040] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9792.145603] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9792.164632] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9792.171204] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 9792.177078] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 9792.181925] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 9792.189896] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 9792.197114] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 9792.208199] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 9792.217227] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9792.221216] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 9792.230571] Call Trace: [ 9792.232895] [ 9792.235033] ? show_regs.cold.9+0x22/0x2f [ 9792.240334] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9792.242089] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9792.252526] ? watchdog+0x40/0x40 [ 9792.257576] ? hrtimer_interrupt+0x128/0x2c0 [ 9792.261056] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9792.266964] ? apic_timer_interrupt+0xf/0x20 [ 9792.272781] [ 9792.274849] ? native_safe_halt+0x2e/0x50 [ 9792.278831] kvm_wait+0xcd/0xe0 [ 9792.283601] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9792.290060] do_raw_spin_lock+0x177/0x190 [ 9792.293213] _raw_spin_lock+0x21/0x30 [ 9792.296834] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9792.300097] process_one_work+0x2c8/0x700 [ 9792.302533] worker_thread+0x296/0x6e0 [ 9792.304856] ? rescuer_thread+0x570/0x570 [ 9792.307628] kthread+0x1d1/0x200 [ 9792.309842] ? set_kthread_struct+0x70/0x70 [ 9792.312148] ret_from_fork+0x1f/0x30 [ 9796.063936] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [ 9796.082927] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9796.146151] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9796.154355] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9796.166731] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9796.170939] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9796.184393] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9796.190675] RAX: 000000000000678a RBX: ffff96848811cae0 RCX: 0000000000000001 [ 9796.196417] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 9796.206176] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9796.213411] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 9796.220143] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 9796.229650] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 9796.238606] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9796.243764] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 9796.252239] Call Trace: [ 9796.254761] [ 9796.260100] ? show_regs.cold.9+0x22/0x2f [ 9796.261433] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9796.268063] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9796.273837] ? watchdog+0x40/0x40 [ 9796.279096] ? hrtimer_interrupt+0x128/0x2c0 [ 9796.280431] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9796.282535] ? apic_timer_interrupt+0xf/0x20 [ 9796.283916] [ 9796.285327] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9796.289897] do_raw_spin_lock+0x177/0x190 [ 9796.295824] _raw_spin_lock+0x21/0x30 [ 9796.298832] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9796.301015] ? do_raw_spin_unlock+0x75/0x190 [ 9796.303883] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9796.307942] ? do_wait_intr+0xf0/0xf0 [ 9796.312847] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9796.317151] mdc_close+0x53b/0xd70 [mdc] [ 9796.320224] lmv_close+0x18d/0x490 [lmv] [ 9796.324332] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 9796.326344] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9796.337086] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9796.341216] ll_md_close+0x274/0x800 [lustre] [ 9796.349318] ll_file_release+0x128/0xc50 [lustre] [ 9796.353854] __fput+0x10d/0x440 [ 9796.357358] ____fput+0x12/0x20 [ 9796.359859] task_work_run+0xc6/0x110 [ 9796.365054] exit_to_usermode_loop+0x1dd/0x1f0 [ 9796.373705] do_syscall_64+0x3d6/0x440 [ 9796.380652] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9796.384647] RIP: 0033:0x7ff54af45cd8 [ 9796.385910] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9796.398643] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9796.401831] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 9796.411197] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 9796.414001] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 9796.418111] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 9796.420958] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9804.056449] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 9804.066735] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9804.116407] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9804.128996] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9804.141512] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9804.148848] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9804.173725] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9804.183581] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 9804.192777] RDX: 0000000000007f1f RSI: 0000000000000003 RDI: ffff96848811cae0 [ 9804.199813] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 9804.208222] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 9804.213253] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9804.218877] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 9804.225014] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9804.232110] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 9804.240028] Call Trace: [ 9804.241680] [ 9804.245646] ? show_regs.cold.9+0x22/0x2f [ 9804.250846] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9804.256091] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9804.260780] ? watchdog+0x40/0x40 [ 9804.262968] ? hrtimer_interrupt+0x128/0x2c0 [ 9804.265686] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9804.269254] ? apic_timer_interrupt+0xf/0x20 [ 9804.273362] [ 9804.275112] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9804.281689] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9804.286923] do_raw_spin_lock+0x177/0x190 [ 9804.292257] _raw_spin_lock+0x21/0x30 [ 9804.293932] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9804.301441] obd_mod_free+0x46/0x140 [ptlrpc] [ 9804.306809] mdc_commit_open+0x91/0xb0 [mdc] [ 9804.309382] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9804.312727] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9804.316731] after_reply+0xe2c/0x1780 [ptlrpc] [ 9804.319711] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9804.322947] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9804.325768] ? do_wait_intr+0xf0/0xf0 [ 9804.327802] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9804.331685] kthread+0x1d1/0x200 [ 9804.333567] ? set_kthread_struct+0x70/0x70 [ 9804.336491] ret_from_fork+0x1f/0x30 [ 9808.060025] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 9808.062989] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9808.125836] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9808.142320] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9808.159038] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9808.166579] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9808.184991] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9808.193417] RAX: 00000000000023a3 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 9808.205216] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 9808.213173] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9808.219703] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 9808.232425] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 9808.244713] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 9808.269330] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9808.281721] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 9808.294222] Call Trace: [ 9808.298059] [ 9808.301630] ? show_regs.cold.9+0x22/0x2f [ 9808.305609] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9808.314478] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9808.315508] ? watchdog+0x40/0x40 [ 9808.316221] ? hrtimer_interrupt+0x128/0x2c0 [ 9808.317187] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9808.331021] ? apic_timer_interrupt+0xf/0x20 [ 9808.342490] [ 9808.343280] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9808.344919] do_raw_spin_lock+0x177/0x190 [ 9808.352737] _raw_spin_lock+0x21/0x30 [ 9808.357639] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9808.361604] ? do_raw_spin_unlock+0x75/0x190 [ 9808.366712] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9808.372407] ? do_wait_intr+0xf0/0xf0 [ 9808.376002] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9808.382252] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9808.386347] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9808.391435] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9808.397999] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9808.407311] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9808.414370] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9808.417576] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9808.422840] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9808.434427] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9808.441318] lmv_intent_open+0x338/0x1150 [lmv] [ 9808.447696] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9808.452031] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9808.455749] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9808.460095] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9808.465962] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9808.471538] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9808.476380] ? do_raw_spin_unlock+0x75/0x190 [ 9808.481719] ? _raw_spin_unlock+0x12/0x30 [ 9808.483175] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9808.490750] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9808.500859] ll_atomic_open+0x36a/0xd70 [lustre] [ 9808.508522] lookup_open+0x356/0xb40 [ 9808.512505] do_last+0x313/0xfc0 [ 9808.519006] ? nd_jump_root+0xe5/0x160 [ 9808.523572] ? path_init+0x437/0x520 [ 9808.534100] path_openat+0xf7/0x500 [ 9808.535866] ? do_raw_spin_unlock+0x75/0x190 [ 9808.540344] do_filp_open+0x99/0x140 [ 9808.544292] ? getname_flags+0x6e/0x330 [ 9808.553275] ? __check_object_size+0xff/0x256 [ 9808.557881] ? do_raw_spin_unlock+0x75/0x190 [ 9808.562419] ? _raw_spin_unlock+0x12/0x30 [ 9808.572036] do_sys_openat2+0x2b4/0x410 [ 9808.574442] do_sys_open+0x73/0xa0 [ 9808.576007] __x64_sys_openat+0x24/0x30 [ 9808.581987] do_syscall_64+0xc1/0x440 [ 9808.585837] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9808.589292] RIP: 0033:0x7ff54af45332 [ 9808.592181] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9808.604946] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9808.616968] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 9808.628457] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 9808.635200] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 9808.641449] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 9808.651379] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 9808.863139] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7849s! [ 9808.866903] Showing busy workqueues and worker pools: [ 9808.869664] workqueue events: flags=0x0 [ 9808.871818] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9808.876459] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 9808.885015] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 9808.889452] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 9808.901355] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9808.906700] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 9808.914048] workqueue events_long: flags=0x0 [ 9808.919427] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9808.923409] pending: lu_objects_flush [obdclass] [ 9808.926372] workqueue events_unbound: flags=0x2 [ 9808.929031] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 9808.932420] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9808.938751] workqueue events_power_efficient: flags=0x80 [ 9808.942162] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9808.946170] pending: neigh_periodic_work [ 9808.948472] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9808.952347] pending: do_cache_clean, check_lifetime [ 9808.955571] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9808.959675] pending: neigh_periodic_work [ 9808.961677] workqueue writeback: flags=0x4a [ 9808.963325] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9808.966375] pending: wb_workfn, wb_workfn [ 9808.968660] workqueue xprtiod: flags=0xa [ 9808.971408] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9808.979623] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 9808.991812] workqueue ipv6_addrconf: flags=0x40008 [ 9808.996608] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9809.007743] pending: addrconf_verify_work [ 9809.015734] workqueue knbd0-recv: flags=0x1a [ 9809.017831] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9809.027430] in-flight: 657:recv_work [ 9809.029580] workqueue ptlrpc_pinger: flags=0x2 [ 9809.031216] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9809.036886] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 9809.040731] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7850s workers=4 idle: 20831 12642 28859 [ 9809.046758] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9799s workers=2 idle: 659 [ 9820.059988] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 9820.065956] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9820.087937] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9820.094493] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9820.100649] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9820.104852] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9820.108768] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9820.119097] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9820.122959] RAX: 0000000000007e8d RBX: ffff96848811cae0 RCX: 0000000000000001 [ 9820.126617] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 9820.132872] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 9820.136151] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 9820.140191] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 9820.143750] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 9820.147414] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9820.149612] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 9820.153428] Call Trace: [ 9820.154352] [ 9820.155013] ? show_regs.cold.9+0x22/0x2f [ 9820.156726] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9820.158868] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9820.161654] ? watchdog+0x40/0x40 [ 9820.163127] ? hrtimer_interrupt+0x128/0x2c0 [ 9820.165755] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9820.167832] ? apic_timer_interrupt+0xf/0x20 [ 9820.169461] [ 9820.170511] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9820.172630] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9820.175521] do_raw_spin_lock+0x177/0x190 [ 9820.176916] _raw_spin_lock+0x21/0x30 [ 9820.178864] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9820.187340] process_one_work+0x2c8/0x700 [ 9820.191352] worker_thread+0x296/0x6e0 [ 9820.195142] ? rescuer_thread+0x570/0x570 [ 9820.196929] kthread+0x1d1/0x200 [ 9820.198670] ? set_kthread_struct+0x70/0x70 [ 9820.201050] ret_from_fork+0x1f/0x30 [ 9824.062786] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 9824.076792] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9824.115461] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9824.117921] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9824.127625] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9824.130457] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9824.137936] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9824.143793] RAX: 0000000000007f83 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 9824.147703] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 9824.151013] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9824.155591] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 9824.167881] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 9824.175641] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 9824.182390] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9824.185702] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 9824.189384] Call Trace: [ 9824.190497] [ 9824.191498] ? show_regs.cold.9+0x22/0x2f [ 9824.194254] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9824.199067] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9824.204177] ? watchdog+0x40/0x40 [ 9824.205360] ? hrtimer_interrupt+0x128/0x2c0 [ 9824.207472] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9824.209095] ? apic_timer_interrupt+0xf/0x20 [ 9824.215546] [ 9824.219584] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9824.222987] do_raw_spin_lock+0x177/0x190 [ 9824.233385] _raw_spin_lock+0x21/0x30 [ 9824.238270] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9824.247597] ? do_raw_spin_unlock+0x75/0x190 [ 9824.249230] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9824.254325] ? do_wait_intr+0xf0/0xf0 [ 9824.258406] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9824.259828] mdc_close+0x53b/0xd70 [mdc] [ 9824.260811] lmv_close+0x18d/0x490 [lmv] [ 9824.265623] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 9824.272664] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9824.280612] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9824.285284] ll_md_close+0x274/0x800 [lustre] [ 9824.286567] ll_file_release+0x128/0xc50 [lustre] [ 9824.288908] __fput+0x10d/0x440 [ 9824.290719] ____fput+0x12/0x20 [ 9824.292459] task_work_run+0xc6/0x110 [ 9824.294910] exit_to_usermode_loop+0x1dd/0x1f0 [ 9824.296979] do_syscall_64+0x3d6/0x440 [ 9824.299939] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9824.303110] RIP: 0033:0x7ff54af45cd8 [ 9824.307699] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9824.324261] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9824.330359] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 9824.334566] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 9824.340660] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 9824.344260] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 9824.349359] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9836.055207] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 9836.059519] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9836.062832] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 9836.084853] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9836.084869] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9836.084873] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9836.084901] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9836.084906] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9836.084912] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 9836.084915] RDX: 0000000000000f31 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 9836.084917] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 9836.084920] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 9836.084922] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9836.084926] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 9836.084929] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9836.084931] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 9836.084939] Call Trace: [ 9836.084945] [ 9836.084951] ? show_regs.cold.9+0x22/0x2f [ 9836.084974] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9836.084982] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9836.084988] ? watchdog+0x40/0x40 [ 9836.084992] ? hrtimer_interrupt+0x128/0x2c0 [ 9836.084996] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9836.085006] ? apic_timer_interrupt+0xf/0x20 [ 9836.085022] [ 9836.085026] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9836.085031] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9836.085037] do_raw_spin_lock+0x177/0x190 [ 9836.085043] _raw_spin_lock+0x21/0x30 [ 9836.085050] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9836.094674] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) [ 9836.100312] obd_mod_free+0x46/0x140 [ptlrpc] [ 9836.100632] mdc_commit_open+0x91/0xb0 [mdc] [ 9836.100670] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9836.111779] mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9836.111850] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9836.111857] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9836.111860] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9836.111881] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9836.111886] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 9836.111895] RAX: 000000000000728b RBX: ffff96848811cae0 RCX: 0000000000000001 [ 9836.111897] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 9836.111900] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9836.111902] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 9836.111905] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 9836.111908] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 9836.111911] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9836.111914] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 9836.111922] Call Trace: [ 9836.111934] [ 9836.111941] ? show_regs.cold.9+0x22/0x2f [ 9836.111952] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9836.111959] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9836.111964] ? watchdog+0x40/0x40 [ 9836.111968] ? hrtimer_interrupt+0x128/0x2c0 [ 9836.111972] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9836.111981] ? apic_timer_interrupt+0xf/0x20 [ 9836.111986] [ 9836.111989] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9836.111995] do_raw_spin_lock+0x177/0x190 [ 9836.112000] _raw_spin_lock+0x21/0x30 [ 9836.112006] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9836.112394] ? do_raw_spin_unlock+0x75/0x190 [ 9836.112399] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9836.112656] ? do_wait_intr+0xf0/0xf0 [ 9836.112665] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9836.112849] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9836.113191] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9836.113419] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9836.113541] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9836.113581] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9836.113610] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9836.113637] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9836.113842] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9836.114081] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9836.114119] lmv_intent_open+0x338/0x1150 [lmv] [ 9836.114155] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9836.114315] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9836.114417] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9836.114502] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9836.114647] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9836.114816] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9836.114911] ? do_raw_spin_unlock+0x75/0x190 [ 9836.114919] ? _raw_spin_unlock+0x12/0x30 [ 9836.114926] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9836.114967] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9836.114995] ll_atomic_open+0x36a/0xd70 [lustre] [ 9836.115109] lookup_open+0x356/0xb40 [ 9836.115119] do_last+0x313/0xfc0 [ 9836.115125] ? nd_jump_root+0xe5/0x160 [ 9836.115130] ? path_init+0x437/0x520 [ 9836.115136] path_openat+0xf7/0x500 [ 9836.115141] ? do_raw_spin_unlock+0x75/0x190 [ 9836.115146] do_filp_open+0x99/0x140 [ 9836.115153] ? getname_flags+0x6e/0x330 [ 9836.115159] ? __check_object_size+0xff/0x256 [ 9836.115166] ? do_raw_spin_unlock+0x75/0x190 [ 9836.115171] ? _raw_spin_unlock+0x12/0x30 [ 9836.115176] do_sys_openat2+0x2b4/0x410 [ 9836.115181] do_sys_open+0x73/0xa0 [ 9836.115185] __x64_sys_openat+0x24/0x30 [ 9836.115189] do_syscall_64+0xc1/0x440 [ 9836.115203] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9836.115283] RIP: 0033:0x7ff54af45332 [ 9836.115291] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9836.115301] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9836.115306] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 9836.115309] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 9836.115311] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 9836.115313] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 9836.115316] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 9836.673450] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9836.677989] after_reply+0xe2c/0x1780 [ptlrpc] [ 9836.685690] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9836.694937] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9836.697204] ? do_wait_intr+0xf0/0xf0 [ 9836.699045] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9836.703150] kthread+0x1d1/0x200 [ 9836.706103] ? set_kthread_struct+0x70/0x70 [ 9836.713259] ret_from_fork+0x1f/0x30 [ 9839.071025] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7880s! [ 9839.078495] Showing busy workqueues and worker pools: [ 9839.088797] workqueue events: flags=0x0 [ 9839.089928] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9839.097723] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 9839.102670] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 9839.105082] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 9839.116132] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9839.120745] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 9839.129404] workqueue events_long: flags=0x0 [ 9839.132563] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9839.135961] pending: lu_objects_flush [obdclass] [ 9839.138661] workqueue events_unbound: flags=0x2 [ 9839.140775] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 9839.143561] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9839.156153] workqueue events_power_efficient: flags=0x80 [ 9839.165016] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9839.170163] pending: neigh_periodic_work [ 9839.176175] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9839.184650] pending: do_cache_clean, check_lifetime [ 9839.190774] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9839.195191] pending: neigh_periodic_work [ 9839.198463] workqueue writeback: flags=0x4a [ 9839.202887] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9839.213110] pending: wb_workfn, wb_workfn [ 9839.218684] workqueue xprtiod: flags=0xa [ 9839.226481] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9839.231875] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 9839.236439] workqueue ipv6_addrconf: flags=0x40008 [ 9839.238670] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9839.245052] pending: addrconf_verify_work [ 9839.247684] workqueue knbd0-recv: flags=0x1a [ 9839.249238] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9839.251480] in-flight: 657:recv_work [ 9839.253082] workqueue ptlrpc_pinger: flags=0x2 [ 9839.254492] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9839.257422] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 9839.262270] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7880s workers=4 idle: 20831 12642 28859 [ 9839.267264] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9829s workers=2 idle: 659 [ 9848.059491] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 9848.062299] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9848.082291] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9848.085838] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9848.089342] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9848.091998] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9848.094062] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9848.100566] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9848.104117] RAX: 0000000000006b84 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 9848.106784] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 9848.109516] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 9848.112586] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 9848.115529] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 9848.118256] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 9848.122361] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9848.124430] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 9848.127166] Call Trace: [ 9848.128260] [ 9848.129239] ? show_regs.cold.9+0x22/0x2f [ 9848.130490] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9848.132013] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9848.133535] ? watchdog+0x40/0x40 [ 9848.134536] ? hrtimer_interrupt+0x128/0x2c0 [ 9848.136239] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9848.137879] ? apic_timer_interrupt+0xf/0x20 [ 9848.139520] [ 9848.140369] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9848.142052] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9848.143918] do_raw_spin_lock+0x177/0x190 [ 9848.145379] _raw_spin_lock+0x21/0x30 [ 9848.146491] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9848.148621] process_one_work+0x2c8/0x700 [ 9848.150145] worker_thread+0x296/0x6e0 [ 9848.151453] ? rescuer_thread+0x570/0x570 [ 9848.152694] kthread+0x1d1/0x200 [ 9848.153736] ? set_kthread_struct+0x70/0x70 [ 9848.155517] ret_from_fork+0x1f/0x30 [ 9852.066968] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 9852.080270] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9852.150586] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9852.167293] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9852.177903] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9852.190599] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9852.207195] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9852.214931] RAX: 000000000000118c RBX: ffff96848811cae0 RCX: 0000000000000001 [ 9852.230030] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 9852.244516] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9852.251383] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 9852.272549] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 9852.283186] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 9852.291274] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9852.298218] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 9852.305217] Call Trace: [ 9852.306398] [ 9852.309344] ? show_regs.cold.9+0x22/0x2f [ 9852.313762] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9852.318984] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9852.323382] ? watchdog+0x40/0x40 [ 9852.327396] ? hrtimer_interrupt+0x128/0x2c0 [ 9852.329262] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9852.334892] ? apic_timer_interrupt+0xf/0x20 [ 9852.340600] [ 9852.348253] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9852.352158] do_raw_spin_lock+0x177/0x190 [ 9852.359401] _raw_spin_lock+0x21/0x30 [ 9852.360687] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9852.365634] ? do_raw_spin_unlock+0x75/0x190 [ 9852.369436] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9852.373605] ? do_wait_intr+0xf0/0xf0 [ 9852.377929] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9852.383021] mdc_close+0x53b/0xd70 [mdc] [ 9852.389299] lmv_close+0x18d/0x490 [lmv] [ 9852.396293] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 9852.399757] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9852.405344] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9852.411648] ll_md_close+0x274/0x800 [lustre] [ 9852.413097] ll_file_release+0x128/0xc50 [lustre] [ 9852.418123] __fput+0x10d/0x440 [ 9852.421509] ____fput+0x12/0x20 [ 9852.426275] task_work_run+0xc6/0x110 [ 9852.430525] exit_to_usermode_loop+0x1dd/0x1f0 [ 9852.433144] do_syscall_64+0x3d6/0x440 [ 9852.435051] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9852.436549] RIP: 0033:0x7ff54af45cd8 [ 9852.438935] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9852.458156] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9852.467433] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 9852.474884] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 9852.481032] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 9852.489810] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 9852.496795] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9864.060241] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 9864.060262] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 9864.071945] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9864.072221] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9864.072231] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9864.072234] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9864.072252] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9864.072256] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9864.072261] RAX: 000000000000452a RBX: ffff96848811cae0 RCX: 0000000000000001 [ 9864.072263] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 9864.072266] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9864.072268] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 9864.072270] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 9864.072274] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 9864.072277] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9864.072279] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 9864.072288] Call Trace: [ 9864.072294] [ 9864.072301] ? show_regs.cold.9+0x22/0x2f [ 9864.072325] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9864.072332] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9864.072337] ? watchdog+0x40/0x40 [ 9864.072341] ? hrtimer_interrupt+0x128/0x2c0 [ 9864.072345] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9864.072353] ? apic_timer_interrupt+0xf/0x20 [ 9864.072358] [ 9864.072361] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9864.072368] do_raw_spin_lock+0x177/0x190 [ 9864.072373] _raw_spin_lock+0x21/0x30 [ 9864.072379] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9864.074147] ? do_raw_spin_unlock+0x75/0x190 [ 9864.074156] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9864.074359] ? do_wait_intr+0xf0/0xf0 [ 9864.074370] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9864.074577] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9864.083579] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9864.084757] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9864.084767] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9864.084770] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9864.084787] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9864.084791] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9864.084795] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 9864.084797] RDX: 0000000000004966 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 9864.084799] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 9864.084802] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 9864.084804] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9864.084808] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 9864.084811] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9864.084814] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 9864.084822] Call Trace: [ 9864.084828] [ 9864.086398] ? show_regs.cold.9+0x22/0x2f [ 9864.086426] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9864.086436] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9864.086441] ? watchdog+0x40/0x40 [ 9864.086445] ? hrtimer_interrupt+0x128/0x2c0 [ 9864.086449] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9864.086458] ? apic_timer_interrupt+0xf/0x20 [ 9864.086462] [ 9864.086466] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9864.086472] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9864.086477] do_raw_spin_lock+0x177/0x190 [ 9864.086483] _raw_spin_lock+0x21/0x30 [ 9864.086490] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9864.089178] obd_mod_free+0x46/0x140 [ptlrpc] [ 9864.146091] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9864.148460] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9864.149027] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9864.149075] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9864.149109] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9864.149153] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9864.180729] mdc_commit_open+0x91/0xb0 [mdc] [ 9864.180796] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9864.194246] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9864.197187] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9864.197237] lmv_intent_open+0x338/0x1150 [lmv] [ 9864.204468] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9864.218169] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9864.218388] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9864.218518] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9864.218550] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9864.218758] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9864.218858] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9864.218948] ? do_raw_spin_unlock+0x75/0x190 [ 9864.238525] after_reply+0xe2c/0x1780 [ptlrpc] [ 9864.238850] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9864.242064] ? _raw_spin_unlock+0x12/0x30 [ 9864.242083] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9864.242108] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9864.242279] ll_atomic_open+0x36a/0xd70 [lustre] [ 9864.242408] lookup_open+0x356/0xb40 [ 9864.242418] do_last+0x313/0xfc0 [ 9864.242423] ? nd_jump_root+0xe5/0x160 [ 9864.242428] ? path_init+0x437/0x520 [ 9864.242433] path_openat+0xf7/0x500 [ 9864.242437] ? do_raw_spin_unlock+0x75/0x190 [ 9864.242443] do_filp_open+0x99/0x140 [ 9864.242450] ? getname_flags+0x6e/0x330 [ 9864.242454] ? __check_object_size+0xff/0x256 [ 9864.242462] ? do_raw_spin_unlock+0x75/0x190 [ 9864.242465] ? _raw_spin_unlock+0x12/0x30 [ 9864.242470] do_sys_openat2+0x2b4/0x410 [ 9864.242475] do_sys_open+0x73/0xa0 [ 9864.242479] __x64_sys_openat+0x24/0x30 [ 9864.242483] do_syscall_64+0xc1/0x440 [ 9864.242489] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9864.242495] RIP: 0033:0x7ff54af45332 [ 9864.242503] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9864.242507] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9864.242513] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 9864.242515] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 9864.242518] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 9864.242519] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 9864.242522] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 9864.800428] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9864.807422] ? do_wait_intr+0xf0/0xf0 [ 9864.809494] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9864.817187] kthread+0x1d1/0x200 [ 9864.821175] ? set_kthread_struct+0x70/0x70 [ 9864.823102] ret_from_fork+0x1f/0x30 [ 9869.279506] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7910s! [ 9869.287019] Showing busy workqueues and worker pools: [ 9869.289732] workqueue events: flags=0x0 [ 9869.292138] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9869.297843] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 9869.307677] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 9869.314774] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 9869.326740] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9869.339024] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 9869.347189] workqueue events_long: flags=0x0 [ 9869.356315] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9869.373277] pending: lu_objects_flush [obdclass] [ 9869.382372] workqueue events_unbound: flags=0x2 [ 9869.389235] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 9869.401952] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9869.413334] workqueue events_power_efficient: flags=0x80 [ 9869.422936] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9869.438247] pending: neigh_periodic_work [ 9869.441798] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9869.453437] pending: do_cache_clean, check_lifetime [ 9869.462714] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9869.464603] pending: neigh_periodic_work [ 9869.470673] workqueue writeback: flags=0x4a [ 9869.474962] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9869.478545] pending: wb_workfn, wb_workfn [ 9869.482425] workqueue xprtiod: flags=0xa [ 9869.484453] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9869.489876] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 9869.496930] workqueue ipv6_addrconf: flags=0x40008 [ 9869.500681] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9869.505829] pending: addrconf_verify_work [ 9869.508505] workqueue knbd0-recv: flags=0x1a [ 9869.512472] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9869.518349] in-flight: 657:recv_work [ 9869.522042] workqueue ptlrpc_pinger: flags=0x2 [ 9869.526217] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9869.532236] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 9869.538098] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7910s workers=4 idle: 20831 12642 28859 [ 9869.546471] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9859s workers=2 idle: 659 [ 9876.060486] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 9876.067599] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9876.101240] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9876.108620] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9876.114310] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9876.119294] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9876.122537] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9876.130810] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9876.134436] RAX: 0000000000000e8a RBX: ffff96848811cae0 RCX: 0000000000000001 [ 9876.137865] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 9876.141295] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 9876.144731] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 9876.148078] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 9876.151517] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 9876.155290] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9876.157963] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 9876.162132] Call Trace: [ 9876.163364] [ 9876.164209] ? show_regs.cold.9+0x22/0x2f [ 9876.166235] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9876.169338] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9876.171351] ? watchdog+0x40/0x40 [ 9876.173013] ? hrtimer_interrupt+0x128/0x2c0 [ 9876.174997] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9876.177123] ? apic_timer_interrupt+0xf/0x20 [ 9876.179073] [ 9876.180060] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9876.182398] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9876.185125] do_raw_spin_lock+0x177/0x190 [ 9876.187257] _raw_spin_lock+0x21/0x30 [ 9876.188982] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9876.191791] process_one_work+0x2c8/0x700 [ 9876.193515] worker_thread+0x296/0x6e0 [ 9876.195307] ? rescuer_thread+0x570/0x570 [ 9876.197421] kthread+0x1d1/0x200 [ 9876.198966] ? set_kthread_struct+0x70/0x70 [ 9876.200797] ret_from_fork+0x1f/0x30 [ 9880.058014] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 9880.061085] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9880.079276] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9880.084449] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9880.088908] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9880.091386] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9880.099165] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9880.101259] RAX: 000000000000291e RBX: ffff96848811cae0 RCX: 0000000000000001 [ 9880.104189] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 9880.110379] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9880.114318] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 9880.117630] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 9880.121144] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 9880.125399] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9880.128363] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 9880.132081] Call Trace: [ 9880.133295] [ 9880.134108] ? show_regs.cold.9+0x22/0x2f [ 9880.136029] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9880.138323] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9880.140914] ? watchdog+0x40/0x40 [ 9880.141895] ? hrtimer_interrupt+0x128/0x2c0 [ 9880.143041] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9880.144543] ? apic_timer_interrupt+0xf/0x20 [ 9880.146344] [ 9880.147346] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9880.150007] do_raw_spin_lock+0x177/0x190 [ 9880.151959] _raw_spin_lock+0x21/0x30 [ 9880.153830] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9880.156385] ? do_raw_spin_unlock+0x75/0x190 [ 9880.159134] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9880.162151] ? do_wait_intr+0xf0/0xf0 [ 9880.164220] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9880.167114] mdc_close+0x53b/0xd70 [mdc] [ 9880.169051] lmv_close+0x18d/0x490 [lmv] [ 9880.172763] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 9880.180877] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9880.183939] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9880.190334] ll_md_close+0x274/0x800 [lustre] [ 9880.195789] ll_file_release+0x128/0xc50 [lustre] [ 9880.199900] __fput+0x10d/0x440 [ 9880.203391] ____fput+0x12/0x20 [ 9880.207245] task_work_run+0xc6/0x110 [ 9880.211052] exit_to_usermode_loop+0x1dd/0x1f0 [ 9880.215162] do_syscall_64+0x3d6/0x440 [ 9880.220214] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9880.222295] RIP: 0033:0x7ff54af45cd8 [ 9880.223982] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9880.232759] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9880.238991] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 9880.251318] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 9880.262620] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 9880.269413] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 9880.275686] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9892.062331] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 9892.068720] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9892.099760] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9892.107286] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9892.117488] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9892.120976] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9892.132742] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 9892.141836] RAX: 0000000000006d85 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 9892.147899] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 9892.152760] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9892.160603] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 9892.166954] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 9892.176858] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 9892.185883] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9892.189064] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 9892.193772] Call Trace: [ 9892.196417] [ 9892.199533] ? show_regs.cold.9+0x22/0x2f [ 9892.201647] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9892.203824] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9892.210865] ? watchdog+0x40/0x40 [ 9892.211613] ? hrtimer_interrupt+0x128/0x2c0 [ 9892.212372] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9892.213333] ? apic_timer_interrupt+0xf/0x20 [ 9892.223648] [ 9892.227170] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9892.231941] do_raw_spin_lock+0x177/0x190 [ 9892.235180] _raw_spin_lock+0x21/0x30 [ 9892.237751] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9892.241151] ? do_raw_spin_unlock+0x75/0x190 [ 9892.243871] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9892.247849] ? do_wait_intr+0xf0/0xf0 [ 9892.248843] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9892.253804] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9892.259723] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9892.264477] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9892.267075] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9892.270754] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9892.272395] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9892.273608] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9892.276406] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9892.281797] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9892.282925] lmv_intent_open+0x338/0x1150 [lmv] [ 9892.283835] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9892.286726] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9892.288927] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9892.290373] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9892.294863] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9892.297642] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9892.300154] ? do_raw_spin_unlock+0x75/0x190 [ 9892.306570] ? _raw_spin_unlock+0x12/0x30 [ 9892.310595] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9892.311899] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9892.313801] ll_atomic_open+0x36a/0xd70 [lustre] [ 9892.317188] lookup_open+0x356/0xb40 [ 9892.317932] do_last+0x313/0xfc0 [ 9892.320625] ? nd_jump_root+0xe5/0x160 [ 9892.321671] ? path_init+0x437/0x520 [ 9892.322587] path_openat+0xf7/0x500 [ 9892.323796] ? do_raw_spin_unlock+0x75/0x190 [ 9892.325192] do_filp_open+0x99/0x140 [ 9892.325924] ? getname_flags+0x6e/0x330 [ 9892.326915] ? __check_object_size+0xff/0x256 [ 9892.328632] ? do_raw_spin_unlock+0x75/0x190 [ 9892.329679] ? _raw_spin_unlock+0x12/0x30 [ 9892.330697] do_sys_openat2+0x2b4/0x410 [ 9892.331854] do_sys_open+0x73/0xa0 [ 9892.332835] __x64_sys_openat+0x24/0x30 [ 9892.334424] do_syscall_64+0xc1/0x440 [ 9892.335533] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9892.337635] RIP: 0033:0x7ff54af45332 [ 9892.338941] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9892.347194] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9892.350596] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 9892.352335] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 9892.354684] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 9892.356886] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 9892.358551] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 9896.056881] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [ 9896.064097] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9896.116344] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9896.129231] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9896.132955] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9896.148010] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9896.179155] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9896.185933] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 9896.196633] RDX: 0000000000005545 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 9896.200581] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 9896.210185] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 9896.219225] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9896.232307] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 9896.243404] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9896.245295] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 9896.254931] Call Trace: [ 9896.255709] [ 9896.256388] ? show_regs.cold.9+0x22/0x2f [ 9896.265696] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9896.269098] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9896.276422] ? watchdog+0x40/0x40 [ 9896.281113] ? hrtimer_interrupt+0x128/0x2c0 [ 9896.284755] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9896.293359] ? apic_timer_interrupt+0xf/0x20 [ 9896.294980] [ 9896.299073] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9896.311535] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9896.320154] do_raw_spin_lock+0x177/0x190 [ 9896.322124] _raw_spin_lock+0x21/0x30 [ 9896.329819] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9896.333911] obd_mod_free+0x46/0x140 [ptlrpc] [ 9896.342112] mdc_commit_open+0x91/0xb0 [mdc] [ 9896.348205] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9896.350164] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9896.358991] after_reply+0xe2c/0x1780 [ptlrpc] [ 9896.362113] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9896.366452] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9896.372237] ? do_wait_intr+0xf0/0xf0 [ 9896.374476] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9896.380325] kthread+0x1d1/0x200 [ 9896.381353] ? set_kthread_struct+0x70/0x70 [ 9896.383913] ret_from_fork+0x1f/0x30 [ 9900.003628] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7941s! [ 9900.009783] Showing busy workqueues and worker pools: [ 9900.020641] workqueue events: flags=0x0 [ 9900.027606] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9900.045000] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 9900.065887] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 9900.079421] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 9900.106037] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9900.111778] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 9900.119463] workqueue events_long: flags=0x0 [ 9900.121965] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9900.125581] pending: lu_objects_flush [obdclass] [ 9900.140338] workqueue events_unbound: flags=0x2 [ 9900.146514] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 9900.149562] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9900.157055] workqueue events_power_efficient: flags=0x80 [ 9900.160095] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9900.168985] pending: neigh_periodic_work [ 9900.172384] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9900.178678] pending: do_cache_clean, check_lifetime [ 9900.184806] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9900.196522] pending: neigh_periodic_work [ 9900.201129] workqueue writeback: flags=0x4a [ 9900.207669] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9900.212384] pending: wb_workfn, wb_workfn [ 9900.220741] workqueue xprtiod: flags=0xa [ 9900.225602] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9900.230141] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 9900.234623] workqueue ipv6_addrconf: flags=0x40008 [ 9900.237084] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9900.240246] pending: addrconf_verify_work [ 9900.246024] workqueue knbd0-recv: flags=0x1a [ 9900.250636] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9900.260111] in-flight: 657:recv_work [ 9900.265580] workqueue ptlrpc_pinger: flags=0x2 [ 9900.273188] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9900.287175] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 9900.293402] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7941s workers=4 idle: 20831 12642 28859 [ 9900.300984] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9890s workers=2 idle: 659 [ 9904.062941] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 9904.074368] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9904.131837] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9904.140448] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9904.160878] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9904.165587] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9904.172225] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9904.191661] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9904.206896] RAX: 000000000000376b RBX: ffff96848811cae0 RCX: 0000000000000001 [ 9904.212155] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 9904.222935] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 9904.234639] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 9904.240245] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 9904.246610] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 9904.250204] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9904.253616] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 9904.262296] Call Trace: [ 9904.269063] [ 9904.270153] ? show_regs.cold.9+0x22/0x2f [ 9904.274138] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9904.280480] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9904.283941] ? watchdog+0x40/0x40 [ 9904.288097] ? hrtimer_interrupt+0x128/0x2c0 [ 9904.292564] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9904.294303] ? apic_timer_interrupt+0xf/0x20 [ 9904.298313] [ 9904.299385] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9904.306767] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9904.311958] do_raw_spin_lock+0x177/0x190 [ 9904.314908] _raw_spin_lock+0x21/0x30 [ 9904.321764] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9904.325754] process_one_work+0x2c8/0x700 [ 9904.330117] worker_thread+0x296/0x6e0 [ 9904.336224] ? rescuer_thread+0x570/0x570 [ 9904.339996] kthread+0x1d1/0x200 [ 9904.341141] ? set_kthread_struct+0x70/0x70 [ 9904.342588] ret_from_fork+0x1f/0x30 [ 9908.062810] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [ 9908.073396] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9908.111832] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9908.122931] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9908.132929] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9908.137501] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9908.156651] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9908.165527] RAX: 0000000000003e17 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 9908.170394] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 9908.177122] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9908.183569] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 9908.188933] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 9908.196710] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 9908.201969] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9908.204998] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 9908.214018] Call Trace: [ 9908.216156] [ 9908.217063] ? show_regs.cold.9+0x22/0x2f [ 9908.225408] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9908.231978] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9908.236875] ? watchdog+0x40/0x40 [ 9908.245866] ? hrtimer_interrupt+0x128/0x2c0 [ 9908.248912] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9908.254879] ? apic_timer_interrupt+0xf/0x20 [ 9908.257778] [ 9908.260064] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9908.263506] do_raw_spin_lock+0x177/0x190 [ 9908.266590] _raw_spin_lock+0x21/0x30 [ 9908.269972] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9908.273728] ? do_raw_spin_unlock+0x75/0x190 [ 9908.275807] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9908.278388] ? do_wait_intr+0xf0/0xf0 [ 9908.280760] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9908.289997] mdc_close+0x53b/0xd70 [mdc] [ 9908.295921] lmv_close+0x18d/0x490 [lmv] [ 9908.302088] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 9908.307856] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9908.317977] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9908.323985] ll_md_close+0x274/0x800 [lustre] [ 9908.327257] ll_file_release+0x128/0xc50 [lustre] [ 9908.330786] __fput+0x10d/0x440 [ 9908.334820] ____fput+0x12/0x20 [ 9908.336775] task_work_run+0xc6/0x110 [ 9908.339235] exit_to_usermode_loop+0x1dd/0x1f0 [ 9908.346922] do_syscall_64+0x3d6/0x440 [ 9908.350242] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9908.356073] RIP: 0033:0x7ff54af45cd8 [ 9908.370119] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9908.390044] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9908.397064] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 9908.410417] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 9908.418665] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 9908.430029] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 9908.434079] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9920.060706] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 9920.070525] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9920.115354] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9920.129674] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9920.138066] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9920.145557] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9920.178601] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 9920.193942] RAX: 0000000000007d07 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 9920.201011] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 9920.209576] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9920.217377] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 9920.223691] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 9920.232601] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 9920.238936] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9920.245494] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 9920.251141] Call Trace: [ 9920.253183] [ 9920.253811] ? show_regs.cold.9+0x22/0x2f [ 9920.255295] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9920.256807] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9920.258283] ? watchdog+0x40/0x40 [ 9920.259772] ? hrtimer_interrupt+0x128/0x2c0 [ 9920.261346] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9920.262547] ? apic_timer_interrupt+0xf/0x20 [ 9920.263769] [ 9920.264336] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9920.265960] do_raw_spin_lock+0x177/0x190 [ 9920.267788] _raw_spin_lock+0x21/0x30 [ 9920.268788] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9920.270483] ? do_raw_spin_unlock+0x75/0x190 [ 9920.271580] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9920.273085] ? do_wait_intr+0xf0/0xf0 [ 9920.274703] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9920.276534] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9920.278025] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9920.279845] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9920.281445] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9920.282782] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9920.284066] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9920.285351] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9920.287000] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9920.288750] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9920.290566] lmv_intent_open+0x338/0x1150 [lmv] [ 9920.291803] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9920.293355] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9920.295412] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9920.296787] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9920.298355] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9920.300505] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9920.302577] ? do_raw_spin_unlock+0x75/0x190 [ 9920.304503] ? _raw_spin_unlock+0x12/0x30 [ 9920.306335] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9920.308661] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9920.311449] ll_atomic_open+0x36a/0xd70 [lustre] [ 9920.314144] lookup_open+0x356/0xb40 [ 9920.315784] do_last+0x313/0xfc0 [ 9920.317326] ? nd_jump_root+0xe5/0x160 [ 9920.319229] ? path_init+0x437/0x520 [ 9920.320861] path_openat+0xf7/0x500 [ 9920.322304] ? do_raw_spin_unlock+0x75/0x190 [ 9920.324169] do_filp_open+0x99/0x140 [ 9920.326002] ? getname_flags+0x6e/0x330 [ 9920.327860] ? __check_object_size+0xff/0x256 [ 9920.329821] ? do_raw_spin_unlock+0x75/0x190 [ 9920.331661] ? _raw_spin_unlock+0x12/0x30 [ 9920.333889] do_sys_openat2+0x2b4/0x410 [ 9920.335793] do_sys_open+0x73/0xa0 [ 9920.337497] __x64_sys_openat+0x24/0x30 [ 9920.339096] do_syscall_64+0xc1/0x440 [ 9920.341273] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9920.343120] RIP: 0033:0x7ff54af45332 [ 9920.344601] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9920.352976] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9920.356279] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 9920.359110] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 9920.362874] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 9920.365979] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 9920.369258] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 9928.055011] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [ 9928.060258] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9928.085483] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9928.089267] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9928.092899] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9928.095240] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9928.102275] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9928.104820] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 9928.108621] RDX: 0000000000004a82 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 9928.111376] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 9928.114270] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 9928.116632] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9928.119177] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 9928.123792] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9928.126787] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 9928.130751] Call Trace: [ 9928.131814] [ 9928.132738] ? show_regs.cold.9+0x22/0x2f [ 9928.134320] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9928.136792] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9928.139684] ? watchdog+0x40/0x40 [ 9928.141107] ? hrtimer_interrupt+0x128/0x2c0 [ 9928.142943] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9928.144909] ? apic_timer_interrupt+0xf/0x20 [ 9928.147328] [ 9928.148163] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9928.150821] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9928.153209] do_raw_spin_lock+0x177/0x190 [ 9928.155578] _raw_spin_lock+0x21/0x30 [ 9928.158077] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9928.164937] obd_mod_free+0x46/0x140 [ptlrpc] [ 9928.167659] mdc_commit_open+0x91/0xb0 [mdc] [ 9928.169851] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9928.172370] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9928.175727] after_reply+0xe2c/0x1780 [ptlrpc] [ 9928.180787] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9928.184611] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9928.188307] ? do_wait_intr+0xf0/0xf0 [ 9928.196515] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9928.202109] kthread+0x1d1/0x200 [ 9928.204435] ? set_kthread_struct+0x70/0x70 [ 9928.206545] ret_from_fork+0x1f/0x30 [ 9930.719083] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7971s! [ 9930.724234] Showing busy workqueues and worker pools: [ 9930.727911] workqueue events: flags=0x0 [ 9930.730832] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9930.735391] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 9930.740796] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 9930.746436] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 9930.754933] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9930.759176] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 9930.762810] workqueue events_long: flags=0x0 [ 9930.765904] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9930.772018] pending: lu_objects_flush [obdclass] [ 9930.779494] workqueue events_unbound: flags=0x2 [ 9930.782433] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 9930.786474] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9930.793012] workqueue events_power_efficient: flags=0x80 [ 9930.796091] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9930.799581] pending: neigh_periodic_work [ 9930.802232] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9930.805420] pending: do_cache_clean, check_lifetime [ 9930.808428] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9930.814373] pending: neigh_periodic_work [ 9930.815926] workqueue writeback: flags=0x4a [ 9930.822657] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9930.825333] pending: wb_workfn, wb_workfn [ 9930.827921] workqueue xprtiod: flags=0xa [ 9930.829834] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9930.835353] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 9930.841828] workqueue ipv6_addrconf: flags=0x40008 [ 9930.844300] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9930.848148] pending: addrconf_verify_work [ 9930.852276] workqueue knbd0-recv: flags=0x1a [ 9930.856465] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9930.871932] in-flight: 657:recv_work [ 9930.873452] workqueue ptlrpc_pinger: flags=0x2 [ 9930.876297] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9930.880404] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 9930.888351] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7972s workers=4 idle: 20831 12642 28859 [ 9930.894279] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9921s workers=2 idle: 659 [ 9932.062203] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 9932.074463] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9932.143924] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9932.161515] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9932.178337] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9932.188208] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9932.192177] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9932.216757] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9932.221184] RAX: 000000000000568f RBX: ffff96848811cae0 RCX: 0000000000000001 [ 9932.231894] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 9932.240268] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 9932.249741] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 9932.256701] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 9932.262554] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 9932.272878] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9932.282027] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 9932.288343] Call Trace: [ 9932.293313] [ 9932.295365] ? show_regs.cold.9+0x22/0x2f [ 9932.301417] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9932.306635] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9932.311999] ? watchdog+0x40/0x40 [ 9932.315357] ? hrtimer_interrupt+0x128/0x2c0 [ 9932.319676] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9932.324152] ? apic_timer_interrupt+0xf/0x20 [ 9932.335361] [ 9932.338189] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9932.341264] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9932.345097] do_raw_spin_lock+0x177/0x190 [ 9932.346857] _raw_spin_lock+0x21/0x30 [ 9932.349371] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9932.354520] process_one_work+0x2c8/0x700 [ 9932.358253] worker_thread+0x296/0x6e0 [ 9932.360456] ? rescuer_thread+0x570/0x570 [ 9932.362700] kthread+0x1d1/0x200 [ 9932.366276] ? set_kthread_struct+0x70/0x70 [ 9932.375298] ret_from_fork+0x1f/0x30 [ 9934.071114] rcu: INFO: rcu_sched self-detected stall on CPU [ 9934.074127] rcu: 0-...!: (4157896 ticks this GP) idle=98a/1/0x4000000000000002 softirq=125154/125154 fqs=959 [ 9934.080263] (t=7980231 jiffies g=253585 q=5452) [ 9934.086630] rcu: rcu_sched kthread starved for 7975191 jiffies! g253585 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [ 9934.096695] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 9934.104639] rcu: RCU grace-period kthread stack dump: [ 9934.113294] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 9934.120255] Call Trace: [ 9934.121539] __schedule+0x351/0xcb0 [ 9934.123289] schedule+0xc0/0x180 [ 9934.128521] schedule_timeout+0xb4/0x190 [ 9934.134496] ? do_raw_spin_unlock+0x75/0x190 [ 9934.139188] ? __next_timer_interrupt+0x160/0x160 [ 9934.144405] ? prepare_to_swait_event+0xf3/0x140 [ 9934.147224] rcu_gp_kthread+0x8dd/0x15b0 [ 9934.148674] ? force_qs_rnp+0x2c0/0x2c0 [ 9934.154008] kthread+0x1d1/0x200 [ 9934.160812] ? set_kthread_struct+0x70/0x70 [ 9934.168162] ret_from_fork+0x1f/0x30 [ 9934.171726] rcu: Stack dump where RCU GP kthread last ran: [ 9934.175605] Sending NMI from CPU 0 to CPUs 1: [ 9934.182865] NMI backtrace for cpu 1 [ 9934.182872] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9934.182875] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9934.182876] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9934.182878] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9934.182880] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 [ 9934.182883] RAX: 00000000000072f7 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 9934.182884] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 9934.182886] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9934.182887] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 9934.182888] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 9934.182889] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 9934.182891] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9934.182892] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 9934.182894] Call Trace: [ 9934.182895] [ 9934.182896] ? show_regs.cold.9+0x22/0x2f [ 9934.182897] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 9934.182898] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 9934.182899] ? nmi_handle+0xbd/0x210 [ 9934.182900] ? default_do_nmi+0x4b/0x1b0 [ 9934.182901] ? do_nmi+0x280/0x380 [ 9934.182902] ? end_repeat_nmi+0x16/0x1f [ 9934.182903] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9934.182904] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9934.182906] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9934.182907] [ 9934.182908] do_raw_spin_lock+0x177/0x190 [ 9934.182909] _raw_spin_lock+0x21/0x30 [ 9934.182910] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9934.182911] ? do_raw_spin_unlock+0x75/0x190 [ 9934.182912] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9934.182913] ? do_wait_intr+0xf0/0xf0 [ 9934.182914] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9934.182916] mdc_close+0x53b/0xd70 [mdc] [ 9934.182916] lmv_close+0x18d/0x490 [lmv] [ 9934.182917] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 9934.182918] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9934.182920] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9934.182921] ll_md_close+0x274/0x800 [lustre] [ 9934.182922] ll_file_release+0x128/0xc50 [lustre] [ 9934.182923] __fput+0x10d/0x440 [ 9934.182924] ____fput+0x12/0x20 [ 9934.182925] task_work_run+0xc6/0x110 [ 9934.182926] exit_to_usermode_loop+0x1dd/0x1f0 [ 9934.182927] do_syscall_64+0x3d6/0x440 [ 9934.182928] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9934.182929] RIP: 0033:0x7ff54af45cd8 [ 9934.182930] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9934.182932] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9934.182934] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 9934.182936] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 9934.182937] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 9934.182939] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 9934.182940] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9934.184348] NMI backtrace for cpu 0 [ 9934.671089] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9934.685156] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9934.695597] Call Trace: [ 9934.697473] [ 9934.698651] ? dump_stack+0xbb/0x10e [ 9934.700875] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 9934.703511] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 9934.706814] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 9934.710627] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 9934.713633] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 9934.723414] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 9934.729810] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 9934.735940] ? update_process_times+0x6d/0xc0 [ 9934.741681] ? tick_sched_handle.isra.18+0x27/0x90 [ 9934.747710] ? tick_sched_timer+0x53/0xb0 [ 9934.750136] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9934.755489] ? tick_sched_handle.isra.18+0x90/0x90 [ 9934.761759] ? hrtimer_interrupt+0x128/0x2c0 [ 9934.764334] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9934.769739] ? apic_timer_interrupt+0xf/0x20 [ 9934.778557] [ 9934.779761] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9934.787735] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9934.795449] ? do_raw_spin_lock+0x177/0x190 [ 9934.797655] ? _raw_spin_lock+0x21/0x30 [ 9934.803257] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9934.806344] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 9934.816102] ? mdc_commit_open+0x91/0xb0 [mdc] [ 9934.820357] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9934.826161] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9934.829462] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 9934.833694] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9934.837042] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9934.841732] ? do_wait_intr+0xf0/0xf0 [ 9934.844665] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9934.849205] ? kthread+0x1d1/0x200 [ 9934.851368] ? set_kthread_struct+0x70/0x70 [ 9934.853790] ? ret_from_fork+0x1f/0x30 [ 9948.060026] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 9948.063297] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9948.085866] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9948.090915] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9948.095389] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9948.097701] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9948.106280] RSP: 0000:ffffaf9fc15234f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9948.110091] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 9948.113120] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 9948.115388] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9948.118682] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 9948.122599] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 9948.125497] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 9948.127962] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9948.130730] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 9948.134199] Call Trace: [ 9948.135627] [ 9948.136687] ? show_regs.cold.9+0x22/0x2f [ 9948.138576] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9948.140783] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9948.143325] ? watchdog+0x40/0x40 [ 9948.144986] ? hrtimer_interrupt+0x128/0x2c0 [ 9948.146996] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9948.149120] ? apic_timer_interrupt+0xf/0x20 [ 9948.151173] [ 9948.152402] ? native_safe_halt+0x2e/0x50 [ 9948.154140] kvm_wait+0xcd/0xe0 [ 9948.155180] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9948.158090] do_raw_spin_lock+0x177/0x190 [ 9948.160547] _raw_spin_lock+0x21/0x30 [ 9948.162325] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9948.165553] ? do_raw_spin_unlock+0x75/0x190 [ 9948.167588] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9948.170289] ? do_wait_intr+0xf0/0xf0 [ 9948.172083] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9948.175035] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9948.177993] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9948.181580] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9948.184190] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9948.186942] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9948.189301] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9948.191707] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9948.194829] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9948.198001] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9948.200611] lmv_intent_open+0x338/0x1150 [lmv] [ 9948.202860] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9948.205499] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9948.207108] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9948.208482] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9948.210699] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9948.213082] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9948.215414] ? do_raw_spin_unlock+0x75/0x190 [ 9948.217565] ? _raw_spin_unlock+0x12/0x30 [ 9948.219119] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9948.221972] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9948.224164] ll_atomic_open+0x36a/0xd70 [lustre] [ 9948.226307] lookup_open+0x356/0xb40 [ 9948.228002] do_last+0x313/0xfc0 [ 9948.229669] ? nd_jump_root+0xe5/0x160 [ 9948.231825] ? path_init+0x437/0x520 [ 9948.233504] path_openat+0xf7/0x500 [ 9948.235120] ? do_raw_spin_unlock+0x75/0x190 [ 9948.237109] do_filp_open+0x99/0x140 [ 9948.238852] ? getname_flags+0x6e/0x330 [ 9948.240602] ? __check_object_size+0xff/0x256 [ 9948.242548] ? do_raw_spin_unlock+0x75/0x190 [ 9948.244290] ? _raw_spin_unlock+0x12/0x30 [ 9948.245993] do_sys_openat2+0x2b4/0x410 [ 9948.247704] do_sys_open+0x73/0xa0 [ 9948.249185] __x64_sys_openat+0x24/0x30 [ 9948.251014] do_syscall_64+0xc1/0x440 [ 9948.252672] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9948.254886] RIP: 0033:0x7ff54af45332 [ 9948.256508] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9948.265057] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9948.268569] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 9948.271869] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 9948.275886] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 9948.279202] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 9948.283017] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 9960.058620] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [ 9960.059073] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [ 9960.059768] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 9960.059783] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9960.059878] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9960.059884] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9960.059892] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9960.060292] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9960.060308] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9960.060312] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9960.060317] RAX: 0000000000006096 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 9960.060319] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 9960.060321] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 9960.060324] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 9960.060327] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 9960.060331] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 9960.060334] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9960.060337] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 9960.060347] Call Trace: [ 9960.060356] [ 9960.060363] ? show_regs.cold.9+0x22/0x2f [ 9960.060375] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9960.060382] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9960.060387] ? watchdog+0x40/0x40 [ 9960.060391] ? hrtimer_interrupt+0x128/0x2c0 [ 9960.060396] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9960.060407] ? apic_timer_interrupt+0xf/0x20 [ 9960.060412] [ 9960.060415] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9960.060421] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9960.060426] do_raw_spin_lock+0x177/0x190 [ 9960.060430] _raw_spin_lock+0x21/0x30 [ 9960.060435] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9960.060643] process_one_work+0x2c8/0x700 [ 9960.060653] worker_thread+0x296/0x6e0 [ 9960.060658] ? rescuer_thread+0x570/0x570 [ 9960.060661] kthread+0x1d1/0x200 [ 9960.060667] ? set_kthread_struct+0x70/0x70 [ 9960.060672] ret_from_fork+0x1f/0x30 [ 9960.065228] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 [ 9960.071525] Modules linked in: [ 9960.074805] dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9960.074859] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9960.074866] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9960.074869] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9960.074883] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9960.074888] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9960.074893] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 9960.074895] RDX: 0000000000002f12 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 9960.074897] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 9960.074899] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 9960.074901] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9960.074904] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 9960.074907] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9960.074909] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 9960.074918] Call Trace: [ 9960.074924] [ 9960.074929] ? show_regs.cold.9+0x22/0x2f [ 9960.074940] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9960.074947] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9960.074952] ? watchdog+0x40/0x40 [ 9960.074956] ? hrtimer_interrupt+0x128/0x2c0 [ 9960.074961] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9960.074971] ? apic_timer_interrupt+0xf/0x20 [ 9960.074976] [ 9960.074979] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9960.074984] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9960.074989] do_raw_spin_lock+0x177/0x190 [ 9960.074994] _raw_spin_lock+0x21/0x30 [ 9960.075001] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9960.075385] obd_mod_free+0x46/0x140 [ptlrpc] [ 9960.075584] mdc_commit_open+0x91/0xb0 [mdc] [ 9960.075618] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9960.075933] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9960.076128] after_reply+0xe2c/0x1780 [ptlrpc] [ 9960.076495] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9960.076896] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9960.077114] ? do_wait_intr+0xf0/0xf0 [ 9960.077124] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9960.077359] kthread+0x1d1/0x200 [ 9960.077365] ? set_kthread_struct+0x70/0x70 [ 9960.077370] ret_from_fork+0x1f/0x30 [ 9960.305623] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9960.319103] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9960.322876] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9960.325874] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9960.327434] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9960.332226] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9960.333999] RAX: 0000000000007f86 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 9960.335870] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 9960.337893] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9960.339918] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 9960.341780] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 9960.344063] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 9960.346252] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9960.347821] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 9960.349471] Call Trace: [ 9960.350167] [ 9960.350734] ? show_regs.cold.9+0x22/0x2f [ 9960.351717] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9960.353010] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9960.353981] ? watchdog+0x40/0x40 [ 9960.354818] ? hrtimer_interrupt+0x128/0x2c0 [ 9960.355816] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9960.356966] ? apic_timer_interrupt+0xf/0x20 [ 9960.358075] [ 9960.358902] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9960.360960] do_raw_spin_lock+0x177/0x190 [ 9960.362338] _raw_spin_lock+0x21/0x30 [ 9960.363603] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9960.365401] ? do_raw_spin_unlock+0x75/0x190 [ 9960.368429] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9960.371344] ? do_wait_intr+0xf0/0xf0 [ 9960.372965] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9960.375630] mdc_close+0x53b/0xd70 [mdc] [ 9960.376856] lmv_close+0x18d/0x490 [lmv] [ 9960.377947] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 9960.380388] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9960.383288] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9960.385571] ll_md_close+0x274/0x800 [lustre] [ 9960.387841] ll_file_release+0x128/0xc50 [lustre] [ 9960.390403] __fput+0x10d/0x440 [ 9960.391779] ____fput+0x12/0x20 [ 9960.393222] task_work_run+0xc6/0x110 [ 9960.394819] exit_to_usermode_loop+0x1dd/0x1f0 [ 9960.395978] do_syscall_64+0x3d6/0x440 [ 9960.397178] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9960.398314] RIP: 0033:0x7ff54af45cd8 [ 9960.399535] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9960.406932] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9960.409403] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 9960.411303] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 9960.414270] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 9960.416818] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 9960.418501] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9960.930129] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8002s! [ 9960.935815] Showing busy workqueues and worker pools: [ 9960.942279] workqueue events: flags=0x0 [ 9960.943761] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9960.951126] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 9960.962328] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 9960.976425] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 9960.985597] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9960.996743] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 9961.008402] workqueue events_long: flags=0x0 [ 9961.014028] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9961.022481] pending: lu_objects_flush [obdclass] [ 9961.031385] workqueue events_unbound: flags=0x2 [ 9961.038393] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 9961.044979] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9961.067347] workqueue events_power_efficient: flags=0x80 [ 9961.073452] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9961.080221] pending: neigh_periodic_work [ 9961.086439] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9961.091942] pending: do_cache_clean, check_lifetime [ 9961.098881] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9961.103768] pending: neigh_periodic_work [ 9961.106892] workqueue writeback: flags=0x4a [ 9961.111124] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9961.115816] pending: wb_workfn, wb_workfn [ 9961.119826] workqueue xprtiod: flags=0xa [ 9961.122820] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9961.128050] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 9961.136624] workqueue ipv6_addrconf: flags=0x40008 [ 9961.138638] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9961.151396] pending: addrconf_verify_work [ 9961.159458] workqueue knbd0-recv: flags=0x1a [ 9961.162688] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9961.166924] in-flight: 657:recv_work [ 9961.171748] workqueue ptlrpc_pinger: flags=0x2 [ 9961.178174] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9961.184240] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 9961.191794] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8002s workers=4 idle: 20831 12642 28859 [ 9961.200869] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9951s workers=2 idle: 659 [ 9976.063936] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [ 9976.081889] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9976.159198] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9976.179525] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9976.204949] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9976.214147] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9976.259721] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9976.271992] RAX: 0000000000007f79 RBX: ffff96848811cae0 RCX: 0000000000000001 [ 9976.277151] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [ 9976.289167] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9976.293743] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [ 9976.309763] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [ 9976.321254] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [ 9976.333669] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9976.341457] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [ 9976.348140] Call Trace: [ 9976.349397] [ 9976.350595] ? show_regs.cold.9+0x22/0x2f [ 9976.352244] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9976.353948] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9976.357581] ? watchdog+0x40/0x40 [ 9976.359171] ? hrtimer_interrupt+0x128/0x2c0 [ 9976.366849] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9976.371323] ? apic_timer_interrupt+0xf/0x20 [ 9976.373721] [ 9976.378233] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9976.386854] do_raw_spin_lock+0x177/0x190 [ 9976.389624] _raw_spin_lock+0x21/0x30 [ 9976.391025] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9976.394819] ? do_raw_spin_unlock+0x75/0x190 [ 9976.397342] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9976.415245] ? do_wait_intr+0xf0/0xf0 [ 9976.418767] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9976.422825] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9976.426368] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9976.434837] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9976.456002] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9976.459804] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9976.469720] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9976.483366] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9976.498129] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9976.507247] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9976.517382] lmv_intent_open+0x338/0x1150 [lmv] [ 9976.528620] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9976.536107] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9976.548804] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9976.554713] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9976.560910] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9976.564686] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9976.570724] ? do_raw_spin_unlock+0x75/0x190 [ 9976.579062] ? _raw_spin_unlock+0x12/0x30 [ 9976.590125] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9976.596839] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9976.600426] ll_atomic_open+0x36a/0xd70 [lustre] [ 9976.610843] lookup_open+0x356/0xb40 [ 9976.616261] do_last+0x313/0xfc0 [ 9976.622961] ? nd_jump_root+0xe5/0x160 [ 9976.629036] ? path_init+0x437/0x520 [ 9976.634617] path_openat+0xf7/0x500 [ 9976.639825] ? do_raw_spin_unlock+0x75/0x190 [ 9976.646055] do_filp_open+0x99/0x140 [ 9976.652178] ? getname_flags+0x6e/0x330 [ 9976.658849] ? __check_object_size+0xff/0x256 [ 9976.672641] ? do_raw_spin_unlock+0x75/0x190 [ 9976.679353] ? _raw_spin_unlock+0x12/0x30 [ 9976.688111] do_sys_openat2+0x2b4/0x410 [ 9976.692269] do_sys_open+0x73/0xa0 [ 9976.699497] __x64_sys_openat+0x24/0x30 [ 9976.705551] do_syscall_64+0xc1/0x440 [ 9976.709250] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9976.712850] RIP: 0033:0x7ff54af45332 [ 9976.716825] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9976.732103] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9976.737210] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [ 9976.741144] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [ 9976.744756] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [ 9976.748922] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [ 9976.752935] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [ 9988.055259] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [ 9988.058656] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9988.061590] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [ 9988.061689] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 9988.061704] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9988.061785] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9988.061792] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9988.061800] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9988.062157] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9988.062177] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9988.062182] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9988.062187] RAX: 000000000000581b RBX: ffff96848811cae0 RCX: 0000000000000001 [ 9988.062190] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [ 9988.062193] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 9988.062195] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [ 9988.062198] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [ 9988.062203] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [ 9988.062210] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9988.062212] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [ 9988.062222] Call Trace: [ 9988.062230] [ 9988.062241] ? show_regs.cold.9+0x22/0x2f [ 9988.062254] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9988.062264] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9988.062270] ? watchdog+0x40/0x40 [ 9988.062275] ? hrtimer_interrupt+0x128/0x2c0 [ 9988.062279] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9988.062289] ? apic_timer_interrupt+0xf/0x20 [ 9988.062294] [ 9988.062297] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9988.062306] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9988.062312] do_raw_spin_lock+0x177/0x190 [ 9988.062317] _raw_spin_lock+0x21/0x30 [ 9988.062322] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9988.063411] process_one_work+0x2c8/0x700 [ 9988.063422] worker_thread+0x296/0x6e0 [ 9988.063973] ? rescuer_thread+0x570/0x570 [ 9988.063978] kthread+0x1d1/0x200 [ 9988.063984] ? set_kthread_struct+0x70/0x70 [ 9988.063989] ret_from_fork+0x1f/0x30 [ 9988.110760] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9988.110775] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9988.110778] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9988.110792] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9988.110797] RSP: 0000:ffffaf9fc1b97bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9988.110803] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [ 9988.110806] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff96848811cae0 [ 9988.110808] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [ 9988.110810] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [ 9988.110813] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9988.110816] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [ 9988.110819] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9988.110822] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [ 9988.110830] Call Trace: [ 9988.110836] [ 9988.110842] ? show_regs.cold.9+0x22/0x2f [ 9988.111190] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9988.111196] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9988.111201] ? watchdog+0x40/0x40 [ 9988.111205] ? hrtimer_interrupt+0x128/0x2c0 [ 9988.111210] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9988.111217] ? apic_timer_interrupt+0xf/0x20 [ 9988.111222] [ 9988.111226] ? native_safe_halt+0x2e/0x50 [ 9988.111231] kvm_wait+0xcd/0xe0 [ 9988.111239] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9988.111246] do_raw_spin_lock+0x177/0x190 [ 9988.111251] _raw_spin_lock+0x21/0x30 [ 9988.111258] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9988.118234] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9988.118320] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9988.118327] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9988.118330] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9988.118344] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9988.118349] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9988.118353] RAX: 000000000000750e RBX: ffff96848811cae0 RCX: 0000000000000001 [ 9988.118355] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [ 9988.118357] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9988.118359] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [ 9988.118466] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [ 9988.118637] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [ 9988.118643] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9988.118645] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [ 9988.118653] Call Trace: [ 9988.118659] [ 9988.118664] ? show_regs.cold.9+0x22/0x2f [ 9988.118675] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9988.118682] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9988.118686] ? watchdog+0x40/0x40 [ 9988.118690] ? hrtimer_interrupt+0x128/0x2c0 [ 9988.118694] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9988.118704] ? apic_timer_interrupt+0xf/0x20 [ 9988.118708] [ 9988.118712] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9988.118719] do_raw_spin_lock+0x177/0x190 [ 9988.118724] _raw_spin_lock+0x21/0x30 [ 9988.118730] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9988.124737] obd_mod_free+0x46/0x140 [ptlrpc] [ 9988.125081] mdc_commit_open+0x91/0xb0 [mdc] [ 9988.125120] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9988.125322] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9988.125826] after_reply+0xe2c/0x1780 [ptlrpc] [ 9988.126251] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9988.126802] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9988.127333] ? do_wait_intr+0xf0/0xf0 [ 9988.127351] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9988.128387] kthread+0x1d1/0x200 [ 9988.128394] ? set_kthread_struct+0x70/0x70 [ 9988.128398] ret_from_fork+0x1f/0x30 [ 9988.711252] ? do_raw_spin_unlock+0x75/0x190 [ 9988.712616] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9988.714421] ? do_wait_intr+0xf0/0xf0 [ 9988.715637] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9988.718028] mdc_close+0x53b/0xd70 [mdc] [ 9988.719757] lmv_close+0x18d/0x490 [lmv] [ 9988.721518] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 9988.724242] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9988.726247] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9988.728309] ll_md_close+0x274/0x800 [lustre] [ 9988.730189] ll_file_release+0x128/0xc50 [lustre] [ 9988.732302] __fput+0x10d/0x440 [ 9988.733631] ____fput+0x12/0x20 [ 9988.735097] task_work_run+0xc6/0x110 [ 9988.736652] exit_to_usermode_loop+0x1dd/0x1f0 [ 9988.738551] do_syscall_64+0x3d6/0x440 [ 9988.739772] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9988.741755] RIP: 0033:0x7ff54af45cd8 [ 9988.743571] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9988.752667] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9988.755175] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [ 9988.760433] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [ 9988.764288] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [ 9988.767874] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [ 9988.771738] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9991.651851] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8032s! [ 9991.662639] Showing busy workqueues and worker pools: [ 9991.666688] workqueue events: flags=0x0 [ 9991.670152] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9991.675353] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [ 9991.686888] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 9991.696818] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [ 9991.705912] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9991.712577] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [ 9991.720906] workqueue events_long: flags=0x0 [ 9991.727029] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9991.734317] pending: lu_objects_flush [obdclass] [ 9991.744645] workqueue events_unbound: flags=0x2 [ 9991.752087] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 9991.759771] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9991.778050] workqueue events_power_efficient: flags=0x80 [ 9991.785247] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9991.800874] pending: neigh_periodic_work [ 9991.803660] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9991.812136] pending: do_cache_clean, check_lifetime [ 9991.816973] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9991.822138] pending: neigh_periodic_work [ 9991.826447] workqueue writeback: flags=0x4a [ 9991.833670] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9991.838598] pending: wb_workfn, wb_workfn [ 9991.846439] workqueue xprtiod: flags=0xa [ 9991.852763] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9991.855797] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [ 9991.865733] workqueue ipv6_addrconf: flags=0x40008 [ 9991.870687] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9991.875711] pending: addrconf_verify_work [ 9991.878715] workqueue knbd0-recv: flags=0x1a [ 9991.881236] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9991.884824] in-flight: 657:recv_work [ 9991.887163] workqueue ptlrpc_pinger: flags=0x2 [ 9991.891270] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9991.893644] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 9991.900095] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8033s workers=4 idle: 20831 12642 28859 [ 9991.905077] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9982s workers=2 idle: 659 [10004.060956] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [10004.071363] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10004.119639] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10004.130693] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10004.139980] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10004.144694] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10004.178071] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10004.182145] RAX: 00000000000065a1 RBX: ffff96848811cae0 RCX: 0000000000000001 [10004.197923] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [10004.203354] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [10004.211585] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [10004.223871] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [10004.231441] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [10004.244834] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10004.247632] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [10004.253715] Call Trace: [10004.257990] [10004.261144] ? show_regs.cold.9+0x22/0x2f [10004.267762] ? watchdog_timer_fn.cold.6+0x62/0x10a [10004.273600] ? __hrtimer_run_queues+0x1c2/0x4d0 [10004.281034] ? watchdog+0x40/0x40 [10004.286510] ? hrtimer_interrupt+0x128/0x2c0 [10004.292577] ? smp_apic_timer_interrupt+0xa6/0x270 [10004.298531] ? apic_timer_interrupt+0xf/0x20 [10004.310397] [10004.313545] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10004.328400] do_raw_spin_lock+0x177/0x190 [10004.333790] _raw_spin_lock+0x21/0x30 [10004.339473] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10004.353445] ? do_raw_spin_unlock+0x75/0x190 [10004.362605] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10004.364200] ? do_wait_intr+0xf0/0xf0 [10004.373285] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10004.375043] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10004.392466] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10004.395993] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10004.401541] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10004.408578] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10004.415612] mdc_intent_lock+0x353/0x7e0 [mdc] [10004.421288] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10004.426001] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10004.429576] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10004.432084] lmv_intent_open+0x338/0x1150 [lmv] [10004.434002] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10004.436561] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10004.439627] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10004.441727] ll_intent_lock+0xc5/0xaa0 [lustre] [10004.444563] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10004.447092] ll_lookup_it+0xe0e/0x2270 [lustre] [10004.450179] ? do_raw_spin_unlock+0x75/0x190 [10004.453783] ? _raw_spin_unlock+0x12/0x30 [10004.458075] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10004.461657] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10004.465481] ll_atomic_open+0x36a/0xd70 [lustre] [10004.469924] lookup_open+0x356/0xb40 [10004.472492] do_last+0x313/0xfc0 [10004.475580] ? nd_jump_root+0xe5/0x160 [10004.478118] ? path_init+0x437/0x520 [10004.481128] path_openat+0xf7/0x500 [10004.484647] ? do_raw_spin_unlock+0x75/0x190 [10004.492589] do_filp_open+0x99/0x140 [10004.496003] ? getname_flags+0x6e/0x330 [10004.498567] ? __check_object_size+0xff/0x256 [10004.502889] ? do_raw_spin_unlock+0x75/0x190 [10004.508714] ? _raw_spin_unlock+0x12/0x30 [10004.515477] do_sys_openat2+0x2b4/0x410 [10004.522577] do_sys_open+0x73/0xa0 [10004.525688] __x64_sys_openat+0x24/0x30 [10004.527697] do_syscall_64+0xc1/0x440 [10004.530700] entry_SYSCALL_64_after_hwframe+0x49/0xae [10004.535544] RIP: 0033:0x7ff54af45332 [10004.539909] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10004.555050] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10004.565833] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [10004.572627] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [10004.584793] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [10004.595820] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [10004.608796] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [10016.056645] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [10016.059009] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [10016.062183] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10016.062408] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10016.062418] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10016.062421] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10016.062443] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10016.062448] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10016.062453] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [10016.062456] RDX: 0000000000003a7c RSI: 0000000000000003 RDI: ffff96848811cae0 [10016.062459] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [10016.062462] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [10016.062464] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10016.062467] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [10016.062470] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10016.062473] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [10016.062482] Call Trace: [10016.062489] [10016.062495] ? show_regs.cold.9+0x22/0x2f [10016.062505] ? watchdog_timer_fn.cold.6+0x62/0x10a [10016.062513] ? __hrtimer_run_queues+0x1c2/0x4d0 [10016.062518] ? watchdog+0x40/0x40 [10016.062522] ? hrtimer_interrupt+0x128/0x2c0 [10016.062526] ? smp_apic_timer_interrupt+0xa6/0x270 [10016.064562] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [10016.064575] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10016.064650] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10016.064656] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10016.064659] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10016.064671] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10016.064676] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10016.064680] RAX: 000000000000791b RBX: ffff96848811cae0 RCX: 0000000000000001 [10016.064682] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [10016.064685] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [10016.064687] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [10016.064689] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [10016.064692] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [10016.064695] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10016.064698] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [10016.064706] Call Trace: [10016.064709] [10016.064714] ? show_regs.cold.9+0x22/0x2f [10016.064723] ? watchdog_timer_fn.cold.6+0x62/0x10a [10016.064730] ? __hrtimer_run_queues+0x1c2/0x4d0 [10016.064734] ? watchdog+0x40/0x40 [10016.064738] ? hrtimer_interrupt+0x128/0x2c0 [10016.064741] ? smp_apic_timer_interrupt+0xa6/0x270 [10016.064748] ? apic_timer_interrupt+0xf/0x20 [10016.064752] [10016.064755] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10016.064761] do_raw_spin_lock+0x177/0x190 [10016.064766] _raw_spin_lock+0x21/0x30 [10016.064771] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10016.068313] ? do_raw_spin_unlock+0x75/0x190 [10016.068322] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10016.069337] ? do_wait_intr+0xf0/0xf0 [10016.069346] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10016.070448] mdc_close+0x53b/0xd70 [mdc] [10016.071399] lmv_close+0x18d/0x490 [lmv] [10016.071434] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [10016.071707] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10016.071849] ll_md_real_close+0xa5/0x2a0 [lustre] [10016.071933] ll_md_close+0x274/0x800 [lustre] [10016.071998] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver [10016.072026] ll_file_release+0x128/0xc50 [lustre] [10016.072075] intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul [10016.072117] __fput+0x10d/0x440 [10016.072128] ____fput+0x12/0x20 [10016.072133] task_work_run+0xc6/0x110 [10016.072141] exit_to_usermode_loop+0x1dd/0x1f0 [10016.072148] do_syscall_64+0x3d6/0x440 [10016.072153] entry_SYSCALL_64_after_hwframe+0x49/0xae [10016.072162] RIP: 0033:0x7ff54af45cd8 [10016.072170] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10016.072175] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10016.072180] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [10016.072182] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [10016.072185] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [10016.072187] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [10016.072189] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10016.072091] crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10016.072299] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10016.072307] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10016.072315] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10016.072715] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10016.072803] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10016.072808] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10016.072812] RAX: 0000000000007c84 RBX: ffff96848811cae0 RCX: 0000000000000001 [10016.072815] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [10016.072818] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [10016.072820] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [10016.072822] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [10016.072827] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [10016.072830] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10016.072832] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [10016.072841] Call Trace: [10016.072847] [10016.072853] ? show_regs.cold.9+0x22/0x2f [10016.072865] ? watchdog_timer_fn.cold.6+0x62/0x10a [10016.072872] ? __hrtimer_run_queues+0x1c2/0x4d0 [10016.072878] ? watchdog+0x40/0x40 [10016.072883] ? hrtimer_interrupt+0x128/0x2c0 [10016.072887] ? smp_apic_timer_interrupt+0xa6/0x270 [10016.072897] ? apic_timer_interrupt+0xf/0x20 [10016.072901] [10016.072904] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10016.072910] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10016.072915] do_raw_spin_lock+0x177/0x190 [10016.072920] _raw_spin_lock+0x21/0x30 [10016.072925] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10016.118958] ? apic_timer_interrupt+0xf/0x20 [10016.119406] [10016.119411] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10016.119419] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10016.119423] do_raw_spin_lock+0x177/0x190 [10016.119430] _raw_spin_lock+0x21/0x30 [10016.119439] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10016.119743] obd_mod_free+0x46/0x140 [ptlrpc] [10016.119952] mdc_commit_open+0x91/0xb0 [mdc] [10016.120067] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10016.120397] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10016.120594] after_reply+0xe2c/0x1780 [ptlrpc] [10016.120803] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10016.130639] process_one_work+0x2c8/0x700 [10016.130861] worker_thread+0x296/0x6e0 [10016.130867] ? rescuer_thread+0x570/0x570 [10016.130871] kthread+0x1d1/0x200 [10016.130875] ? set_kthread_struct+0x70/0x70 [10016.130879] ret_from_fork+0x1f/0x30 [10016.541023] ptlrpcd+0xa82/0xe40 [ptlrpc] [10016.547190] ? do_wait_intr+0xf0/0xf0 [10016.548655] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10016.551666] kthread+0x1d1/0x200 [10016.552853] ? set_kthread_struct+0x70/0x70 [10016.558813] ret_from_fork+0x1f/0x30 [10022.372433] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8063s! [10022.378774] Showing busy workqueues and worker pools: [10022.381941] workqueue events: flags=0x0 [10022.384596] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10022.388533] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [10022.393876] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [10022.398580] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [10022.405765] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10022.410711] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [10022.415456] workqueue events_long: flags=0x0 [10022.418182] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10022.422936] pending: lu_objects_flush [obdclass] [10022.426593] workqueue events_unbound: flags=0x2 [10022.431475] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [10022.437632] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10022.445227] workqueue events_power_efficient: flags=0x80 [10022.447962] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10022.456546] pending: neigh_periodic_work [10022.461243] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10022.469608] pending: do_cache_clean, check_lifetime [10022.473421] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10022.480342] pending: neigh_periodic_work [10022.484837] workqueue writeback: flags=0x4a [10022.490519] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10022.497464] pending: wb_workfn, wb_workfn [10022.500079] workqueue xprtiod: flags=0xa [10022.502012] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10022.512380] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [10022.518598] workqueue ipv6_addrconf: flags=0x40008 [10022.526137] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10022.530682] pending: addrconf_verify_work [10022.534870] workqueue knbd0-recv: flags=0x1a [10022.537249] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10022.542472] in-flight: 657:recv_work [10022.548632] workqueue ptlrpc_pinger: flags=0x2 [10022.552524] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10022.556372] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [10022.562255] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8063s workers=4 idle: 20831 12642 28859 [10022.567718] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10012s workers=2 idle: 659 [10032.060127] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [10032.063880] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10032.089161] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10032.094877] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10032.099360] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10032.101658] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10032.110452] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10032.114406] RAX: 0000000000007e77 RBX: ffff96848811cae0 RCX: 0000000000000001 [10032.118469] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [10032.123079] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [10032.127033] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [10032.131770] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [10032.135257] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [10032.139902] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10032.144490] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [10032.148686] Call Trace: [10032.149807] [10032.151473] ? show_regs.cold.9+0x22/0x2f [10032.153719] ? watchdog_timer_fn.cold.6+0x62/0x10a [10032.156865] ? __hrtimer_run_queues+0x1c2/0x4d0 [10032.159925] ? watchdog+0x40/0x40 [10032.162352] ? hrtimer_interrupt+0x128/0x2c0 [10032.165356] ? smp_apic_timer_interrupt+0xa6/0x270 [10032.169808] ? apic_timer_interrupt+0xf/0x20 [10032.173361] [10032.174590] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10032.178150] do_raw_spin_lock+0x177/0x190 [10032.180292] _raw_spin_lock+0x21/0x30 [10032.183829] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10032.189544] ? do_raw_spin_unlock+0x75/0x190 [10032.193287] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10032.203419] ? do_wait_intr+0xf0/0xf0 [10032.205733] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10032.211464] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10032.215135] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10032.221319] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10032.223328] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10032.226825] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10032.231468] mdc_intent_lock+0x353/0x7e0 [mdc] [10032.235916] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10032.239380] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10032.242545] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10032.245159] lmv_intent_open+0x338/0x1150 [lmv] [10032.247688] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10032.250472] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10032.252854] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10032.255295] ll_intent_lock+0xc5/0xaa0 [lustre] [10032.257756] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10032.260455] ll_lookup_it+0xe0e/0x2270 [lustre] [10032.262756] ? do_raw_spin_unlock+0x75/0x190 [10032.264848] ? _raw_spin_unlock+0x12/0x30 [10032.266792] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10032.269102] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10032.271567] ll_atomic_open+0x36a/0xd70 [lustre] [10032.273749] lookup_open+0x356/0xb40 [10032.275451] do_last+0x313/0xfc0 [10032.276892] ? nd_jump_root+0xe5/0x160 [10032.278838] ? path_init+0x437/0x520 [10032.281051] path_openat+0xf7/0x500 [10032.283274] ? do_raw_spin_unlock+0x75/0x190 [10032.285659] do_filp_open+0x99/0x140 [10032.287509] ? getname_flags+0x6e/0x330 [10032.289204] ? __check_object_size+0xff/0x256 [10032.291815] ? do_raw_spin_unlock+0x75/0x190 [10032.293828] ? _raw_spin_unlock+0x12/0x30 [10032.296544] do_sys_openat2+0x2b4/0x410 [10032.298566] do_sys_open+0x73/0xa0 [10032.300209] __x64_sys_openat+0x24/0x30 [10032.302345] do_syscall_64+0xc1/0x440 [10032.304234] entry_SYSCALL_64_after_hwframe+0x49/0xae [10032.306675] RIP: 0033:0x7ff54af45332 [10032.308821] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10032.318018] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10032.320385] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [10032.323765] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [10032.326806] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [10032.330363] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [10032.335577] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [10044.059274] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [10044.060955] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [10044.072743] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10044.072822] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10044.072829] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10044.072831] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10044.072845] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10044.072848] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10044.072853] RAX: 00000000000010e7 RBX: ffff96848811cae0 RCX: 0000000000000001 [10044.072855] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [10044.072857] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [10044.072859] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [10044.072861] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [10044.072864] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [10044.072867] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10044.072870] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [10044.072878] Call Trace: [10044.072884] [10044.072889] ? show_regs.cold.9+0x22/0x2f [10044.072899] ? watchdog_timer_fn.cold.6+0x62/0x10a [10044.072905] ? __hrtimer_run_queues+0x1c2/0x4d0 [10044.072910] ? watchdog+0x40/0x40 [10044.072914] ? hrtimer_interrupt+0x128/0x2c0 [10044.072918] ? smp_apic_timer_interrupt+0xa6/0x270 [10044.072927] ? apic_timer_interrupt+0xf/0x20 [10044.072931] [10044.072945] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10044.072952] do_raw_spin_lock+0x177/0x190 [10044.072957] _raw_spin_lock+0x21/0x30 [10044.072963] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10044.073266] ? do_raw_spin_unlock+0x75/0x190 [10044.073273] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10044.074129] ? do_wait_intr+0xf0/0xf0 [10044.074137] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10044.074593] mdc_close+0x53b/0xd70 [mdc] [10044.074635] lmv_close+0x18d/0x490 [lmv] [10044.074665] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [10044.075390] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10044.076215] ll_md_real_close+0xa5/0x2a0 [lustre] [10044.076642] ll_md_close+0x274/0x800 [lustre] [10044.076729] ll_file_release+0x128/0xc50 [lustre] [10044.076818] __fput+0x10d/0x440 [10044.077119] ____fput+0x12/0x20 [10044.077124] task_work_run+0xc6/0x110 [10044.077132] exit_to_usermode_loop+0x1dd/0x1f0 [10044.077140] do_syscall_64+0x3d6/0x440 [10044.077145] entry_SYSCALL_64_after_hwframe+0x49/0xae [10044.077155] RIP: 0033:0x7ff54af45cd8 [10044.077163] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10044.077168] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10044.077173] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [10044.077176] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [10044.077178] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [10044.077181] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [10044.077183] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10044.298861] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10044.368949] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10044.382498] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10044.399798] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10044.407535] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10044.420607] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10044.438832] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10044.444555] RAX: 0000000000006c87 RBX: ffff96848811cae0 RCX: 0000000000000001 [10044.453169] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [10044.457110] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [10044.462164] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [10044.465705] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [10044.473201] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [10044.480118] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10044.486764] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [10044.496580] Call Trace: [10044.500737] [10044.501507] ? show_regs.cold.9+0x22/0x2f [10044.507373] ? watchdog_timer_fn.cold.6+0x62/0x10a [10044.511619] ? __hrtimer_run_queues+0x1c2/0x4d0 [10044.515641] ? watchdog+0x40/0x40 [10044.519671] ? hrtimer_interrupt+0x128/0x2c0 [10044.523532] ? smp_apic_timer_interrupt+0xa6/0x270 [10044.528389] ? apic_timer_interrupt+0xf/0x20 [10044.532556] [10044.533796] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10044.535843] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10044.540344] do_raw_spin_lock+0x177/0x190 [10044.542123] _raw_spin_lock+0x21/0x30 [10044.547685] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10044.556129] process_one_work+0x2c8/0x700 [10044.558960] worker_thread+0x296/0x6e0 [10044.560446] ? rescuer_thread+0x570/0x570 [10044.565497] kthread+0x1d1/0x200 [10044.567982] ? set_kthread_struct+0x70/0x70 [10044.571891] ret_from_fork+0x1f/0x30 [10048.055283] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [10048.064682] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10048.088921] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10048.093769] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10048.098348] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10048.100832] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10048.109553] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10048.113581] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [10048.117305] RDX: 0000000000002a4d RSI: 0000000000000003 RDI: ffff96848811cae0 [10048.120837] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [10048.124369] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [10048.127899] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10048.130675] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [10048.136573] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10048.141602] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [10048.151838] Call Trace: [10048.153635] [10048.154991] ? show_regs.cold.9+0x22/0x2f [10048.157669] ? watchdog_timer_fn.cold.6+0x62/0x10a [10048.160901] ? __hrtimer_run_queues+0x1c2/0x4d0 [10048.163664] ? watchdog+0x40/0x40 [10048.166340] ? hrtimer_interrupt+0x128/0x2c0 [10048.174113] ? smp_apic_timer_interrupt+0xa6/0x270 [10048.185884] ? apic_timer_interrupt+0xf/0x20 [10048.193605] [10048.196606] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10048.203356] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10048.211587] do_raw_spin_lock+0x177/0x190 [10048.217473] _raw_spin_lock+0x21/0x30 [10048.219939] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10048.223118] obd_mod_free+0x46/0x140 [ptlrpc] [10048.226622] mdc_commit_open+0x91/0xb0 [mdc] [10048.230647] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10048.235137] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10048.247104] after_reply+0xe2c/0x1780 [ptlrpc] [10048.250608] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10048.254369] ptlrpcd+0xa82/0xe40 [ptlrpc] [10048.256291] ? do_wait_intr+0xf0/0xf0 [10048.257454] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10048.260091] kthread+0x1d1/0x200 [10048.261719] ? set_kthread_struct+0x70/0x70 [10048.263361] ret_from_fork+0x1f/0x30 [10053.089677] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8094s! [10053.098634] Showing busy workqueues and worker pools: [10053.104556] workqueue events: flags=0x0 [10053.106367] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10053.111455] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [10053.119240] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [10053.124492] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [10053.133525] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10053.137156] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [10053.144459] workqueue events_long: flags=0x0 [10053.150406] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10053.153634] pending: lu_objects_flush [obdclass] [10053.155622] workqueue events_unbound: flags=0x2 [10053.156843] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [10053.169289] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10053.182415] workqueue events_power_efficient: flags=0x80 [10053.189691] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10053.198692] pending: neigh_periodic_work [10053.207898] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10053.209862] pending: do_cache_clean, check_lifetime [10053.215510] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10053.224605] pending: neigh_periodic_work [10053.230543] workqueue writeback: flags=0x4a [10053.232507] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10053.240900] pending: wb_workfn, wb_workfn [10053.242703] workqueue xprtiod: flags=0xa [10053.244531] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10053.249856] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [10053.254586] workqueue ipv6_addrconf: flags=0x40008 [10053.257212] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10053.260547] pending: addrconf_verify_work [10053.263148] workqueue knbd0-recv: flags=0x1a [10053.265642] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10053.269350] in-flight: 657:recv_work [10053.270769] workqueue ptlrpc_pinger: flags=0x2 [10053.273468] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10053.275920] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [10053.277953] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8094s workers=4 idle: 20831 12642 28859 [10053.280358] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10043s workers=2 idle: 659 [10060.065408] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [10060.068363] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10060.095550] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10060.101219] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10060.110059] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10060.118383] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10060.126785] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10060.130926] RAX: 0000000000007d0d RBX: ffff96848811cae0 RCX: 0000000000000001 [10060.134352] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [10060.137532] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [10060.140905] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [10060.144793] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [10060.148748] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [10060.153040] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10060.156507] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [10060.161779] Call Trace: [10060.164895] [10060.167251] ? show_regs.cold.9+0x22/0x2f [10060.172003] ? watchdog_timer_fn.cold.6+0x62/0x10a [10060.178296] ? __hrtimer_run_queues+0x1c2/0x4d0 [10060.179835] ? watchdog+0x40/0x40 [10060.193974] ? hrtimer_interrupt+0x128/0x2c0 [10060.195464] ? smp_apic_timer_interrupt+0xa6/0x270 [10060.211565] ? apic_timer_interrupt+0xf/0x20 [10060.214653] [10060.219003] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10060.224189] do_raw_spin_lock+0x177/0x190 [10060.230542] _raw_spin_lock+0x21/0x30 [10060.238474] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10060.242565] ? do_raw_spin_unlock+0x75/0x190 [10060.244609] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10060.247080] ? do_wait_intr+0xf0/0xf0 [10060.248293] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10060.251743] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10060.264591] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10060.271173] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10060.276373] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10060.281970] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10060.284947] mdc_intent_lock+0x353/0x7e0 [mdc] [10060.287471] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10060.296635] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10060.302604] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10060.304605] lmv_intent_open+0x338/0x1150 [lmv] [10060.316014] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10060.318374] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10060.322145] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10060.323680] ll_intent_lock+0xc5/0xaa0 [lustre] [10060.336639] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10060.340749] ll_lookup_it+0xe0e/0x2270 [lustre] [10060.344242] ? do_raw_spin_unlock+0x75/0x190 [10060.348326] ? _raw_spin_unlock+0x12/0x30 [10060.351754] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10060.357280] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10060.362385] ll_atomic_open+0x36a/0xd70 [lustre] [10060.367164] lookup_open+0x356/0xb40 [10060.369176] do_last+0x313/0xfc0 [10060.371483] ? nd_jump_root+0xe5/0x160 [10060.374219] ? path_init+0x437/0x520 [10060.376930] path_openat+0xf7/0x500 [10060.378430] ? do_raw_spin_unlock+0x75/0x190 [10060.380249] do_filp_open+0x99/0x140 [10060.384053] ? getname_flags+0x6e/0x330 [10060.386491] ? __check_object_size+0xff/0x256 [10060.388560] ? do_raw_spin_unlock+0x75/0x190 [10060.392933] ? _raw_spin_unlock+0x12/0x30 [10060.398139] do_sys_openat2+0x2b4/0x410 [10060.402616] do_sys_open+0x73/0xa0 [10060.405293] __x64_sys_openat+0x24/0x30 [10060.408810] do_syscall_64+0xc1/0x440 [10060.410595] entry_SYSCALL_64_after_hwframe+0x49/0xae [10060.413156] RIP: 0033:0x7ff54af45332 [10060.415057] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10060.439433] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10060.448610] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [10060.454404] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [10060.458251] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [10060.464561] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [10060.470961] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [10072.060601] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [10072.063834] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [10072.065090] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10072.065190] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10072.065200] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10072.065210] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10072.065598] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10072.065611] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10072.065615] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10072.065620] RAX: 0000000000006ee6 RBX: ffff96848811cae0 RCX: 0000000000000001 [10072.065622] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [10072.065625] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [10072.065627] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [10072.065629] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [10072.065634] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [10072.065636] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10072.065638] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [10072.065647] Call Trace: [10072.065654] [10072.065660] ? show_regs.cold.9+0x22/0x2f [10072.065670] ? watchdog_timer_fn.cold.6+0x62/0x10a [10072.065677] ? __hrtimer_run_queues+0x1c2/0x4d0 [10072.065681] ? watchdog+0x40/0x40 [10072.065685] ? hrtimer_interrupt+0x128/0x2c0 [10072.065689] ? smp_apic_timer_interrupt+0xa6/0x270 [10072.065699] ? apic_timer_interrupt+0xf/0x20 [10072.065704] [10072.065707] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10072.065711] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10072.065716] do_raw_spin_lock+0x177/0x190 [10072.065720] _raw_spin_lock+0x21/0x30 [10072.065871] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10072.066098] process_one_work+0x2c8/0x700 [10072.066113] worker_thread+0x296/0x6e0 [10072.066118] ? rescuer_thread+0x570/0x570 [10072.066200] kthread+0x1d1/0x200 [10072.066207] ? set_kthread_struct+0x70/0x70 [10072.066211] ret_from_fork+0x1f/0x30 [10072.175617] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10072.199054] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10072.204956] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10072.209463] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10072.212774] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10072.222680] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10072.226141] RAX: 0000000000007786 RBX: ffff96848811cae0 RCX: 0000000000000001 [10072.229587] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [10072.233857] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [10072.237869] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [10072.240915] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [10072.244164] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [10072.248003] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10072.250663] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [10072.257711] Call Trace: [10072.259475] [10072.260786] ? show_regs.cold.9+0x22/0x2f [10072.264656] ? watchdog_timer_fn.cold.6+0x62/0x10a [10072.267749] ? __hrtimer_run_queues+0x1c2/0x4d0 [10072.270108] ? watchdog+0x40/0x40 [10072.271931] ? hrtimer_interrupt+0x128/0x2c0 [10072.274180] ? smp_apic_timer_interrupt+0xa6/0x270 [10072.276829] ? apic_timer_interrupt+0xf/0x20 [10072.279712] [10072.280861] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10072.283226] do_raw_spin_lock+0x177/0x190 [10072.285377] _raw_spin_lock+0x21/0x30 [10072.287788] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10072.290477] ? do_raw_spin_unlock+0x75/0x190 [10072.292414] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10072.295650] ? do_wait_intr+0xf0/0xf0 [10072.297735] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10072.301063] mdc_close+0x53b/0xd70 [mdc] [10072.303381] lmv_close+0x18d/0x490 [lmv] [10072.305213] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [10072.308238] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10072.315711] ll_md_real_close+0xa5/0x2a0 [lustre] [10072.318421] ll_md_close+0x274/0x800 [lustre] [10072.321219] ll_file_release+0x128/0xc50 [lustre] [10072.323752] __fput+0x10d/0x440 [10072.325736] ____fput+0x12/0x20 [10072.327802] task_work_run+0xc6/0x110 [10072.329933] exit_to_usermode_loop+0x1dd/0x1f0 [10072.332536] do_syscall_64+0x3d6/0x440 [10072.334148] entry_SYSCALL_64_after_hwframe+0x49/0xae [10072.336607] RIP: 0033:0x7ff54af45cd8 [10072.338384] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10072.344530] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10072.347355] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [10072.349543] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [10072.351711] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [10072.353691] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [10072.355988] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10080.055045] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [10080.062692] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10080.102724] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10080.108816] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10080.113802] RIP: 0010:native_safe_halt+0x2e/0x50 [10080.117956] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10080.128865] RSP: 0000:ffffaf9fc1b97bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10080.135380] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [10080.139697] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff96848811cae0 [10080.145920] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [10080.149793] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [10080.156049] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10080.161167] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [10080.166960] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10080.171801] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [10080.177342] Call Trace: [10080.180498] [10080.182313] ? show_regs.cold.9+0x22/0x2f [10080.187174] ? watchdog_timer_fn.cold.6+0x62/0x10a [10080.194261] ? __hrtimer_run_queues+0x1c2/0x4d0 [10080.199725] ? watchdog+0x40/0x40 [10080.201867] ? hrtimer_interrupt+0x128/0x2c0 [10080.203286] ? smp_apic_timer_interrupt+0xa6/0x270 [10080.210225] ? apic_timer_interrupt+0xf/0x20 [10080.212671] [10080.214309] ? native_safe_halt+0x2e/0x50 [10080.217982] kvm_wait+0xcd/0xe0 [10080.220197] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10080.221981] do_raw_spin_lock+0x177/0x190 [10080.232772] _raw_spin_lock+0x21/0x30 [10080.234058] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10080.241351] obd_mod_free+0x46/0x140 [ptlrpc] [10080.251004] mdc_commit_open+0x91/0xb0 [mdc] [10080.253218] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10080.256281] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10080.261180] after_reply+0xe2c/0x1780 [ptlrpc] [10080.263747] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10080.266803] ptlrpcd+0xa82/0xe40 [ptlrpc] [10080.269306] ? do_wait_intr+0xf0/0xf0 [10080.271125] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10080.274090] kthread+0x1d1/0x200 [10080.276016] ? set_kthread_struct+0x70/0x70 [10080.277964] ret_from_fork+0x1f/0x30 [10083.296848] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8124s! [10083.307348] Showing busy workqueues and worker pools: [10083.313786] workqueue events: flags=0x0 [10083.318777] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10083.331631] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [10083.352734] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [10083.356752] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [10083.367650] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10083.376914] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [10083.384244] workqueue events_long: flags=0x0 [10083.387067] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10083.394243] pending: lu_objects_flush [obdclass] [10083.398424] workqueue events_unbound: flags=0x2 [10083.399958] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [10083.405258] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10083.412587] workqueue events_power_efficient: flags=0x80 [10083.417716] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10083.423941] pending: neigh_periodic_work [10083.426990] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10083.432172] pending: do_cache_clean, check_lifetime [10083.437510] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10083.444572] pending: neigh_periodic_work [10083.447728] workqueue writeback: flags=0x4a [10083.451673] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10083.458674] pending: wb_workfn, wb_workfn [10083.464395] workqueue xprtiod: flags=0xa [10083.466221] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10083.469767] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [10083.476280] workqueue ipv6_addrconf: flags=0x40008 [10083.481703] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10083.487862] pending: addrconf_verify_work [10083.491093] workqueue knbd0-recv: flags=0x1a [10083.493171] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10083.500054] in-flight: 657:recv_work [10083.503852] workqueue ptlrpc_pinger: flags=0x2 [10083.509115] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10083.514975] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [10083.517230] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8124s workers=4 idle: 20831 12642 28859 [10083.524579] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10073s workers=2 idle: 659 [10088.060081] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [10088.073985] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10088.112240] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10088.120082] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10088.131608] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10088.135886] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10088.144348] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10088.149853] RAX: 0000000000007904 RBX: ffff96848811cae0 RCX: 0000000000000001 [10088.153639] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [10088.158688] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [10088.163975] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [10088.169687] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [10088.174332] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [10088.179950] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10088.184634] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [10088.189176] Call Trace: [10088.191729] [10088.193110] ? show_regs.cold.9+0x22/0x2f [10088.195723] ? watchdog_timer_fn.cold.6+0x62/0x10a [10088.198217] ? __hrtimer_run_queues+0x1c2/0x4d0 [10088.201026] ? watchdog+0x40/0x40 [10088.203081] ? hrtimer_interrupt+0x128/0x2c0 [10088.206196] ? smp_apic_timer_interrupt+0xa6/0x270 [10088.209128] ? apic_timer_interrupt+0xf/0x20 [10088.212417] [10088.213303] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10088.218054] do_raw_spin_lock+0x177/0x190 [10088.220560] _raw_spin_lock+0x21/0x30 [10088.222876] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10088.226508] ? do_raw_spin_unlock+0x75/0x190 [10088.228018] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10088.230189] ? do_wait_intr+0xf0/0xf0 [10088.231428] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10088.234068] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10088.236740] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10088.240205] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10088.243028] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10088.245755] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10088.248035] mdc_intent_lock+0x353/0x7e0 [mdc] [10088.250247] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10088.255186] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10088.258357] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10088.261867] lmv_intent_open+0x338/0x1150 [lmv] [10088.264108] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10088.267886] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10088.273362] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10088.280731] ll_intent_lock+0xc5/0xaa0 [lustre] [10088.286515] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10088.289091] ll_lookup_it+0xe0e/0x2270 [lustre] [10088.294405] ? do_raw_spin_unlock+0x75/0x190 [10088.297251] ? _raw_spin_unlock+0x12/0x30 [10088.302210] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10088.305974] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10088.311905] ll_atomic_open+0x36a/0xd70 [lustre] [10088.317854] lookup_open+0x356/0xb40 [10088.320477] do_last+0x313/0xfc0 [10088.321634] ? nd_jump_root+0xe5/0x160 [10088.323700] ? path_init+0x437/0x520 [10088.325697] path_openat+0xf7/0x500 [10088.328889] ? do_raw_spin_unlock+0x75/0x190 [10088.330912] do_filp_open+0x99/0x140 [10088.333783] ? getname_flags+0x6e/0x330 [10088.335873] ? __check_object_size+0xff/0x256 [10088.338839] ? do_raw_spin_unlock+0x75/0x190 [10088.340907] ? _raw_spin_unlock+0x12/0x30 [10088.343642] do_sys_openat2+0x2b4/0x410 [10088.345848] do_sys_open+0x73/0xa0 [10088.349097] __x64_sys_openat+0x24/0x30 [10088.351449] do_syscall_64+0xc1/0x440 [10088.353323] entry_SYSCALL_64_after_hwframe+0x49/0xae [10088.356053] RIP: 0033:0x7ff54af45332 [10088.358208] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10088.368316] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10088.375103] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [10088.384183] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [10088.389066] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [10088.395160] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [10088.407763] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [10100.058032] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [10100.059034] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [10100.059800] Modules linked in: [10100.068542] Modules linked in: [10100.068544] lustre(O) osp(O) ofd(O) [10100.069793] lustre(O) [10100.071125] lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10100.071205] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10100.071212] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10100.071215] RIP: 0010:native_safe_halt+0x2e/0x50 [10100.071273] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10100.071287] RSP: 0018:ffffaf9fc1543ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10100.071292] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [10100.071294] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c20b4154 [10100.071297] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [10100.071299] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [10100.071301] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [10100.071304] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [10100.071307] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10100.071309] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [10100.071316] Call Trace: [10100.071333] [10100.071342] ? show_regs.cold.9+0x22/0x2f [10100.071353] ? watchdog_timer_fn.cold.6+0x62/0x10a [10100.071360] ? __hrtimer_run_queues+0x1c2/0x4d0 [10100.071364] ? watchdog+0x40/0x40 [10100.071436] ? hrtimer_interrupt+0x128/0x2c0 [10100.071441] ? smp_apic_timer_interrupt+0xa6/0x270 [10100.071448] ? apic_timer_interrupt+0xf/0x20 [10100.071461] [10100.071465] ? native_safe_halt+0x2e/0x50 [10100.071469] kvm_wait+0xcd/0xe0 [10100.071477] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10100.071487] do_raw_spin_lock+0x177/0x190 [10100.071491] _raw_spin_lock+0x21/0x30 [10100.071496] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10100.078730] osp(O) [10100.080693] ? do_raw_spin_unlock+0x75/0x190 [10100.118972] ofd(O) [10100.126582] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10100.133040] lod(O) [10100.135095] ? do_wait_intr+0xf0/0xf0 [10100.149888] mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs [10100.159815] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10100.160830] mdc_close+0x53b/0xd70 [mdc] [10100.160886] lmv_close+0x18d/0x490 [lmv] [10100.160919] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [10100.164089] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10100.177094] ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10100.177128] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10100.177136] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10100.177142] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10100.177634] RIP: 0010:native_safe_halt+0x2e/0x50 [10100.177644] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10100.177648] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10100.177653] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [10100.177655] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c2134154 [10100.177658] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [10100.177659] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [10100.177662] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [10100.177665] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [10100.177668] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10100.177671] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [10100.177679] Call Trace: [10100.177685] [10100.177691] ? show_regs.cold.9+0x22/0x2f [10100.177701] ? watchdog_timer_fn.cold.6+0x62/0x10a [10100.177708] ? __hrtimer_run_queues+0x1c2/0x4d0 [10100.177713] ? watchdog+0x40/0x40 [10100.177717] ? hrtimer_interrupt+0x128/0x2c0 [10100.177720] ? smp_apic_timer_interrupt+0xa6/0x270 [10100.177726] ? apic_timer_interrupt+0xf/0x20 [10100.177731] [10100.177734] ? native_safe_halt+0x2e/0x50 [10100.177739] kvm_wait+0xcd/0xe0 [10100.177746] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10100.177754] do_raw_spin_lock+0x177/0x190 [10100.177759] _raw_spin_lock+0x21/0x30 [10100.177763] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10100.178073] process_one_work+0x2c8/0x700 [10100.178082] worker_thread+0x296/0x6e0 [10100.178102] ? rescuer_thread+0x570/0x570 [10100.178106] kthread+0x1d1/0x200 [10100.178111] ? set_kthread_struct+0x70/0x70 [10100.178114] ret_from_fork+0x1f/0x30 [10100.189492] ll_md_real_close+0xa5/0x2a0 [lustre] [10100.189656] ll_md_close+0x274/0x800 [lustre] [10100.189762] ll_file_release+0x128/0xc50 [lustre] [10100.190118] __fput+0x10d/0x440 [10100.190129] ____fput+0x12/0x20 [10100.190134] task_work_run+0xc6/0x110 [10100.190142] exit_to_usermode_loop+0x1dd/0x1f0 [10100.190150] do_syscall_64+0x3d6/0x440 [10100.190156] entry_SYSCALL_64_after_hwframe+0x49/0xae [10100.190171] RIP: 0033:0x7ff54af45cd8 [10100.190178] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10100.190182] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10100.190188] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [10100.190190] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [10100.190193] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [10100.190195] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [10100.190197] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10108.056894] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [10108.059309] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10108.099693] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10108.113835] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10108.121837] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10108.141637] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10108.163793] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10108.171187] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [10108.179604] RDX: 00000000000074e7 RSI: 0000000000000003 RDI: ffff96848811cae0 [10108.190560] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [10108.201108] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [10108.220693] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10108.225288] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [10108.239115] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10108.250726] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [10108.266051] Call Trace: [10108.267167] [10108.269233] ? show_regs.cold.9+0x22/0x2f [10108.271183] ? watchdog_timer_fn.cold.6+0x62/0x10a [10108.279090] ? __hrtimer_run_queues+0x1c2/0x4d0 [10108.286014] ? watchdog+0x40/0x40 [10108.287199] ? hrtimer_interrupt+0x128/0x2c0 [10108.298465] ? smp_apic_timer_interrupt+0xa6/0x270 [10108.300667] ? apic_timer_interrupt+0xf/0x20 [10108.305934] [10108.306751] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10108.308854] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10108.310870] do_raw_spin_lock+0x177/0x190 [10108.312818] _raw_spin_lock+0x21/0x30 [10108.317945] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10108.321163] obd_mod_free+0x46/0x140 [ptlrpc] [10108.323477] mdc_commit_open+0x91/0xb0 [mdc] [10108.325746] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10108.327565] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10108.329811] after_reply+0xe2c/0x1780 [ptlrpc] [10108.331547] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10108.339522] ptlrpcd+0xa82/0xe40 [ptlrpc] [10108.341693] ? do_wait_intr+0xf0/0xf0 [10108.346862] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10108.350447] kthread+0x1d1/0x200 [10108.354102] ? set_kthread_struct+0x70/0x70 [10108.357832] ret_from_fork+0x1f/0x30 [10114.015729] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8155s! [10114.021919] Showing busy workqueues and worker pools: [10114.025492] workqueue events: flags=0x0 [10114.027176] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10114.029501] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [10114.034443] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [10114.037206] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [10114.043537] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10114.047405] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [10114.051117] workqueue events_long: flags=0x0 [10114.053793] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10114.057847] pending: lu_objects_flush [obdclass] [10114.061265] workqueue events_unbound: flags=0x2 [10114.063876] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [10114.067995] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10114.073610] workqueue events_power_efficient: flags=0x80 [10114.077464] rcu: INFO: rcu_sched self-detected stall on CPU [10114.077477] rcu: 0-...!: (4256369 ticks this GP) idle=98a/1/0x4000000000000004 softirq=125154/125154 fqs=959 [10114.077487] (t=8160237 jiffies g=253585 q=5496) [10114.077491] rcu: rcu_sched kthread starved for 8155197 jiffies! g253585 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [10114.077495] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [10114.077497] rcu: RCU grace-period kthread stack dump: [10114.077499] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [10114.077506] Call Trace: [10114.077513] __schedule+0x351/0xcb0 [10114.077528] schedule+0xc0/0x180 [10114.077534] schedule_timeout+0xb4/0x190 [10114.077538] ? do_raw_spin_unlock+0x75/0x190 [10114.077545] ? __next_timer_interrupt+0x160/0x160 [10114.077553] ? prepare_to_swait_event+0xf3/0x140 [10114.077561] rcu_gp_kthread+0x8dd/0x15b0 [10114.077567] ? force_qs_rnp+0x2c0/0x2c0 [10114.077570] kthread+0x1d1/0x200 [10114.077576] ? set_kthread_struct+0x70/0x70 [10114.077580] ret_from_fork+0x1f/0x30 [10114.077657] rcu: Stack dump where RCU GP kthread last ran: [10114.077659] Sending NMI from CPU 0 to CPUs 2: [10114.077723] NMI backtrace for cpu 2 skipped: idling at native_safe_halt+0x2e/0x50 [10114.078681] NMI backtrace for cpu 0 [10114.078689] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10114.078696] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10114.078698] Call Trace: [10114.078702] [10114.078705] ? dump_stack+0xbb/0x10e [10114.078722] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [10114.078728] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [10114.078734] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [10114.078739] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [10114.078745] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [10114.078749] ? rcu_dump_cpu_stacks+0x11b/0x171 [10114.078796] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [10114.078802] ? update_process_times+0x6d/0xc0 [10114.078809] ? tick_sched_handle.isra.18+0x27/0x90 [10114.078814] ? tick_sched_timer+0x53/0xb0 [10114.078818] ? __hrtimer_run_queues+0x1c2/0x4d0 [10114.078822] ? tick_sched_handle.isra.18+0x90/0x90 [10114.078827] ? hrtimer_interrupt+0x128/0x2c0 [10114.078830] ? smp_apic_timer_interrupt+0xa6/0x270 [10114.078836] ? apic_timer_interrupt+0xf/0x20 [10114.078841] ? apic_timer_interrupt+0xa/0x20 [10114.078847] ? console_unlock+0x62d/0x940 [10114.078858] ? vprintk_emit+0x274/0x540 [10114.078862] ? apic_timer_interrupt+0xa/0x20 [10114.078867] ? vprintk_default+0x21/0x30 [10114.078870] ? vprintk+0x5b/0x1d0 [10114.078874] ? printk+0x5f/0x7d [10114.078879] ? show_workqueue_state.cold.41+0xec/0x1ee [10114.078885] ? wq_watchdog_timer_fn+0x17c/0x1a0 [10114.078890] ? show_workqueue_state+0x1d0/0x1d0 [10114.078893] ? call_timer_fn+0x53/0x270 [10114.078898] ? run_timer_softirq+0x6da/0x830 [10114.078903] ? show_workqueue_state+0x1d0/0x1d0 [10114.078907] ? kvm_clock_get_cycles+0x2c/0x50 [10114.078915] ? kvm_sched_clock_read+0x2c/0x50 [10114.078919] ? sched_clock+0xd/0x20 [10114.078926] ? __do_softirq+0x134/0x4e8 [10114.078932] ? irq_exit_rcu+0x1c6/0x1e0 [10114.078937] ? irq_exit+0x12/0x30 [10114.078940] ? smp_apic_timer_interrupt+0xe0/0x270 [10114.078944] ? apic_timer_interrupt+0xf/0x20 [10114.078949] [10114.078953] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10114.078957] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10114.078961] ? do_raw_spin_lock+0x177/0x190 [10114.078965] ? _raw_spin_lock+0x21/0x30 [10114.078971] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10114.079373] ? obd_mod_free+0x46/0x140 [ptlrpc] [10114.079676] ? mdc_commit_open+0x91/0xb0 [mdc] [10114.079721] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10114.080001] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10114.080405] ? after_reply+0xe2c/0x1780 [ptlrpc] [10114.080686] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10114.080905] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [10114.081406] ? do_wait_intr+0xf0/0xf0 [10114.081465] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10114.081992] ? kthread+0x1d1/0x200 [10114.081998] ? set_kthread_struct+0x70/0x70 [10114.082003] ? ret_from_fork+0x1f/0x30 [10114.297846] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10114.301039] pending: neigh_periodic_work [10114.303574] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10114.307273] pending: do_cache_clean, check_lifetime [10114.309890] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10114.312769] pending: neigh_periodic_work [10114.314581] workqueue writeback: flags=0x4a [10114.316643] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10114.319515] pending: wb_workfn, wb_workfn [10114.321924] workqueue xprtiod: flags=0xa [10114.323559] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10114.326513] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [10114.330198] workqueue ipv6_addrconf: flags=0x40008 [10114.332559] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10114.335591] pending: addrconf_verify_work [10114.337793] workqueue knbd0-recv: flags=0x1a [10114.340670] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10114.344162] in-flight: 657:recv_work [10114.345699] workqueue ptlrpc_pinger: flags=0x2 [10114.347731] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10114.350091] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [10114.352861] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8155s workers=4 idle: 20831 12642 28859 [10114.356895] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10104s workers=2 idle: 659 [10116.060024] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [10116.066356] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10116.089735] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10116.095536] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10116.100614] RIP: 0010:native_safe_halt+0x2e/0x50 [10116.104853] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10116.123001] RSP: 0000:ffffaf9fc15234f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10116.127843] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [10116.133485] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [10116.138749] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [10116.141788] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [10116.146238] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [10116.149909] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [10116.155782] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10116.159440] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [10116.168603] Call Trace: [10116.174010] [10116.176925] ? show_regs.cold.9+0x22/0x2f [10116.179658] ? watchdog_timer_fn.cold.6+0x62/0x10a [10116.182340] ? __hrtimer_run_queues+0x1c2/0x4d0 [10116.184811] ? watchdog+0x40/0x40 [10116.186874] ? hrtimer_interrupt+0x128/0x2c0 [10116.188334] ? smp_apic_timer_interrupt+0xa6/0x270 [10116.192731] ? apic_timer_interrupt+0xf/0x20 [10116.195874] [10116.197395] ? native_safe_halt+0x2e/0x50 [10116.199811] kvm_wait+0xcd/0xe0 [10116.201778] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10116.205045] do_raw_spin_lock+0x177/0x190 [10116.208387] _raw_spin_lock+0x21/0x30 [10116.211033] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10116.214922] ? do_raw_spin_unlock+0x75/0x190 [10116.217066] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10116.220402] ? do_wait_intr+0xf0/0xf0 [10116.225053] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10116.231505] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10116.239034] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10116.247651] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10116.252564] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10116.257244] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10116.259690] mdc_intent_lock+0x353/0x7e0 [mdc] [10116.261753] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10116.264527] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10116.267831] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10116.270165] lmv_intent_open+0x338/0x1150 [lmv] [10116.272975] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10116.275792] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10116.278234] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10116.280542] ll_intent_lock+0xc5/0xaa0 [lustre] [10116.282984] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10116.285825] ll_lookup_it+0xe0e/0x2270 [lustre] [10116.288709] ? do_raw_spin_unlock+0x75/0x190 [10116.290809] ? _raw_spin_unlock+0x12/0x30 [10116.294729] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10116.298788] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10116.304507] ll_atomic_open+0x36a/0xd70 [lustre] [10116.308980] lookup_open+0x356/0xb40 [10116.310428] do_last+0x313/0xfc0 [10116.311645] ? nd_jump_root+0xe5/0x160 [10116.314201] ? path_init+0x437/0x520 [10116.316307] path_openat+0xf7/0x500 [10116.318187] ? do_raw_spin_unlock+0x75/0x190 [10116.320336] do_filp_open+0x99/0x140 [10116.322359] ? getname_flags+0x6e/0x330 [10116.324450] ? __check_object_size+0xff/0x256 [10116.327363] ? do_raw_spin_unlock+0x75/0x190 [10116.330919] ? _raw_spin_unlock+0x12/0x30 [10116.332979] do_sys_openat2+0x2b4/0x410 [10116.335249] do_sys_open+0x73/0xa0 [10116.337225] __x64_sys_openat+0x24/0x30 [10116.339353] do_syscall_64+0xc1/0x440 [10116.341340] entry_SYSCALL_64_after_hwframe+0x49/0xae [10116.344487] RIP: 0033:0x7ff54af45332 [10116.347068] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10116.356600] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10116.360935] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [10116.364547] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [10116.367416] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [10116.371065] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [10116.375084] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [10128.060246] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [10128.062623] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [10128.070446] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10128.071826] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10128.071834] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10128.071837] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10128.071854] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10128.071858] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10128.071863] RAX: 0000000000000889 RBX: ffff96848811cae0 RCX: 0000000000000001 [10128.071866] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [10128.071868] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [10128.071870] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [10128.071873] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [10128.071876] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [10128.071879] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10128.071882] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [10128.071891] Call Trace: [10128.071897] [10128.071903] ? show_regs.cold.9+0x22/0x2f [10128.071913] ? watchdog_timer_fn.cold.6+0x62/0x10a [10128.071919] ? __hrtimer_run_queues+0x1c2/0x4d0 [10128.071924] ? watchdog+0x40/0x40 [10128.071929] ? hrtimer_interrupt+0x128/0x2c0 [10128.071933] ? smp_apic_timer_interrupt+0xa6/0x270 [10128.071943] ? apic_timer_interrupt+0xf/0x20 [10128.071947] [10128.071950] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10128.071958] do_raw_spin_lock+0x177/0x190 [10128.071962] _raw_spin_lock+0x21/0x30 [10128.071966] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10128.073227] ? do_raw_spin_unlock+0x75/0x190 [10128.073234] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10128.073364] ? do_wait_intr+0xf0/0xf0 [10128.073371] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10128.074254] mdc_close+0x53b/0xd70 [mdc] [10128.074289] lmv_close+0x18d/0x490 [lmv] [10128.074309] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [10128.074428] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10128.074560] ll_md_real_close+0xa5/0x2a0 [lustre] [10128.075726] ll_md_close+0x274/0x800 [lustre] [10128.075798] ll_file_release+0x128/0xc50 [lustre] [10128.075860] __fput+0x10d/0x440 [10128.075870] ____fput+0x12/0x20 [10128.075875] task_work_run+0xc6/0x110 [10128.075881] exit_to_usermode_loop+0x1dd/0x1f0 [10128.075888] do_syscall_64+0x3d6/0x440 [10128.075892] entry_SYSCALL_64_after_hwframe+0x49/0xae [10128.075901] RIP: 0033:0x7ff54af45cd8 [10128.075908] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10128.075912] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10128.075917] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [10128.075919] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [10128.075921] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [10128.075924] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [10128.075926] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10128.415214] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10128.485423] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10128.500646] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10128.514785] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10128.524222] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10128.534828] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10128.573390] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10128.593289] RAX: 0000000000007b86 RBX: ffff96848811cae0 RCX: 0000000000000001 [10128.606792] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [10128.609472] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [10128.612224] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [10128.615348] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [10128.617983] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [10128.621575] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10128.624442] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [10128.627468] Call Trace: [10128.628503] [10128.633548] ? show_regs.cold.9+0x22/0x2f [10128.636815] ? watchdog_timer_fn.cold.6+0x62/0x10a [10128.638899] ? __hrtimer_run_queues+0x1c2/0x4d0 [10128.643297] ? watchdog+0x40/0x40 [10128.644605] ? hrtimer_interrupt+0x128/0x2c0 [10128.646187] ? smp_apic_timer_interrupt+0xa6/0x270 [10128.648459] ? apic_timer_interrupt+0xf/0x20 [10128.649996] [10128.650761] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10128.655338] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10128.656887] do_raw_spin_lock+0x177/0x190 [10128.659327] _raw_spin_lock+0x21/0x30 [10128.661485] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10128.663853] process_one_work+0x2c8/0x700 [10128.666370] worker_thread+0x296/0x6e0 [10128.667317] ? rescuer_thread+0x570/0x570 [10128.669974] kthread+0x1d1/0x200 [10128.671406] ? set_kthread_struct+0x70/0x70 [10128.676524] ret_from_fork+0x1f/0x30 [10140.056561] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [10140.076101] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10140.115387] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10140.126544] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10140.135228] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10140.138060] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10140.154510] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10140.163130] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [10140.173150] RDX: 00000000000027f2 RSI: 0000000000000003 RDI: ffff96848811cae0 [10140.177514] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [10140.181364] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [10140.187515] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10140.193699] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [10140.200097] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10140.205473] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [10140.215684] Call Trace: [10140.216760] [10140.217805] ? show_regs.cold.9+0x22/0x2f [10140.221608] ? watchdog_timer_fn.cold.6+0x62/0x10a [10140.223812] ? __hrtimer_run_queues+0x1c2/0x4d0 [10140.228655] ? watchdog+0x40/0x40 [10140.229761] ? hrtimer_interrupt+0x128/0x2c0 [10140.232795] ? smp_apic_timer_interrupt+0xa6/0x270 [10140.237008] ? apic_timer_interrupt+0xf/0x20 [10140.238596] [10140.239644] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10140.245879] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10140.250090] do_raw_spin_lock+0x177/0x190 [10140.250832] _raw_spin_lock+0x21/0x30 [10140.252329] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10140.263523] obd_mod_free+0x46/0x140 [ptlrpc] [10140.266686] mdc_commit_open+0x91/0xb0 [mdc] [10140.268113] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10140.270593] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10140.273390] after_reply+0xe2c/0x1780 [ptlrpc] [10140.276625] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10140.281811] ptlrpcd+0xa82/0xe40 [ptlrpc] [10140.284056] ? do_wait_intr+0xf0/0xf0 [10140.285278] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10140.295221] kthread+0x1d1/0x200 [10140.295964] ? set_kthread_struct+0x70/0x70 [10140.297010] ret_from_fork+0x1f/0x30 [10144.060021] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [10144.066332] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10144.119459] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10144.131072] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10144.136314] RIP: 0010:native_safe_halt+0x2e/0x50 [10144.140151] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10144.151204] RSP: 0000:ffffaf9fc15234f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10144.154289] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [10144.158431] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c21b4154 [10144.162018] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [10144.169249] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [10144.173121] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [10144.189463] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [10144.197345] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10144.204226] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [10144.209344] Call Trace: [10144.210296] [10144.212209] ? show_regs.cold.9+0x22/0x2f [10144.214070] ? watchdog_timer_fn.cold.6+0x62/0x10a [10144.216272] ? __hrtimer_run_queues+0x1c2/0x4d0 [10144.222160] ? watchdog+0x40/0x40 [10144.226745] ? hrtimer_interrupt+0x128/0x2c0 [10144.232985] ? smp_apic_timer_interrupt+0xa6/0x270 [10144.239383] ? apic_timer_interrupt+0xf/0x20 [10144.245235] [10144.247494] ? native_safe_halt+0x2e/0x50 [10144.252446] kvm_wait+0xcd/0xe0 [10144.256100] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10144.257483] do_raw_spin_lock+0x177/0x190 [10144.260052] _raw_spin_lock+0x21/0x30 [10144.261288] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10144.264012] ? do_raw_spin_unlock+0x75/0x190 [10144.266504] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10144.270089] ? do_wait_intr+0xf0/0xf0 [10144.273614] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10144.276672] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10144.279489] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10144.282482] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10144.286717] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10144.296445] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10144.298059] mdc_intent_lock+0x353/0x7e0 [mdc] [10144.301065] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10144.305354] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10144.308157] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10144.311793] lmv_intent_open+0x338/0x1150 [lmv] [10144.320628] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10144.326700] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10144.335110] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10144.339457] ll_intent_lock+0xc5/0xaa0 [lustre] [10144.342430] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10144.345983] ll_lookup_it+0xe0e/0x2270 [lustre] [10144.349107] ? do_raw_spin_unlock+0x75/0x190 [10144.352507] ? _raw_spin_unlock+0x12/0x30 [10144.356171] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10144.360905] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10144.364313] ll_atomic_open+0x36a/0xd70 [lustre] [10144.368985] lookup_open+0x356/0xb40 [10144.370429] do_last+0x313/0xfc0 [10144.373059] ? nd_jump_root+0xe5/0x160 [10144.374692] ? path_init+0x437/0x520 [10144.376702] path_openat+0xf7/0x500 [10144.377747] ? do_raw_spin_unlock+0x75/0x190 [10144.379095] do_filp_open+0x99/0x140 [10144.387424] ? getname_flags+0x6e/0x330 [10144.388840] ? __check_object_size+0xff/0x256 [10144.391438] ? do_raw_spin_unlock+0x75/0x190 [10144.393246] ? _raw_spin_unlock+0x12/0x30 [10144.395520] do_sys_openat2+0x2b4/0x410 [10144.399004] do_sys_open+0x73/0xa0 [10144.400657] __x64_sys_openat+0x24/0x30 [10144.402781] do_syscall_64+0xc1/0x440 [10144.404433] entry_SYSCALL_64_after_hwframe+0x49/0xae [10144.407200] RIP: 0033:0x7ff54af45332 [10144.408490] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10144.421132] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10144.426830] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [10144.430085] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [10144.433228] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [10144.441013] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [10144.446897] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [10144.735030] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8185s! [10144.744815] Showing busy workqueues and worker pools: [10144.749848] workqueue events: flags=0x0 [10144.753015] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10144.755914] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [10144.763932] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [10144.769308] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [10144.782970] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10144.795170] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [10144.803130] workqueue events_long: flags=0x0 [10144.809119] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10144.814526] pending: lu_objects_flush [obdclass] [10144.821421] workqueue events_unbound: flags=0x2 [10144.824606] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [10144.833332] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10144.839789] workqueue events_power_efficient: flags=0x80 [10144.842688] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10144.846362] pending: neigh_periodic_work [10144.848793] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10144.852088] pending: do_cache_clean, check_lifetime [10144.854767] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10144.858711] pending: neigh_periodic_work [10144.860098] workqueue writeback: flags=0x4a [10144.861316] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10144.864399] pending: wb_workfn, wb_workfn [10144.865839] workqueue xprtiod: flags=0xa [10144.867648] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10144.870674] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [10144.874921] workqueue ipv6_addrconf: flags=0x40008 [10144.877273] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10144.880326] pending: addrconf_verify_work [10144.882688] workqueue knbd0-recv: flags=0x1a [10144.884734] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10144.888187] in-flight: 657:recv_work [10144.889666] workqueue ptlrpc_pinger: flags=0x2 [10144.891702] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10144.898452] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [10144.901717] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8186s workers=4 idle: 20831 12642 28859 [10144.909135] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10135s workers=2 idle: 659 [10156.058211] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [10156.059007] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [10156.066450] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10156.066531] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10156.066539] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10156.066542] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10156.066557] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10156.066561] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10156.066566] RAX: 000000000000223b RBX: ffff96848811cae0 RCX: 0000000000000001 [10156.066569] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [10156.066571] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [10156.066574] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [10156.066576] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [10156.066580] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [10156.066583] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10156.066585] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [10156.066594] Call Trace: [10156.066600] [10156.066605] ? show_regs.cold.9+0x22/0x2f [10156.066614] ? watchdog_timer_fn.cold.6+0x62/0x10a [10156.066620] ? __hrtimer_run_queues+0x1c2/0x4d0 [10156.066624] ? watchdog+0x40/0x40 [10156.066628] ? hrtimer_interrupt+0x128/0x2c0 [10156.066632] ? smp_apic_timer_interrupt+0xa6/0x270 [10156.066640] ? apic_timer_interrupt+0xf/0x20 [10156.066645] [10156.066648] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10156.066655] do_raw_spin_lock+0x177/0x190 [10156.066660] _raw_spin_lock+0x21/0x30 [10156.066665] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10156.076207] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10156.076465] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10156.076471] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10156.076478] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10156.078230] RIP: 0010:__pv_queued_spin_lock_slowpath+0x328/0x6d0 [10156.078247] Code: d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 05 18 d4 dc 02 01 e0 48 83 05 06 d4 dc 02 01 48 83 05 be d5 dc 02 01 8b 45 08 85 [10156.078252] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10156.078256] RAX: 0000000000007540 RBX: ffff96848811cae0 RCX: 0000000000000001 [10156.078262] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [10156.078264] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [10156.078266] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [10156.078269] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [10156.078272] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [10156.078291] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10156.078297] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [10156.078307] Call Trace: [10156.078313] [10156.078321] ? show_regs.cold.9+0x22/0x2f [10156.078331] ? watchdog_timer_fn.cold.6+0x62/0x10a [10156.078338] ? __hrtimer_run_queues+0x1c2/0x4d0 [10156.078342] ? watchdog+0x40/0x40 [10156.078346] ? hrtimer_interrupt+0x128/0x2c0 [10156.078349] ? smp_apic_timer_interrupt+0xa6/0x270 [10156.078357] ? apic_timer_interrupt+0xf/0x20 [10156.078362] [10156.078365] ? __pv_queued_spin_lock_slowpath+0x328/0x6d0 [10156.078370] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10156.078375] do_raw_spin_lock+0x177/0x190 [10156.078380] _raw_spin_lock+0x21/0x30 [10156.078385] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10156.078719] process_one_work+0x2c8/0x700 [10156.078725] worker_thread+0x296/0x6e0 [10156.078729] ? rescuer_thread+0x570/0x570 [10156.078733] kthread+0x1d1/0x200 [10156.078738] ? set_kthread_struct+0x70/0x70 [10156.078742] ret_from_fork+0x1f/0x30 [10156.354877] ? do_raw_spin_unlock+0x75/0x190 [10156.356900] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10156.359012] ? do_wait_intr+0xf0/0xf0 [10156.360468] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10156.362673] mdc_close+0x53b/0xd70 [mdc] [10156.364581] lmv_close+0x18d/0x490 [lmv] [10156.366108] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [10156.368380] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10156.370949] ll_md_real_close+0xa5/0x2a0 [lustre] [10156.372581] ll_md_close+0x274/0x800 [lustre] [10156.374618] ll_file_release+0x128/0xc50 [lustre] [10156.377316] __fput+0x10d/0x440 [10156.378636] ____fput+0x12/0x20 [10156.380119] task_work_run+0xc6/0x110 [10156.381689] exit_to_usermode_loop+0x1dd/0x1f0 [10156.384187] do_syscall_64+0x3d6/0x440 [10156.387299] entry_SYSCALL_64_after_hwframe+0x49/0xae [10156.389628] RIP: 0033:0x7ff54af45cd8 [10156.393419] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10156.402714] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10156.406603] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [10156.410102] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [10156.413373] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [10156.416964] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [10156.419261] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10172.055016] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [10172.060030] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [10172.061228] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log [10172.066112] Modules linked in: [10172.066114] dm_mod sha512_ssse3 sha512_generic [10172.087263] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul [10172.089078] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10172.092783] crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata [10172.124530] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10172.124540] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10172.124555] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10172.124560] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10172.124566] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [10172.124569] RDX: 000000000000555d RSI: 0000000000000003 RDI: ffff96848811cae0 [10172.124571] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [10172.124573] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [10172.124576] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10172.124580] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [10172.124582] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10172.124585] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [10172.124600] Call Trace: [10172.124606] [10172.124612] ? show_regs.cold.9+0x22/0x2f [10172.124622] ? watchdog_timer_fn.cold.6+0x62/0x10a [10172.124629] ? __hrtimer_run_queues+0x1c2/0x4d0 [10172.124634] ? watchdog+0x40/0x40 [10172.124638] ? hrtimer_interrupt+0x128/0x2c0 [10172.124642] ? smp_apic_timer_interrupt+0xa6/0x270 [10172.124649] ? apic_timer_interrupt+0xf/0x20 [10172.124653] [10172.124657] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10172.124662] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10172.124667] do_raw_spin_lock+0x177/0x190 [10172.124671] _raw_spin_lock+0x21/0x30 [10172.124679] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10172.125114] obd_mod_free+0x46/0x140 [ptlrpc] [10172.125326] mdc_commit_open+0x91/0xb0 [mdc] [10172.125362] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10172.125992] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10172.126194] after_reply+0xe2c/0x1780 [ptlrpc] [10172.126395] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10172.138980] dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10172.139004] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10172.139011] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10172.139014] RIP: 0010:native_safe_halt+0x2e/0x50 [10172.139028] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10172.139032] RSP: 0000:ffffaf9fc15234f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10172.139036] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [10172.139039] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [10172.139041] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [10172.139043] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [10172.139046] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [10172.139050] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [10172.139053] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10172.139056] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [10172.139064] Call Trace: [10172.139069] [10172.139074] ? show_regs.cold.9+0x22/0x2f [10172.139084] ? watchdog_timer_fn.cold.6+0x62/0x10a [10172.139090] ? __hrtimer_run_queues+0x1c2/0x4d0 [10172.139095] ? watchdog+0x40/0x40 [10172.139101] ? hrtimer_interrupt+0x128/0x2c0 [10172.139106] ? smp_apic_timer_interrupt+0xa6/0x270 [10172.139114] ? apic_timer_interrupt+0xf/0x20 [10172.139119] [10172.139122] ? native_safe_halt+0x2e/0x50 [10172.139126] kvm_wait+0xcd/0xe0 [10172.139133] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10172.139142] do_raw_spin_lock+0x177/0x190 [10172.139147] _raw_spin_lock+0x21/0x30 [10172.139152] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10172.150511] ptlrpcd+0xa82/0xe40 [ptlrpc] [10172.167081] ? do_raw_spin_unlock+0x75/0x190 [10172.167101] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10172.167409] ? do_wait_intr+0xf0/0xf0 [10172.167419] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10172.167637] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10172.169651] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10172.172777] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10172.173061] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10172.173104] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10172.173133] mdc_intent_lock+0x353/0x7e0 [mdc] [10172.173158] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10172.173319] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10172.177259] ? do_wait_intr+0xf0/0xf0 [10172.177283] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10172.189347] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10172.189403] lmv_intent_open+0x338/0x1150 [lmv] [10172.189436] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10172.202210] kthread+0x1d1/0x200 [10172.202232] ? set_kthread_struct+0x70/0x70 [10172.202237] ret_from_fork+0x1f/0x30 [10172.573567] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10172.576567] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10172.578255] ll_intent_lock+0xc5/0xaa0 [lustre] [10172.580559] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10172.583065] ll_lookup_it+0xe0e/0x2270 [lustre] [10172.585264] ? do_raw_spin_unlock+0x75/0x190 [10172.587288] ? _raw_spin_unlock+0x12/0x30 [10172.588958] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10172.591271] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10172.593400] ll_atomic_open+0x36a/0xd70 [lustre] [10172.595477] lookup_open+0x356/0xb40 [10172.597034] do_last+0x313/0xfc0 [10172.598473] ? nd_jump_root+0xe5/0x160 [10172.600061] ? path_init+0x437/0x520 [10172.601900] path_openat+0xf7/0x500 [10172.603667] ? do_raw_spin_unlock+0x75/0x190 [10172.605353] do_filp_open+0x99/0x140 [10172.606874] ? getname_flags+0x6e/0x330 [10172.607969] ? __check_object_size+0xff/0x256 [10172.609865] ? do_raw_spin_unlock+0x75/0x190 [10172.611075] ? _raw_spin_unlock+0x12/0x30 [10172.612478] do_sys_openat2+0x2b4/0x410 [10172.614018] do_sys_open+0x73/0xa0 [10172.616257] __x64_sys_openat+0x24/0x30 [10172.618926] do_syscall_64+0xc1/0x440 [10172.621623] entry_SYSCALL_64_after_hwframe+0x49/0xae [10172.625810] RIP: 0033:0x7ff54af45332 [10172.627851] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10172.652049] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10172.658127] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [10172.672118] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [10172.678202] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [10172.685045] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [10172.697490] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [10174.943036] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8216s! [10174.951379] Showing busy workqueues and worker pools: [10174.958220] workqueue events: flags=0x0 [10174.962947] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10174.977995] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [10174.990015] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [10174.998993] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [10175.010043] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10175.014517] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [10175.019490] workqueue events_long: flags=0x0 [10175.025544] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10175.032574] pending: lu_objects_flush [obdclass] [10175.037325] workqueue events_unbound: flags=0x2 [10175.040621] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [10175.048336] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10175.060078] workqueue events_power_efficient: flags=0x80 [10175.071260] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10175.082486] pending: neigh_periodic_work [10175.092063] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10175.101092] pending: do_cache_clean, check_lifetime [10175.111887] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10175.121635] pending: neigh_periodic_work [10175.128855] workqueue writeback: flags=0x4a [10175.138698] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10175.151689] pending: wb_workfn, wb_workfn [10175.159018] workqueue xprtiod: flags=0xa [10175.165366] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10175.171098] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [10175.174899] workqueue ipv6_addrconf: flags=0x40008 [10175.186016] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10175.194498] pending: addrconf_verify_work [10175.197651] workqueue knbd0-recv: flags=0x1a [10175.201673] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10175.215319] in-flight: 657:recv_work [10175.223211] workqueue ptlrpc_pinger: flags=0x2 [10175.230194] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10175.243365] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [10175.258405] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8216s workers=4 idle: 20831 12642 28859 [10175.271358] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10165s workers=2 idle: 659 [10184.059018] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [10184.061035] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [10184.062575] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10184.062679] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10184.062688] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10184.062698] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10184.063154] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10184.063176] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10184.063181] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10184.063186] RAX: 0000000000002c90 RBX: ffff96848811cae0 RCX: 0000000000000001 [10184.063189] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [10184.063192] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [10184.063195] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [10184.063198] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [10184.063202] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [10184.063205] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10184.063208] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [10184.063290] Call Trace: [10184.063299] [10184.063307] ? show_regs.cold.9+0x22/0x2f [10184.063319] ? watchdog_timer_fn.cold.6+0x62/0x10a [10184.063327] ? __hrtimer_run_queues+0x1c2/0x4d0 [10184.063331] ? watchdog+0x40/0x40 [10184.063336] ? hrtimer_interrupt+0x128/0x2c0 [10184.063340] ? smp_apic_timer_interrupt+0xa6/0x270 [10184.063354] ? apic_timer_interrupt+0xf/0x20 [10184.063358] [10184.063361] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10184.063366] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10184.063371] do_raw_spin_lock+0x177/0x190 [10184.063376] _raw_spin_lock+0x21/0x30 [10184.063381] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10184.063807] process_one_work+0x2c8/0x700 [10184.063815] worker_thread+0x296/0x6e0 [10184.063820] ? rescuer_thread+0x570/0x570 [10184.063824] kthread+0x1d1/0x200 [10184.063876] ? set_kthread_struct+0x70/0x70 [10184.063880] ret_from_fork+0x1f/0x30 [10184.209173] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10184.229266] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10184.234417] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10184.238166] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10184.240877] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10184.248819] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10184.252042] RAX: 0000000000001499 RBX: ffff96848811cae0 RCX: 0000000000000001 [10184.255276] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [10184.258625] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [10184.262368] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [10184.265433] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [10184.268600] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [10184.271789] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10184.274289] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [10184.277175] Call Trace: [10184.278428] [10184.279225] ? show_regs.cold.9+0x22/0x2f [10184.281117] ? watchdog_timer_fn.cold.6+0x62/0x10a [10184.282965] ? __hrtimer_run_queues+0x1c2/0x4d0 [10184.285051] ? watchdog+0x40/0x40 [10184.286587] ? hrtimer_interrupt+0x128/0x2c0 [10184.288655] ? smp_apic_timer_interrupt+0xa6/0x270 [10184.290547] ? apic_timer_interrupt+0xf/0x20 [10184.292387] [10184.293408] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10184.295794] do_raw_spin_lock+0x177/0x190 [10184.297559] _raw_spin_lock+0x21/0x30 [10184.299306] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10184.301757] ? do_raw_spin_unlock+0x75/0x190 [10184.303392] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10184.306142] ? do_wait_intr+0xf0/0xf0 [10184.308375] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10184.310999] mdc_close+0x53b/0xd70 [mdc] [10184.312776] lmv_close+0x18d/0x490 [lmv] [10184.314536] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [10184.316936] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10184.319851] ll_md_real_close+0xa5/0x2a0 [lustre] [10184.322318] ll_md_close+0x274/0x800 [lustre] [10184.324265] ll_file_release+0x128/0xc50 [lustre] [10184.326579] __fput+0x10d/0x440 [10184.328051] ____fput+0x12/0x20 [10184.329694] task_work_run+0xc6/0x110 [10184.331548] exit_to_usermode_loop+0x1dd/0x1f0 [10184.333926] do_syscall_64+0x3d6/0x440 [10184.335751] entry_SYSCALL_64_after_hwframe+0x49/0xae [10184.338373] RIP: 0033:0x7ff54af45cd8 [10184.339881] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10184.348494] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10184.351617] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [10184.355548] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [10184.358888] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [10184.365225] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [10184.370357] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10200.055810] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [10200.065234] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul [10200.066708] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [10200.092913] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10200.093657] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10200.093666] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10200.093668] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10200.093688] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10200.093692] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10200.093696] RAX: 0000000000003b84 RBX: ffff96848811cae0 RCX: 0000000000000001 [10200.093699] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [10200.093701] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [10200.093703] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [10200.093705] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [10200.093709] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [10200.093711] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10200.093713] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [10200.093721] Call Trace: [10200.093726] [10200.093731] ? show_regs.cold.9+0x22/0x2f [10200.093741] ? watchdog_timer_fn.cold.6+0x62/0x10a [10200.093748] ? __hrtimer_run_queues+0x1c2/0x4d0 [10200.093752] ? watchdog+0x40/0x40 [10200.093756] ? hrtimer_interrupt+0x128/0x2c0 [10200.093761] ? smp_apic_timer_interrupt+0xa6/0x270 [10200.093769] ? apic_timer_interrupt+0xf/0x20 [10200.093773] [10200.093776] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10200.093782] do_raw_spin_lock+0x177/0x190 [10200.093787] _raw_spin_lock+0x21/0x30 [10200.093791] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10200.094083] ? do_raw_spin_unlock+0x75/0x190 [10200.094089] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10200.094278] ? do_wait_intr+0xf0/0xf0 [10200.094287] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10200.094476] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10200.095806] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10200.096022] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10200.096153] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10200.096193] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10200.096221] mdc_intent_lock+0x353/0x7e0 [mdc] [10200.096248] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10200.096923] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10200.097161] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10200.097194] lmv_intent_open+0x338/0x1150 [lmv] [10200.097228] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10200.098674] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10200.098778] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10200.098805] ll_intent_lock+0xc5/0xaa0 [lustre] [10200.098891] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10200.098972] ll_lookup_it+0xe0e/0x2270 [lustre] [10200.099053] ? do_raw_spin_unlock+0x75/0x190 [10200.099062] ? _raw_spin_unlock+0x12/0x30 [10200.099069] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10200.099096] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10200.099114] ll_atomic_open+0x36a/0xd70 [lustre] [10200.099200] lookup_open+0x356/0xb40 [10200.099209] do_last+0x313/0xfc0 [10200.099214] ? nd_jump_root+0xe5/0x160 [10200.099219] ? path_init+0x437/0x520 [10200.099223] path_openat+0xf7/0x500 [10200.099229] ? do_raw_spin_unlock+0x75/0x190 [10200.099234] do_filp_open+0x99/0x140 [10200.099239] ? getname_flags+0x6e/0x330 [10200.099244] ? __check_object_size+0xff/0x256 [10200.099252] ? do_raw_spin_unlock+0x75/0x190 [10200.099255] ? _raw_spin_unlock+0x12/0x30 [10200.105734] crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10200.105779] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10200.105786] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10200.105789] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10200.105804] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10200.105808] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10200.105813] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [10200.105816] RDX: 00000000000039b9 RSI: 0000000000000003 RDI: ffff96848811cae0 [10200.105818] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [10200.105821] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [10200.105823] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10200.105826] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [10200.105829] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10200.105831] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [10200.105840] Call Trace: [10200.105845] [10200.105851] ? show_regs.cold.9+0x22/0x2f [10200.105863] ? watchdog_timer_fn.cold.6+0x62/0x10a [10200.105870] ? __hrtimer_run_queues+0x1c2/0x4d0 [10200.105875] ? watchdog+0x40/0x40 [10200.105879] ? hrtimer_interrupt+0x128/0x2c0 [10200.105883] ? smp_apic_timer_interrupt+0xa6/0x270 [10200.105892] ? apic_timer_interrupt+0xf/0x20 [10200.105897] [10200.105900] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10200.105905] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10200.105909] do_raw_spin_lock+0x177/0x190 [10200.105914] _raw_spin_lock+0x21/0x30 [10200.105921] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10200.106746] obd_mod_free+0x46/0x140 [ptlrpc] [10200.106952] mdc_commit_open+0x91/0xb0 [mdc] [10200.106991] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10200.107185] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10200.143972] do_sys_openat2+0x2b4/0x410 [10200.143993] do_sys_open+0x73/0xa0 [10200.143999] __x64_sys_openat+0x24/0x30 [10200.144002] do_syscall_64+0xc1/0x440 [10200.144010] entry_SYSCALL_64_after_hwframe+0x49/0xae [10200.144019] RIP: 0033:0x7ff54af45332 [10200.144028] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10200.144036] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10200.144049] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [10200.144052] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [10200.144055] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [10200.144057] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [10200.144059] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [10200.530404] after_reply+0xe2c/0x1780 [ptlrpc] [10200.536026] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10200.542732] ptlrpcd+0xa82/0xe40 [ptlrpc] [10200.546202] ? do_wait_intr+0xf0/0xf0 [10200.552674] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10200.559919] kthread+0x1d1/0x200 [10200.564235] ? set_kthread_struct+0x70/0x70 [10200.569041] ret_from_fork+0x1f/0x30 [10205.664528] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8246s! [10205.676243] Showing busy workqueues and worker pools: [10205.677946] workqueue events: flags=0x0 [10205.679285] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10205.693520] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [10205.702404] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [10205.709817] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [10205.723420] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10205.740702] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [10205.749921] workqueue events_long: flags=0x0 [10205.758916] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10205.768704] pending: lu_objects_flush [obdclass] [10205.772809] workqueue events_unbound: flags=0x2 [10205.778957] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [10205.785077] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10205.799878] workqueue events_power_efficient: flags=0x80 [10205.807492] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10205.816777] pending: neigh_periodic_work [10205.824013] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10205.830514] pending: do_cache_clean, check_lifetime [10205.838752] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10205.849365] pending: neigh_periodic_work [10205.854851] workqueue writeback: flags=0x4a [10205.856634] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10205.860167] pending: wb_workfn, wb_workfn [10205.867037] workqueue xprtiod: flags=0xa [10205.872012] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10205.875186] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [10205.883754] workqueue ipv6_addrconf: flags=0x40008 [10205.888194] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10205.899327] pending: addrconf_verify_work [10205.905198] workqueue knbd0-recv: flags=0x1a [10205.909329] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10205.914004] in-flight: 657:recv_work [10205.920933] workqueue ptlrpc_pinger: flags=0x2 [10205.925815] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10205.933080] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [10205.939041] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8247s workers=4 idle: 20831 12642 28859 [10205.951017] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10196s workers=2 idle: 659 [10212.064992] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [10212.065863] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [10212.071903] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10212.072002] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10212.072012] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10212.072024] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10212.073973] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10212.073998] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10212.074005] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10212.074010] RAX: 0000000000007c96 RBX: ffff96848811cae0 RCX: 0000000000000001 [10212.074012] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [10212.074015] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [10212.074017] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [10212.074019] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [10212.074023] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [10212.074026] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10212.074028] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [10212.074036] Call Trace: [10212.074042] [10212.074050] ? show_regs.cold.9+0x22/0x2f [10212.074060] ? watchdog_timer_fn.cold.6+0x62/0x10a [10212.074068] ? __hrtimer_run_queues+0x1c2/0x4d0 [10212.074073] ? watchdog+0x40/0x40 [10212.074077] ? hrtimer_interrupt+0x128/0x2c0 [10212.074080] ? smp_apic_timer_interrupt+0xa6/0x270 [10212.074090] ? apic_timer_interrupt+0xf/0x20 [10212.074094] [10212.074097] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10212.074101] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10212.074107] do_raw_spin_lock+0x177/0x190 [10212.074111] _raw_spin_lock+0x21/0x30 [10212.074117] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10212.075896] process_one_work+0x2c8/0x700 [10212.075909] worker_thread+0x296/0x6e0 [10212.075918] ? rescuer_thread+0x570/0x570 [10212.075922] kthread+0x1d1/0x200 [10212.075927] ? set_kthread_struct+0x70/0x70 [10212.075930] ret_from_fork+0x1f/0x30 [10212.492104] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10212.590332] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10212.595345] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10212.602876] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10212.606121] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10212.620429] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10212.624682] RAX: 0000000000005bad RBX: ffff96848811cae0 RCX: 0000000000000001 [10212.633559] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [10212.642219] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [10212.647349] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [10212.649291] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [10212.655651] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [10212.662664] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10212.665657] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [10212.678400] Call Trace: [10212.680718] [10212.682502] ? show_regs.cold.9+0x22/0x2f [10212.689157] ? watchdog_timer_fn.cold.6+0x62/0x10a [10212.698665] ? __hrtimer_run_queues+0x1c2/0x4d0 [10212.703032] ? watchdog+0x40/0x40 [10212.711559] ? hrtimer_interrupt+0x128/0x2c0 [10212.714460] ? smp_apic_timer_interrupt+0xa6/0x270 [10212.720906] ? apic_timer_interrupt+0xf/0x20 [10212.730665] [10212.733930] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10212.743925] do_raw_spin_lock+0x177/0x190 [10212.751183] _raw_spin_lock+0x21/0x30 [10212.761289] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10212.777914] ? do_raw_spin_unlock+0x75/0x190 [10212.784931] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10212.793489] ? do_wait_intr+0xf0/0xf0 [10212.799116] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10212.813229] mdc_close+0x53b/0xd70 [mdc] [10212.825737] lmv_close+0x18d/0x490 [lmv] [10212.835157] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [10212.845470] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10212.856121] ll_md_real_close+0xa5/0x2a0 [lustre] [10212.866620] ll_md_close+0x274/0x800 [lustre] [10212.880450] ll_file_release+0x128/0xc50 [lustre] [10212.882200] __fput+0x10d/0x440 [10212.890001] ____fput+0x12/0x20 [10212.891105] task_work_run+0xc6/0x110 [10212.892327] exit_to_usermode_loop+0x1dd/0x1f0 [10212.899604] do_syscall_64+0x3d6/0x440 [10212.909561] entry_SYSCALL_64_after_hwframe+0x49/0xae [10212.925586] RIP: 0033:0x7ff54af45cd8 [10212.929843] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10212.972097] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10212.982227] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [10212.990740] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [10213.001624] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [10213.019093] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [10213.022264] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10228.062176] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [10228.073604] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10228.127453] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10228.142687] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10228.152463] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10228.157846] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10228.177294] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10228.185947] RAX: 0000000000006d85 RBX: ffff96848811cae0 RCX: 0000000000000001 [10228.189415] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [10228.196225] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [10228.199803] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [10228.203141] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [10228.205920] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [10228.214535] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10228.219602] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [10228.222898] Call Trace: [10228.224298] [10228.225026] ? show_regs.cold.9+0x22/0x2f [10228.227581] ? watchdog_timer_fn.cold.6+0x62/0x10a [10228.229819] ? __hrtimer_run_queues+0x1c2/0x4d0 [10228.231686] ? watchdog+0x40/0x40 [10228.233253] ? hrtimer_interrupt+0x128/0x2c0 [10228.235018] ? smp_apic_timer_interrupt+0xa6/0x270 [10228.238784] ? apic_timer_interrupt+0xf/0x20 [10228.240676] [10228.241510] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10228.244645] do_raw_spin_lock+0x177/0x190 [10228.247005] _raw_spin_lock+0x21/0x30 [10228.248869] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10228.250767] ? do_raw_spin_unlock+0x75/0x190 [10228.259374] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10228.266014] ? do_wait_intr+0xf0/0xf0 [10228.268573] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10228.273907] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10228.276479] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10228.279946] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10228.282508] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10228.285315] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10228.287857] mdc_intent_lock+0x353/0x7e0 [mdc] [10228.289623] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10228.292338] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10228.295551] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10228.298045] lmv_intent_open+0x338/0x1150 [lmv] [10228.300047] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10228.302598] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10228.304973] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10228.307074] ll_intent_lock+0xc5/0xaa0 [lustre] [10228.309217] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10228.312280] ll_lookup_it+0xe0e/0x2270 [lustre] [10228.315216] ? do_raw_spin_unlock+0x75/0x190 [10228.319364] ? _raw_spin_unlock+0x12/0x30 [10228.321917] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10228.327905] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10228.334252] ll_atomic_open+0x36a/0xd70 [lustre] [10228.337876] lookup_open+0x356/0xb40 [10228.341294] do_last+0x313/0xfc0 [10228.343316] ? nd_jump_root+0xe5/0x160 [10228.344561] ? path_init+0x437/0x520 [10228.347916] path_openat+0xf7/0x500 [10228.350464] ? do_raw_spin_unlock+0x75/0x190 [10228.352014] do_filp_open+0x99/0x140 [10228.353107] ? getname_flags+0x6e/0x330 [10228.355395] ? __check_object_size+0xff/0x256 [10228.357410] ? do_raw_spin_unlock+0x75/0x190 [10228.361059] ? _raw_spin_unlock+0x12/0x30 [10228.362833] do_sys_openat2+0x2b4/0x410 [10228.370310] do_sys_open+0x73/0xa0 [10228.373012] __x64_sys_openat+0x24/0x30 [10228.375938] do_syscall_64+0xc1/0x440 [10228.379588] entry_SYSCALL_64_after_hwframe+0x49/0xae [10228.382490] RIP: 0033:0x7ff54af45332 [10228.384601] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10228.398630] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10228.403368] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [10228.407228] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [10228.410682] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [10228.414516] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [10228.417844] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [10232.055910] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [10232.060884] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10232.095903] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10232.103860] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10232.115518] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10232.122099] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10232.133837] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10232.147302] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [10232.152207] RDX: 0000000000002911 RSI: 0000000000000003 RDI: ffff96848811cae0 [10232.156383] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [10232.161176] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [10232.165562] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10232.173847] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [10232.180748] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10232.186185] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [10232.195264] Call Trace: [10232.197331] [10232.198725] ? show_regs.cold.9+0x22/0x2f [10232.202796] ? watchdog_timer_fn.cold.6+0x62/0x10a [10232.209700] ? __hrtimer_run_queues+0x1c2/0x4d0 [10232.219193] ? watchdog+0x40/0x40 [10232.221707] ? hrtimer_interrupt+0x128/0x2c0 [10232.224210] ? smp_apic_timer_interrupt+0xa6/0x270 [10232.229014] ? apic_timer_interrupt+0xf/0x20 [10232.232886] [10232.233926] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10232.236710] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10232.241187] do_raw_spin_lock+0x177/0x190 [10232.243827] _raw_spin_lock+0x21/0x30 [10232.245525] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10232.251285] obd_mod_free+0x46/0x140 [ptlrpc] [10232.253817] mdc_commit_open+0x91/0xb0 [mdc] [10232.259389] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10232.261563] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10232.265129] after_reply+0xe2c/0x1780 [ptlrpc] [10232.277805] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10232.285279] ptlrpcd+0xa82/0xe40 [ptlrpc] [10232.290277] ? do_wait_intr+0xf0/0xf0 [10232.299766] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10232.304791] kthread+0x1d1/0x200 [10232.307147] ? set_kthread_struct+0x70/0x70 [10232.311625] ret_from_fork+0x1f/0x30 [10236.384073] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8277s! [10236.395227] Showing busy workqueues and worker pools: [10236.403656] workqueue events: flags=0x0 [10236.410819] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10236.421125] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [10236.438099] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [10236.446659] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [10236.461822] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10236.471793] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [10236.487638] workqueue events_long: flags=0x0 [10236.494794] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10236.497098] pending: lu_objects_flush [obdclass] [10236.503764] workqueue events_unbound: flags=0x2 [10236.505809] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [10236.508142] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10236.512114] workqueue events_power_efficient: flags=0x80 [10236.516680] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10236.518677] pending: neigh_periodic_work [10236.520267] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10236.522541] pending: do_cache_clean, check_lifetime [10236.524347] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10236.526638] pending: neigh_periodic_work [10236.529034] workqueue writeback: flags=0x4a [10236.531109] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10236.533484] pending: wb_workfn, wb_workfn [10236.535161] workqueue xprtiod: flags=0xa [10236.536495] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10236.538803] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [10236.541584] workqueue ipv6_addrconf: flags=0x40008 [10236.543334] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10236.545763] pending: addrconf_verify_work [10236.547627] workqueue knbd0-recv: flags=0x1a [10236.552421] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10236.555297] in-flight: 657:recv_work [10236.556786] workqueue ptlrpc_pinger: flags=0x2 [10236.558754] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10236.560842] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [10236.563272] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8277s workers=4 idle: 20831 12642 28859 [10236.567321] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10226s workers=2 idle: 659 [10240.058360] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [10240.059163] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [10240.061436] Modules linked in: [10240.068666] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) [10240.076661] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10240.076855] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10240.076865] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10240.076869] RIP: 0010:native_safe_halt+0x2e/0x50 [10240.076888] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10240.076893] RSP: 0018:ffffaf9fc1543ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10240.076897] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [10240.076900] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c20b4154 [10240.076913] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [10240.076915] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [10240.077508] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [10240.077516] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [10240.077519] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10240.077521] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [10240.077531] Call Trace: [10240.077538] [10240.077546] ? show_regs.cold.9+0x22/0x2f [10240.077563] ? watchdog_timer_fn.cold.6+0x62/0x10a [10240.077571] ? __hrtimer_run_queues+0x1c2/0x4d0 [10240.077575] ? watchdog+0x40/0x40 [10240.077580] ? hrtimer_interrupt+0x128/0x2c0 [10240.092708] rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10240.092761] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10240.092773] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10240.092782] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10240.093346] RIP: 0010:native_safe_halt+0x2e/0x50 [10240.093358] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10240.093363] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10240.093367] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [10240.093370] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c2134154 [10240.093372] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [10240.093374] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [10240.093377] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [10240.093380] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [10240.093384] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10240.093387] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [10240.093396] Call Trace: [10240.093405] [10240.093412] ? show_regs.cold.9+0x22/0x2f [10240.093426] ? watchdog_timer_fn.cold.6+0x62/0x10a [10240.093704] ? __hrtimer_run_queues+0x1c2/0x4d0 [10240.093709] ? watchdog+0x40/0x40 [10240.093714] ? hrtimer_interrupt+0x128/0x2c0 [10240.093719] ? smp_apic_timer_interrupt+0xa6/0x270 [10240.093729] ? apic_timer_interrupt+0xf/0x20 [10240.093735] [10240.093739] ? native_safe_halt+0x2e/0x50 [10240.093743] kvm_wait+0xcd/0xe0 [10240.093753] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10240.093763] do_raw_spin_lock+0x177/0x190 [10240.093768] _raw_spin_lock+0x21/0x30 [10240.093772] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10240.094240] process_one_work+0x2c8/0x700 [10240.094250] worker_thread+0x296/0x6e0 [10240.094255] ? rescuer_thread+0x570/0x570 [10240.094259] kthread+0x1d1/0x200 [10240.094264] ? set_kthread_struct+0x70/0x70 [10240.094268] ret_from_fork+0x1f/0x30 [10240.396428] ? smp_apic_timer_interrupt+0xa6/0x270 [10240.398571] ? apic_timer_interrupt+0xf/0x20 [10240.400368] [10240.401518] ? native_safe_halt+0x2e/0x50 [10240.403266] kvm_wait+0xcd/0xe0 [10240.404559] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10240.406724] do_raw_spin_lock+0x177/0x190 [10240.408442] _raw_spin_lock+0x21/0x30 [10240.410317] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10240.414234] ? do_raw_spin_unlock+0x75/0x190 [10240.415419] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10240.419855] ? do_wait_intr+0xf0/0xf0 [10240.422335] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10240.426713] mdc_close+0x53b/0xd70 [mdc] [10240.430609] lmv_close+0x18d/0x490 [lmv] [10240.436214] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [10240.439756] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10240.444639] ll_md_real_close+0xa5/0x2a0 [lustre] [10240.445897] ll_md_close+0x274/0x800 [lustre] [10240.447026] ll_file_release+0x128/0xc50 [lustre] [10240.450522] __fput+0x10d/0x440 [10240.452928] ____fput+0x12/0x20 [10240.456805] task_work_run+0xc6/0x110 [10240.458909] exit_to_usermode_loop+0x1dd/0x1f0 [10240.461794] do_syscall_64+0x3d6/0x440 [10240.463868] entry_SYSCALL_64_after_hwframe+0x49/0xae [10240.468711] RIP: 0033:0x7ff54af45cd8 [10240.471273] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10240.481671] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10240.491423] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [10240.496351] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [10240.502828] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [10240.508698] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [10240.514396] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10256.060083] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [10256.063396] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10256.102340] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10256.130029] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10256.132227] RIP: 0010:native_safe_halt+0x2e/0x50 [10256.133697] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10256.138317] RSP: 0000:ffffaf9fc15234f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10256.140409] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [10256.142221] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c21b4154 [10256.145696] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [10256.149034] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [10256.152299] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [10256.155950] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [10256.159690] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10256.162606] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [10256.166048] Call Trace: [10256.167401] [10256.168250] ? show_regs.cold.9+0x22/0x2f [10256.175219] ? watchdog_timer_fn.cold.6+0x62/0x10a [10256.177427] ? __hrtimer_run_queues+0x1c2/0x4d0 [10256.183046] ? watchdog+0x40/0x40 [10256.186710] ? hrtimer_interrupt+0x128/0x2c0 [10256.189436] ? smp_apic_timer_interrupt+0xa6/0x270 [10256.193464] ? apic_timer_interrupt+0xf/0x20 [10256.196127] [10256.197587] ? native_safe_halt+0x2e/0x50 [10256.199753] kvm_wait+0xcd/0xe0 [10256.201708] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10256.203724] do_raw_spin_lock+0x177/0x190 [10256.205677] _raw_spin_lock+0x21/0x30 [10256.207438] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10256.210273] ? do_raw_spin_unlock+0x75/0x190 [10256.211898] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10256.214076] ? do_wait_intr+0xf0/0xf0 [10256.215444] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10256.217710] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10256.220108] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10256.222284] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10256.225833] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10256.232411] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10256.236544] mdc_intent_lock+0x353/0x7e0 [mdc] [10256.240076] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10256.243150] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10256.250509] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10256.252339] lmv_intent_open+0x338/0x1150 [lmv] [10256.261361] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10256.265568] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10256.274158] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10256.277575] ll_intent_lock+0xc5/0xaa0 [lustre] [10256.280324] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10256.286580] ll_lookup_it+0xe0e/0x2270 [lustre] [10256.289305] ? do_raw_spin_unlock+0x75/0x190 [10256.291591] ? _raw_spin_unlock+0x12/0x30 [10256.293578] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10256.297010] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10256.300114] ll_atomic_open+0x36a/0xd70 [lustre] [10256.304863] lookup_open+0x356/0xb40 [10256.310259] do_last+0x313/0xfc0 [10256.314180] ? nd_jump_root+0xe5/0x160 [10256.315800] ? path_init+0x437/0x520 [10256.316532] path_openat+0xf7/0x500 [10256.320580] ? do_raw_spin_unlock+0x75/0x190 [10256.325551] do_filp_open+0x99/0x140 [10256.327334] ? getname_flags+0x6e/0x330 [10256.331404] ? __check_object_size+0xff/0x256 [10256.333057] ? do_raw_spin_unlock+0x75/0x190 [10256.335108] ? _raw_spin_unlock+0x12/0x30 [10256.337155] do_sys_openat2+0x2b4/0x410 [10256.339031] do_sys_open+0x73/0xa0 [10256.340676] __x64_sys_openat+0x24/0x30 [10256.343351] do_syscall_64+0xc1/0x440 [10256.347212] entry_SYSCALL_64_after_hwframe+0x49/0xae [10256.352546] RIP: 0033:0x7ff54af45332 [10256.353795] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10256.366318] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10256.370102] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [10256.373812] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [10256.377147] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [10256.384158] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [10256.392860] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [10264.055088] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [10264.061336] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10264.090380] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10264.099746] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10264.106948] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10264.109364] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10264.121046] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10264.126987] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [10264.134437] RDX: 000000000000112f RSI: 0000000000000003 RDI: ffff96848811cae0 [10264.141005] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [10264.144892] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [10264.151329] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10264.159517] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [10264.164229] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10264.167288] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [10264.169841] Call Trace: [10264.170687] [10264.171469] ? show_regs.cold.9+0x22/0x2f [10264.172887] ? watchdog_timer_fn.cold.6+0x62/0x10a [10264.174619] ? __hrtimer_run_queues+0x1c2/0x4d0 [10264.176180] ? watchdog+0x40/0x40 [10264.181999] ? hrtimer_interrupt+0x128/0x2c0 [10264.185590] ? smp_apic_timer_interrupt+0xa6/0x270 [10264.190464] ? apic_timer_interrupt+0xf/0x20 [10264.194474] [10264.195902] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10264.201300] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10264.203673] do_raw_spin_lock+0x177/0x190 [10264.210690] _raw_spin_lock+0x21/0x30 [10264.211807] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10264.213434] obd_mod_free+0x46/0x140 [ptlrpc] [10264.215386] mdc_commit_open+0x91/0xb0 [mdc] [10264.216700] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10264.218652] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10264.220556] after_reply+0xe2c/0x1780 [ptlrpc] [10264.223554] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10264.226226] ptlrpcd+0xa82/0xe40 [ptlrpc] [10264.228073] ? do_wait_intr+0xf0/0xf0 [10264.229390] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10264.231264] kthread+0x1d1/0x200 [10264.232330] ? set_kthread_struct+0x70/0x70 [10264.233595] ret_from_fork+0x1f/0x30 [10266.594018] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8307s! [10266.605990] Showing busy workqueues and worker pools: [10266.609056] workqueue events: flags=0x0 [10266.617438] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10266.626323] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [10266.634622] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [10266.640465] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [10266.649603] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10266.654628] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [10266.659616] workqueue events_long: flags=0x0 [10266.661992] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10266.667730] pending: lu_objects_flush [obdclass] [10266.671432] workqueue events_unbound: flags=0x2 [10266.674040] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [10266.678153] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10266.685905] workqueue events_power_efficient: flags=0x80 [10266.689094] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10266.692209] pending: neigh_periodic_work [10266.694423] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10266.697844] pending: do_cache_clean, check_lifetime [10266.700468] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10266.703757] pending: neigh_periodic_work [10266.706940] workqueue writeback: flags=0x4a [10266.714960] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10266.721573] pending: wb_workfn, wb_workfn [10266.726991] workqueue xprtiod: flags=0xa [10266.731160] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10266.736726] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [10266.746225] workqueue ipv6_addrconf: flags=0x40008 [10266.748591] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10266.760273] pending: addrconf_verify_work [10266.764856] workqueue knbd0-recv: flags=0x1a [10266.768216] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10266.773291] in-flight: 657:recv_work [10266.777882] workqueue ptlrpc_pinger: flags=0x2 [10266.780359] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10266.783546] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [10266.786048] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8307s workers=4 idle: 20831 12642 28859 [10266.790902] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10257s workers=2 idle: 659 [10268.059175] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [10268.063051] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10268.071532] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [10268.088535] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10268.088556] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10268.088565] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10268.089494] RIP: 0010:native_safe_halt+0x2e/0x50 [10268.089510] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10268.089515] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10268.089520] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [10268.089522] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [10268.089525] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [10268.089527] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [10268.089529] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [10268.089533] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [10268.089536] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10268.089539] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [10268.089548] Call Trace: [10268.089556] [10268.089563] ? show_regs.cold.9+0x22/0x2f [10268.089575] ? watchdog_timer_fn.cold.6+0x62/0x10a [10268.089583] ? __hrtimer_run_queues+0x1c2/0x4d0 [10268.089588] ? watchdog+0x40/0x40 [10268.089592] ? hrtimer_interrupt+0x128/0x2c0 [10268.089596] ? smp_apic_timer_interrupt+0xa6/0x270 [10268.089606] ? apic_timer_interrupt+0xf/0x20 [10268.089610] [10268.089613] ? native_safe_halt+0x2e/0x50 [10268.089618] kvm_wait+0xcd/0xe0 [10268.089627] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10268.089636] do_raw_spin_lock+0x177/0x190 [10268.089641] _raw_spin_lock+0x21/0x30 [10268.089718] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10268.089975] process_one_work+0x2c8/0x700 [10268.089984] worker_thread+0x296/0x6e0 [10268.089989] ? rescuer_thread+0x570/0x570 [10268.089993] kthread+0x1d1/0x200 [10268.089999] ? set_kthread_struct+0x70/0x70 [10268.090003] ret_from_fork+0x1f/0x30 [10268.280130] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10268.316862] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10268.328420] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10268.335800] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10268.341083] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10268.362356] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10268.366936] RAX: 0000000000007e10 RBX: ffff96848811cae0 RCX: 0000000000000001 [10268.376286] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [10268.384975] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [10268.389423] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [10268.394678] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [10268.400107] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [10268.406738] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10268.420085] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [10268.429462] Call Trace: [10268.432164] [10268.432967] ? show_regs.cold.9+0x22/0x2f [10268.438144] ? watchdog_timer_fn.cold.6+0x62/0x10a [10268.443197] ? __hrtimer_run_queues+0x1c2/0x4d0 [10268.446371] ? watchdog+0x40/0x40 [10268.449126] ? hrtimer_interrupt+0x128/0x2c0 [10268.452087] ? smp_apic_timer_interrupt+0xa6/0x270 [10268.459022] ? apic_timer_interrupt+0xf/0x20 [10268.461814] [10268.462629] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10268.472483] do_raw_spin_lock+0x177/0x190 [10268.476553] _raw_spin_lock+0x21/0x30 [10268.479123] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10268.487317] ? do_raw_spin_unlock+0x75/0x190 [10268.490653] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10268.493518] ? do_wait_intr+0xf0/0xf0 [10268.496571] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10268.499750] mdc_close+0x53b/0xd70 [mdc] [10268.503806] lmv_close+0x18d/0x490 [lmv] [10268.506592] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [10268.511210] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10268.514735] ll_md_real_close+0xa5/0x2a0 [lustre] [10268.518994] ll_md_close+0x274/0x800 [lustre] [10268.522016] ll_file_release+0x128/0xc50 [lustre] [10268.534197] __fput+0x10d/0x440 [10268.535342] ____fput+0x12/0x20 [10268.537952] task_work_run+0xc6/0x110 [10268.539342] exit_to_usermode_loop+0x1dd/0x1f0 [10268.546527] do_syscall_64+0x3d6/0x440 [10268.551914] entry_SYSCALL_64_after_hwframe+0x49/0xae [10268.559969] RIP: 0033:0x7ff54af45cd8 [10268.562108] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10268.589506] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10268.596580] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [10268.603109] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [10268.610033] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [10268.614045] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [10268.618983] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10284.060046] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [10284.062135] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10284.075899] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10284.079560] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10284.082308] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10284.084352] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10284.090118] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10284.092304] RAX: 0000000000006d2b RBX: ffff96848811cae0 RCX: 0000000000000001 [10284.094189] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [10284.097879] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [10284.100349] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [10284.103592] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [10284.106153] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [10284.112308] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10284.122340] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [10284.126405] Call Trace: [10284.127422] [10284.128365] ? show_regs.cold.9+0x22/0x2f [10284.132232] ? watchdog_timer_fn.cold.6+0x62/0x10a [10284.134853] ? __hrtimer_run_queues+0x1c2/0x4d0 [10284.137254] ? watchdog+0x40/0x40 [10284.139691] ? hrtimer_interrupt+0x128/0x2c0 [10284.143152] ? smp_apic_timer_interrupt+0xa6/0x270 [10284.146144] ? apic_timer_interrupt+0xf/0x20 [10284.149288] [10284.151183] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10284.157756] do_raw_spin_lock+0x177/0x190 [10284.160098] _raw_spin_lock+0x21/0x30 [10284.161092] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10284.162633] ? do_raw_spin_unlock+0x75/0x190 [10284.170386] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10284.173427] ? do_wait_intr+0xf0/0xf0 [10284.176049] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10284.178868] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10284.181164] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10284.183014] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10284.184893] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10284.187008] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10284.189270] mdc_intent_lock+0x353/0x7e0 [mdc] [10284.192051] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10284.195037] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10284.198214] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10284.200873] lmv_intent_open+0x338/0x1150 [lmv] [10284.202633] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10284.204915] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10284.206542] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10284.209253] ll_intent_lock+0xc5/0xaa0 [lustre] [10284.211762] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10284.214754] ll_lookup_it+0xe0e/0x2270 [lustre] [10284.217200] ? do_raw_spin_unlock+0x75/0x190 [10284.219300] ? _raw_spin_unlock+0x12/0x30 [10284.222868] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10284.228016] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10284.230740] ll_atomic_open+0x36a/0xd70 [lustre] [10284.233514] lookup_open+0x356/0xb40 [10284.235405] do_last+0x313/0xfc0 [10284.237049] ? nd_jump_root+0xe5/0x160 [10284.238794] ? path_init+0x437/0x520 [10284.240382] path_openat+0xf7/0x500 [10284.242302] ? do_raw_spin_unlock+0x75/0x190 [10284.244442] do_filp_open+0x99/0x140 [10284.246355] ? getname_flags+0x6e/0x330 [10284.248307] ? __check_object_size+0xff/0x256 [10284.250393] ? do_raw_spin_unlock+0x75/0x190 [10284.252733] ? _raw_spin_unlock+0x12/0x30 [10284.254533] do_sys_openat2+0x2b4/0x410 [10284.256376] do_sys_open+0x73/0xa0 [10284.258024] __x64_sys_openat+0x24/0x30 [10284.259848] do_syscall_64+0xc1/0x440 [10284.261444] entry_SYSCALL_64_after_hwframe+0x49/0xae [10284.263910] RIP: 0033:0x7ff54af45332 [10284.265814] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10284.273979] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10284.277233] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [10284.280440] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [10284.283706] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [10284.287911] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [10284.290973] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [10292.055089] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [10292.059767] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10292.130542] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10292.137697] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10292.143111] RIP: 0010:native_safe_halt+0x2e/0x50 [10292.145893] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10292.157228] RSP: 0000:ffffaf9fc1b97bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10292.161596] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [10292.165682] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff96848811cae0 [10292.169897] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [10292.173959] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [10292.177726] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10292.181980] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [10292.187295] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10292.191451] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [10292.197126] Call Trace: [10292.198745] [10292.200271] ? show_regs.cold.9+0x22/0x2f [10292.203360] ? watchdog_timer_fn.cold.6+0x62/0x10a [10292.206566] ? __hrtimer_run_queues+0x1c2/0x4d0 [10292.210003] ? watchdog+0x40/0x40 [10292.212957] ? hrtimer_interrupt+0x128/0x2c0 [10292.216805] ? smp_apic_timer_interrupt+0xa6/0x270 [10292.220639] ? apic_timer_interrupt+0xf/0x20 [10292.224665] [10292.225831] ? native_safe_halt+0x2e/0x50 [10292.228126] kvm_wait+0xcd/0xe0 [10292.229415] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10292.232231] do_raw_spin_lock+0x177/0x190 [10292.234169] _raw_spin_lock+0x21/0x30 [10292.237075] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10292.240942] obd_mod_free+0x46/0x140 [ptlrpc] [10292.249022] mdc_commit_open+0x91/0xb0 [mdc] [10292.250595] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10292.258759] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10292.264292] after_reply+0xe2c/0x1780 [ptlrpc] [10292.267378] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10292.270900] ptlrpcd+0xa82/0xe40 [ptlrpc] [10292.273441] ? do_wait_intr+0xf0/0xf0 [10292.276258] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10292.279909] kthread+0x1d1/0x200 [10292.281925] ? set_kthread_struct+0x70/0x70 [10292.283763] ret_from_fork+0x1f/0x30 [10294.080014] rcu: INFO: rcu_sched self-detected stall on CPU [10294.082297] rcu: 0-...!: (4364388 ticks this GP) idle=98a/1/0x4000000000000002 softirq=125154/125154 fqs=959 [10294.085727] (t=8340240 jiffies g=253585 q=5531) [10294.087982] rcu: rcu_sched kthread starved for 8335200 jiffies! g253585 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [10294.091199] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [10294.094206] rcu: RCU grace-period kthread stack dump: [10294.097124] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [10294.100147] Call Trace: [10294.100934] __schedule+0x351/0xcb0 [10294.101980] schedule+0xc0/0x180 [10294.103433] schedule_timeout+0xb4/0x190 [10294.106417] ? do_raw_spin_unlock+0x75/0x190 [10294.108702] ? __next_timer_interrupt+0x160/0x160 [10294.111190] ? prepare_to_swait_event+0xf3/0x140 [10294.113159] rcu_gp_kthread+0x8dd/0x15b0 [10294.114810] ? force_qs_rnp+0x2c0/0x2c0 [10294.116655] kthread+0x1d1/0x200 [10294.118112] ? set_kthread_struct+0x70/0x70 [10294.119799] ret_from_fork+0x1f/0x30 [10294.122114] rcu: Stack dump where RCU GP kthread last ran: [10294.125383] Sending NMI from CPU 0 to CPUs 2: [10294.132075] NMI backtrace for cpu 2 [10294.132085] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10294.132088] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10294.132090] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10294.132093] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10294.132095] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10294.132098] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 [10294.132100] RAX: 0000000000007e5c RBX: ffff96848811cae0 RCX: 0000000000000001 [10294.132102] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [10294.132104] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [10294.132106] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [10294.132107] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [10294.132219] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [10294.132221] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10294.132223] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [10294.132224] Call Trace: [10294.132225] [10294.132227] ? show_regs.cold.9+0x22/0x2f [10294.132228] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [10294.132230] ? nmi_cpu_backtrace_handler+0x15/0x40 [10294.132231] ? nmi_handle+0xbd/0x210 [10294.132232] ? default_do_nmi+0x4b/0x1b0 [10294.132234] ? do_nmi+0x280/0x380 [10294.132235] ? end_repeat_nmi+0x16/0x1f [10294.132237] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10294.132238] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10294.132240] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10294.132241] [10294.132242] do_raw_spin_lock+0x177/0x190 [10294.132243] _raw_spin_lock+0x21/0x30 [10294.132245] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10294.132246] process_one_work+0x2c8/0x700 [10294.132248] worker_thread+0x296/0x6e0 [10294.132250] ? rescuer_thread+0x570/0x570 [10294.132251] kthread+0x1d1/0x200 [10294.132252] ? set_kthread_struct+0x70/0x70 [10294.132254] ret_from_fork+0x1f/0x30 [10294.132617] NMI backtrace for cpu 0 [10294.266282] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10294.274263] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10294.280789] Call Trace: [10294.281548] [10294.282216] ? dump_stack+0xbb/0x10e [10294.284116] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [10294.288805] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [10294.291638] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [10294.295683] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [10294.302804] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [10294.307841] ? rcu_dump_cpu_stacks+0x11b/0x171 [10294.311184] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [10294.316656] ? update_process_times+0x6d/0xc0 [10294.318432] ? tick_sched_handle.isra.18+0x27/0x90 [10294.320035] ? tick_sched_timer+0x53/0xb0 [10294.323197] ? __hrtimer_run_queues+0x1c2/0x4d0 [10294.325883] ? tick_sched_handle.isra.18+0x90/0x90 [10294.328295] ? hrtimer_interrupt+0x128/0x2c0 [10294.330356] ? smp_apic_timer_interrupt+0xa6/0x270 [10294.332607] ? apic_timer_interrupt+0xf/0x20 [10294.335450] [10294.336228] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10294.338120] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10294.343047] ? do_raw_spin_lock+0x177/0x190 [10294.345104] ? _raw_spin_lock+0x21/0x30 [10294.348861] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10294.352127] ? obd_mod_free+0x46/0x140 [ptlrpc] [10294.357162] ? mdc_commit_open+0x91/0xb0 [mdc] [10294.359492] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10294.362579] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10294.367939] ? after_reply+0xe2c/0x1780 [ptlrpc] [10294.370492] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10294.374272] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [10294.376016] ? do_wait_intr+0xf0/0xf0 [10294.379342] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10294.382324] ? kthread+0x1d1/0x200 [10294.383815] ? set_kthread_struct+0x70/0x70 [10294.385645] ? ret_from_fork+0x1f/0x30 [10296.058655] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [10296.062198] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10296.094060] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10296.103531] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10296.109320] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10296.116244] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10296.129530] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [10296.134343] RAX: 0000000000000885 RBX: ffff96848811cae0 RCX: 0000000000000001 [10296.141976] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [10296.144411] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [10296.152031] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [10296.155346] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [10296.159098] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [10296.167209] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10296.178305] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [10296.185227] Call Trace: [10296.188178] [10296.189165] ? show_regs.cold.9+0x22/0x2f [10296.190652] ? watchdog_timer_fn.cold.6+0x62/0x10a [10296.195724] ? __hrtimer_run_queues+0x1c2/0x4d0 [10296.198920] ? watchdog+0x40/0x40 [10296.200347] ? hrtimer_interrupt+0x128/0x2c0 [10296.202271] ? smp_apic_timer_interrupt+0xa6/0x270 [10296.204235] ? apic_timer_interrupt+0xf/0x20 [10296.206843] [10296.209617] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10296.212766] do_raw_spin_lock+0x177/0x190 [10296.214531] _raw_spin_lock+0x21/0x30 [10296.217776] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10296.219665] ? do_raw_spin_unlock+0x75/0x190 [10296.221072] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10296.223820] ? do_wait_intr+0xf0/0xf0 [10296.226839] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10296.232874] mdc_close+0x53b/0xd70 [mdc] [10296.238065] lmv_close+0x18d/0x490 [lmv] [10296.240624] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [10296.244900] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10296.248169] ll_md_real_close+0xa5/0x2a0 [lustre] [10296.250699] ll_md_close+0x274/0x800 [lustre] [10296.256104] ll_file_release+0x128/0xc50 [lustre] [10296.260342] __fput+0x10d/0x440 [10296.263493] ____fput+0x12/0x20 [10296.265989] task_work_run+0xc6/0x110 [10296.270176] exit_to_usermode_loop+0x1dd/0x1f0 [10296.274228] do_syscall_64+0x3d6/0x440 [10296.276690] entry_SYSCALL_64_after_hwframe+0x49/0xae [10296.281168] RIP: 0033:0x7ff54af45cd8 [10296.282593] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10296.293647] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10296.296897] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [10296.299915] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [10296.303018] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [10296.306048] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [10296.308145] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10296.799135] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8337s! [10296.807362] Showing busy workqueues and worker pools: [10296.814885] workqueue events: flags=0x0 [10296.823771] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10296.829891] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [10296.839525] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [10296.847829] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [10296.860622] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10296.866538] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [10296.874716] workqueue events_long: flags=0x0 [10296.878069] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10296.882725] pending: lu_objects_flush [obdclass] [10296.894600] workqueue events_unbound: flags=0x2 [10296.897941] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [10296.904175] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10296.912105] workqueue events_power_efficient: flags=0x80 [10296.919336] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10296.922814] pending: neigh_periodic_work [10296.925474] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10296.929590] pending: do_cache_clean, check_lifetime [10296.932902] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10296.936843] pending: neigh_periodic_work [10296.939591] workqueue writeback: flags=0x4a [10296.941697] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10296.945197] pending: wb_workfn, wb_workfn [10296.947779] workqueue xprtiod: flags=0xa [10296.950001] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10296.954771] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [10296.959531] workqueue ipv6_addrconf: flags=0x40008 [10296.962467] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10296.966986] pending: addrconf_verify_work [10296.970497] workqueue knbd0-recv: flags=0x1a [10296.972281] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10296.975236] in-flight: 657:recv_work [10296.977913] workqueue ptlrpc_pinger: flags=0x2 [10296.981200] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10296.985600] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [10296.990674] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8338s workers=4 idle: 20831 12642 28859 [10296.996041] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10287s workers=2 idle: 659 [10312.060020] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [10312.064545] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10312.088798] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10312.094982] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10312.101731] RIP: 0010:native_safe_halt+0x2e/0x50 [10312.105844] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10312.129811] RSP: 0000:ffffaf9fc15234f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10312.134959] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [10312.138985] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [10312.143440] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [10312.146904] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [10312.153460] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [10312.157962] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [10312.163212] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10312.165934] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [10312.169894] Call Trace: [10312.171200] [10312.172427] ? show_regs.cold.9+0x22/0x2f [10312.174523] ? watchdog_timer_fn.cold.6+0x62/0x10a [10312.176953] ? __hrtimer_run_queues+0x1c2/0x4d0 [10312.179568] ? watchdog+0x40/0x40 [10312.184970] ? hrtimer_interrupt+0x128/0x2c0 [10312.187648] ? smp_apic_timer_interrupt+0xa6/0x270 [10312.191108] ? apic_timer_interrupt+0xf/0x20 [10312.195554] [10312.196710] ? native_safe_halt+0x2e/0x50 [10312.199926] kvm_wait+0xcd/0xe0 [10312.204365] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10312.209681] do_raw_spin_lock+0x177/0x190 [10312.213027] _raw_spin_lock+0x21/0x30 [10312.215003] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10312.219796] ? do_raw_spin_unlock+0x75/0x190 [10312.221202] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10312.223213] ? do_wait_intr+0xf0/0xf0 [10312.225699] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10312.228238] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10312.230356] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10312.233525] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10312.238847] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10312.242188] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10312.244144] mdc_intent_lock+0x353/0x7e0 [mdc] [10312.247526] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10312.250927] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10312.257145] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10312.261889] lmv_intent_open+0x338/0x1150 [lmv] [10312.264543] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10312.267545] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10312.270354] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10312.272472] ll_intent_lock+0xc5/0xaa0 [lustre] [10312.275162] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10312.285764] ll_lookup_it+0xe0e/0x2270 [lustre] [10312.289764] ? do_raw_spin_unlock+0x75/0x190 [10312.291509] ? _raw_spin_unlock+0x12/0x30 [10312.292649] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10312.294809] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10312.299399] ll_atomic_open+0x36a/0xd70 [lustre] [10312.303541] lookup_open+0x356/0xb40 [10312.307780] do_last+0x313/0xfc0 [10312.309620] ? nd_jump_root+0xe5/0x160 [10312.314750] ? path_init+0x437/0x520 [10312.318402] path_openat+0xf7/0x500 [10312.321982] ? do_raw_spin_unlock+0x75/0x190 [10312.324840] do_filp_open+0x99/0x140 [10312.326959] ? getname_flags+0x6e/0x330 [10312.329460] ? __check_object_size+0xff/0x256 [10312.332737] ? do_raw_spin_unlock+0x75/0x190 [10312.334201] ? _raw_spin_unlock+0x12/0x30 [10312.335099] do_sys_openat2+0x2b4/0x410 [10312.335959] do_sys_open+0x73/0xa0 [10312.339008] __x64_sys_openat+0x24/0x30 [10312.341491] do_syscall_64+0xc1/0x440 [10312.343978] entry_SYSCALL_64_after_hwframe+0x49/0xae [10312.349534] RIP: 0033:0x7ff54af45332 [10312.360470] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10312.380813] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10312.384486] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [10312.388489] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [10312.391730] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [10312.398451] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [10312.404832] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [10320.059030] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [10320.068740] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10320.092725] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10320.097471] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10320.104588] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10320.108141] RIP: 0010:native_safe_halt+0x2e/0x50 [10320.110005] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10320.118961] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10320.122252] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [10320.125768] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c2134154 [10320.128540] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [10320.131882] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [10320.135386] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [10320.138626] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [10320.142995] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10320.145339] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [10320.148055] Call Trace: [10320.149437] [10320.150372] ? show_regs.cold.9+0x22/0x2f [10320.152891] ? watchdog_timer_fn.cold.6+0x62/0x10a [10320.155518] ? __hrtimer_run_queues+0x1c2/0x4d0 [10320.157471] ? watchdog+0x40/0x40 [10320.158901] ? hrtimer_interrupt+0x128/0x2c0 [10320.161367] ? smp_apic_timer_interrupt+0xa6/0x270 [10320.163833] ? apic_timer_interrupt+0xf/0x20 [10320.166128] [10320.167231] ? native_safe_halt+0x2e/0x50 [10320.168860] kvm_wait+0xcd/0xe0 [10320.170459] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10320.175659] do_raw_spin_lock+0x177/0x190 [10320.176800] _raw_spin_lock+0x21/0x30 [10320.177942] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10320.180453] process_one_work+0x2c8/0x700 [10320.182121] worker_thread+0x296/0x6e0 [10320.183685] ? rescuer_thread+0x570/0x570 [10320.185528] kthread+0x1d1/0x200 [10320.187425] ? set_kthread_struct+0x70/0x70 [10320.188804] ret_from_fork+0x1f/0x30 [10324.057425] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [10324.058406] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [10324.061463] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10324.061547] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10324.061556] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10324.061559] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10324.061618] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10324.061622] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10324.061627] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [10324.061629] RDX: 000000000000000f RSI: 0000000000000003 RDI: ffff96848811cae0 [10324.061632] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [10324.061634] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [10324.061637] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10324.061640] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [10324.061642] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10324.061645] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [10324.061654] Call Trace: [10324.061671] [10324.061678] ? show_regs.cold.9+0x22/0x2f [10324.061688] ? watchdog_timer_fn.cold.6+0x62/0x10a [10324.061694] ? __hrtimer_run_queues+0x1c2/0x4d0 [10324.061699] ? watchdog+0x40/0x40 [10324.061704] ? hrtimer_interrupt+0x128/0x2c0 [10324.061708] ? smp_apic_timer_interrupt+0xa6/0x270 [10324.061716] ? apic_timer_interrupt+0xf/0x20 [10324.061721] [10324.061725] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10324.061730] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10324.061734] do_raw_spin_lock+0x177/0x190 [10324.061739] _raw_spin_lock+0x21/0x30 [10324.061745] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10324.062024] obd_mod_free+0x46/0x140 [ptlrpc] [10324.066448] Modules linked in: lustre(O) [10324.101396] mdc_commit_open+0x91/0xb0 [mdc] [10324.101446] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10324.101729] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10324.101936] after_reply+0xe2c/0x1780 [ptlrpc] [10324.109491] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10324.109563] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10324.109570] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10324.109573] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10324.109587] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10324.109591] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10324.109596] RAX: 000000000000798f RBX: ffff96848811cae0 RCX: 0000000000000001 [10324.109599] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [10324.109601] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [10324.109603] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [10324.109606] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [10324.109609] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [10324.109612] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10324.109614] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [10324.109623] Call Trace: [10324.109629] [10324.109636] ? show_regs.cold.9+0x22/0x2f [10324.109646] ? watchdog_timer_fn.cold.6+0x62/0x10a [10324.109653] ? __hrtimer_run_queues+0x1c2/0x4d0 [10324.109657] ? watchdog+0x40/0x40 [10324.109660] ? hrtimer_interrupt+0x128/0x2c0 [10324.109664] ? smp_apic_timer_interrupt+0xa6/0x270 [10324.109671] ? apic_timer_interrupt+0xf/0x20 [10324.109675] [10324.109677] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10324.109683] do_raw_spin_lock+0x177/0x190 [10324.109688] _raw_spin_lock+0x21/0x30 [10324.109693] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10324.109960] ? do_raw_spin_unlock+0x75/0x190 [10324.109966] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10324.110337] ? do_wait_intr+0xf0/0xf0 [10324.110344] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10324.110522] mdc_close+0x53b/0xd70 [mdc] [10324.110558] lmv_close+0x18d/0x490 [lmv] [10324.110589] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [10324.110745] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10324.110887] ll_md_real_close+0xa5/0x2a0 [lustre] [10324.110968] ll_md_close+0x274/0x800 [lustre] [10324.111437] ll_file_release+0x128/0xc50 [lustre] [10324.111490] __fput+0x10d/0x440 [10324.111497] ____fput+0x12/0x20 [10324.111500] task_work_run+0xc6/0x110 [10324.111506] exit_to_usermode_loop+0x1dd/0x1f0 [10324.111511] do_syscall_64+0x3d6/0x440 [10324.111514] entry_SYSCALL_64_after_hwframe+0x49/0xae [10324.111519] RIP: 0033:0x7ff54af45cd8 [10324.111525] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10324.111528] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10324.111531] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [10324.111534] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [10324.111536] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [10324.111538] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [10324.111541] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10324.584526] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10324.589859] ptlrpcd+0xa82/0xe40 [ptlrpc] [10324.592447] ? do_wait_intr+0xf0/0xf0 [10324.595539] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10324.599683] kthread+0x1d1/0x200 [10324.601472] ? set_kthread_struct+0x70/0x70 [10324.606101] ret_from_fork+0x1f/0x30 [10327.007216] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8368s! [10327.012822] Showing busy workqueues and worker pools: [10327.014774] workqueue events: flags=0x0 [10327.018425] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10327.023911] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [10327.032511] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [10327.041365] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [10327.050917] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10327.056479] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [10327.065437] workqueue events_long: flags=0x0 [10327.067093] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10327.075290] pending: lu_objects_flush [obdclass] [10327.078951] workqueue events_unbound: flags=0x2 [10327.081316] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [10327.089986] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10327.099445] workqueue events_power_efficient: flags=0x80 [10327.102419] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10327.107913] pending: neigh_periodic_work [10327.110732] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10327.120674] pending: do_cache_clean, check_lifetime [10327.128550] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10327.134002] pending: neigh_periodic_work [10327.139749] workqueue writeback: flags=0x4a [10327.144450] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10327.151968] pending: wb_workfn, wb_workfn [10327.154541] workqueue xprtiod: flags=0xa [10327.165327] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10327.171054] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [10327.179675] workqueue ipv6_addrconf: flags=0x40008 [10327.182372] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10327.188811] pending: addrconf_verify_work [10327.199445] workqueue knbd0-recv: flags=0x1a [10327.205257] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10327.209631] in-flight: 657:recv_work [10327.215580] workqueue ptlrpc_pinger: flags=0x2 [10327.222066] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10327.224992] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [10327.230385] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8368s workers=4 idle: 20831 12642 28859 [10327.234578] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10317s workers=2 idle: 659 [10340.060030] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [10340.064565] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10340.096526] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10340.101620] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10340.105975] RIP: 0010:native_safe_halt+0x2e/0x50 [10340.108266] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10340.116671] RSP: 0000:ffffaf9fc15234f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10340.120077] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [10340.123842] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c21b4154 [10340.126705] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [10340.130106] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [10340.134047] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [10340.137129] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [10340.141117] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10340.143983] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [10340.147584] Call Trace: [10340.148611] [10340.149613] ? show_regs.cold.9+0x22/0x2f [10340.151535] ? watchdog_timer_fn.cold.6+0x62/0x10a [10340.154596] ? __hrtimer_run_queues+0x1c2/0x4d0 [10340.156437] ? watchdog+0x40/0x40 [10340.157839] ? hrtimer_interrupt+0x128/0x2c0 [10340.159534] ? smp_apic_timer_interrupt+0xa6/0x270 [10340.161969] ? apic_timer_interrupt+0xf/0x20 [10340.165000] [10340.165898] ? native_safe_halt+0x2e/0x50 [10340.167552] kvm_wait+0xcd/0xe0 [10340.169442] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10340.171990] do_raw_spin_lock+0x177/0x190 [10340.173772] _raw_spin_lock+0x21/0x30 [10340.175610] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10340.178592] ? do_raw_spin_unlock+0x75/0x190 [10340.180286] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10340.182811] ? do_wait_intr+0xf0/0xf0 [10340.184473] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10340.187516] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10340.190440] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10340.192744] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10340.194715] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10340.197309] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10340.199752] mdc_intent_lock+0x353/0x7e0 [mdc] [10340.201650] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10340.204489] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10340.207626] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10340.209863] lmv_intent_open+0x338/0x1150 [lmv] [10340.212460] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10340.215185] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10340.217534] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10340.221150] ll_intent_lock+0xc5/0xaa0 [lustre] [10340.223777] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10340.226507] ll_lookup_it+0xe0e/0x2270 [lustre] [10340.228897] ? do_raw_spin_unlock+0x75/0x190 [10340.230867] ? _raw_spin_unlock+0x12/0x30 [10340.232838] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10340.235655] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10340.238437] ll_atomic_open+0x36a/0xd70 [lustre] [10340.241182] lookup_open+0x356/0xb40 [10340.243451] do_last+0x313/0xfc0 [10340.245093] ? nd_jump_root+0xe5/0x160 [10340.247070] ? path_init+0x437/0x520 [10340.248761] path_openat+0xf7/0x500 [10340.250208] ? do_raw_spin_unlock+0x75/0x190 [10340.252717] do_filp_open+0x99/0x140 [10340.256027] ? getname_flags+0x6e/0x330 [10340.258001] ? __check_object_size+0xff/0x256 [10340.260385] ? do_raw_spin_unlock+0x75/0x190 [10340.262551] ? _raw_spin_unlock+0x12/0x30 [10340.264360] do_sys_openat2+0x2b4/0x410 [10340.266096] do_sys_open+0x73/0xa0 [10340.267877] __x64_sys_openat+0x24/0x30 [10340.269518] do_syscall_64+0xc1/0x440 [10340.271444] entry_SYSCALL_64_after_hwframe+0x49/0xae [10340.274009] RIP: 0033:0x7ff54af45332 [10340.277919] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10340.288650] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10340.291612] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [10340.295913] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [10340.299816] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [10340.303298] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [10340.306579] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [10348.059627] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [10348.065209] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10348.112684] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10348.120827] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10348.125407] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10348.131511] RIP: 0010:native_safe_halt+0x2e/0x50 [10348.134772] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10348.150402] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10348.156836] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [10348.163750] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c2134154 [10348.168628] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [10348.175150] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [10348.184649] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [10348.191563] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [10348.200903] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10348.218334] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [10348.220885] Call Trace: [10348.230668] [10348.233339] ? show_regs.cold.9+0x22/0x2f [10348.243090] ? watchdog_timer_fn.cold.6+0x62/0x10a [10348.246817] ? __hrtimer_run_queues+0x1c2/0x4d0 [10348.251132] ? watchdog+0x40/0x40 [10348.253691] ? hrtimer_interrupt+0x128/0x2c0 [10348.259134] ? smp_apic_timer_interrupt+0xa6/0x270 [10348.262408] ? apic_timer_interrupt+0xf/0x20 [10348.274525] [10348.275119] ? native_safe_halt+0x2e/0x50 [10348.276497] kvm_wait+0xcd/0xe0 [10348.295071] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10348.298388] do_raw_spin_lock+0x177/0x190 [10348.305109] _raw_spin_lock+0x21/0x30 [10348.306920] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10348.310930] process_one_work+0x2c8/0x700 [10348.313426] worker_thread+0x296/0x6e0 [10348.315082] ? rescuer_thread+0x570/0x570 [10348.317478] kthread+0x1d1/0x200 [10348.318828] ? set_kthread_struct+0x70/0x70 [10348.321380] ret_from_fork+0x1f/0x30 [10352.055077] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [10352.057681] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10352.059548] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [10352.072540] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10352.072556] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10352.072560] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10352.072577] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10352.072582] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10352.072588] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [10352.072590] RDX: 0000000000001a38 RSI: 0000000000000003 RDI: ffff96848811cae0 [10352.072593] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [10352.072595] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [10352.072598] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10352.072602] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [10352.072605] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10352.072610] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [10352.072618] Call Trace: [10352.072623] [10352.072630] ? show_regs.cold.9+0x22/0x2f [10352.072641] ? watchdog_timer_fn.cold.6+0x62/0x10a [10352.072648] ? __hrtimer_run_queues+0x1c2/0x4d0 [10352.072652] ? watchdog+0x40/0x40 [10352.072656] ? hrtimer_interrupt+0x128/0x2c0 [10352.072660] ? smp_apic_timer_interrupt+0xa6/0x270 [10352.072670] ? apic_timer_interrupt+0xf/0x20 [10352.072674] [10352.072676] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10352.072681] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10352.072686] do_raw_spin_lock+0x177/0x190 [10352.072690] _raw_spin_lock+0x21/0x30 [10352.072700] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10352.073097] obd_mod_free+0x46/0x140 [ptlrpc] [10352.073315] mdc_commit_open+0x91/0xb0 [mdc] [10352.073351] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10352.073610] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10352.073821] after_reply+0xe2c/0x1780 [ptlrpc] [10352.074178] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10352.074432] ptlrpcd+0xa82/0xe40 [ptlrpc] [10352.074632] ? do_wait_intr+0xf0/0xf0 [10352.074643] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10352.074848] kthread+0x1d1/0x200 [10352.074855] ? set_kthread_struct+0x70/0x70 [10352.074859] ret_from_fork+0x1f/0x30 [10352.277317] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10352.314343] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10352.322154] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10352.330338] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10352.336070] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10352.346817] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [10352.352127] RAX: 000000000000780d RBX: ffff96848811cae0 RCX: 0000000000000001 [10352.365158] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [10352.373093] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [10352.383134] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [10352.388816] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [10352.392670] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [10352.399143] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10352.403253] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [10352.411672] Call Trace: [10352.413740] [10352.414801] ? show_regs.cold.9+0x22/0x2f [10352.417773] ? watchdog_timer_fn.cold.6+0x62/0x10a [10352.422522] ? __hrtimer_run_queues+0x1c2/0x4d0 [10352.424909] ? watchdog+0x40/0x40 [10352.427174] ? hrtimer_interrupt+0x128/0x2c0 [10352.429984] ? smp_apic_timer_interrupt+0xa6/0x270 [10352.433303] ? apic_timer_interrupt+0xf/0x20 [10352.436213] [10352.438720] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10352.441318] do_raw_spin_lock+0x177/0x190 [10352.443744] _raw_spin_lock+0x21/0x30 [10352.445999] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10352.448931] ? do_raw_spin_unlock+0x75/0x190 [10352.451569] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10352.454534] ? do_wait_intr+0xf0/0xf0 [10352.456785] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10352.460158] mdc_close+0x53b/0xd70 [mdc] [10352.462714] lmv_close+0x18d/0x490 [lmv] [10352.465082] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [10352.468026] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10352.471041] ll_md_real_close+0xa5/0x2a0 [lustre] [10352.473296] ll_md_close+0x274/0x800 [lustre] [10352.475821] ll_file_release+0x128/0xc50 [lustre] [10352.478578] __fput+0x10d/0x440 [10352.480022] ____fput+0x12/0x20 [10352.481619] task_work_run+0xc6/0x110 [10352.484034] exit_to_usermode_loop+0x1dd/0x1f0 [10352.486454] do_syscall_64+0x3d6/0x440 [10352.488548] entry_SYSCALL_64_after_hwframe+0x49/0xae [10352.490896] RIP: 0033:0x7ff54af45cd8 [10352.492747] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10352.501444] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10352.505437] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [10352.509224] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [10352.513286] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [10352.518768] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [10352.524778] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10357.727575] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8398s! [10357.733222] Showing busy workqueues and worker pools: [10357.735339] workqueue events: flags=0x0 [10357.737220] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10357.740523] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [10357.744828] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [10357.748099] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [10357.753139] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10357.755400] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [10357.758128] workqueue events_long: flags=0x0 [10357.759927] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10357.762738] pending: lu_objects_flush [obdclass] [10357.764653] workqueue events_unbound: flags=0x2 [10357.766256] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [10357.768449] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10357.772702] workqueue events_power_efficient: flags=0x80 [10357.774859] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10357.779482] pending: neigh_periodic_work [10357.781420] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10357.787786] pending: do_cache_clean, check_lifetime [10357.790009] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10357.793718] pending: neigh_periodic_work [10357.796915] workqueue writeback: flags=0x4a [10357.798549] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10357.801471] pending: wb_workfn, wb_workfn [10357.803420] workqueue xprtiod: flags=0xa [10357.807168] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10357.809932] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [10357.815072] workqueue ipv6_addrconf: flags=0x40008 [10357.817014] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10357.819597] pending: addrconf_verify_work [10357.821560] workqueue knbd0-recv: flags=0x1a [10357.833920] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10357.838055] in-flight: 657:recv_work [10357.841015] workqueue ptlrpc_pinger: flags=0x2 [10357.843689] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10357.848556] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [10357.851941] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8398s workers=4 idle: 20831 12642 28859 [10357.858048] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10348s workers=2 idle: 659 [10368.060696] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [10368.076124] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10368.112068] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10368.119671] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10368.123920] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10368.126419] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10368.155845] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10368.164259] RAX: 0000000000007830 RBX: ffff96848811cae0 RCX: 0000000000000001 [10368.172474] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [10368.174916] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [10368.191782] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [10368.199699] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [10368.205771] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [10368.216128] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10368.223427] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [10368.233278] Call Trace: [10368.233967] [10368.234482] ? show_regs.cold.9+0x22/0x2f [10368.235598] ? watchdog_timer_fn.cold.6+0x62/0x10a [10368.242595] ? __hrtimer_run_queues+0x1c2/0x4d0 [10368.245611] ? watchdog+0x40/0x40 [10368.255659] ? hrtimer_interrupt+0x128/0x2c0 [10368.257091] ? smp_apic_timer_interrupt+0xa6/0x270 [10368.262563] ? apic_timer_interrupt+0xf/0x20 [10368.267968] [10368.270625] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10368.275876] do_raw_spin_lock+0x177/0x190 [10368.279980] _raw_spin_lock+0x21/0x30 [10368.283822] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10368.292622] ? do_raw_spin_unlock+0x75/0x190 [10368.297581] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10368.301487] ? do_wait_intr+0xf0/0xf0 [10368.305145] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10368.312605] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10368.323540] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10368.331961] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10368.333580] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10368.343887] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10368.352784] mdc_intent_lock+0x353/0x7e0 [mdc] [10368.359469] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10368.364454] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10368.370298] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10368.375232] lmv_intent_open+0x338/0x1150 [lmv] [10368.378410] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10368.383359] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10368.386368] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10368.391523] ll_intent_lock+0xc5/0xaa0 [lustre] [10368.396304] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10368.399629] ll_lookup_it+0xe0e/0x2270 [lustre] [10368.401424] ? do_raw_spin_unlock+0x75/0x190 [10368.404831] ? _raw_spin_unlock+0x12/0x30 [10368.406238] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10368.416498] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10368.423584] ll_atomic_open+0x36a/0xd70 [lustre] [10368.428538] lookup_open+0x356/0xb40 [10368.432102] do_last+0x313/0xfc0 [10368.433944] ? nd_jump_root+0xe5/0x160 [10368.437331] ? path_init+0x437/0x520 [10368.439995] path_openat+0xf7/0x500 [10368.444393] ? do_raw_spin_unlock+0x75/0x190 [10368.447607] do_filp_open+0x99/0x140 [10368.451556] ? getname_flags+0x6e/0x330 [10368.453474] ? __check_object_size+0xff/0x256 [10368.454769] ? do_raw_spin_unlock+0x75/0x190 [10368.458456] ? _raw_spin_unlock+0x12/0x30 [10368.462348] do_sys_openat2+0x2b4/0x410 [10368.465164] do_sys_open+0x73/0xa0 [10368.468896] __x64_sys_openat+0x24/0x30 [10368.470662] do_syscall_64+0xc1/0x440 [10368.472584] entry_SYSCALL_64_after_hwframe+0x49/0xae [10368.476829] RIP: 0033:0x7ff54af45332 [10368.478658] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10368.493416] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10368.500444] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [10368.506765] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [10368.510416] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [10368.513501] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [10368.518199] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [10376.059036] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [10376.062684] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10376.086872] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10376.091712] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10376.095549] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10376.097905] RIP: 0010:native_safe_halt+0x2e/0x50 [10376.099314] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10376.107547] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10376.111707] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [10376.114870] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [10376.118598] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [10376.122942] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [10376.126996] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [10376.130519] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [10376.135606] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10376.138127] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [10376.142029] Call Trace: [10376.143221] [10376.144288] ? show_regs.cold.9+0x22/0x2f [10376.146048] ? watchdog_timer_fn.cold.6+0x62/0x10a [10376.148266] ? __hrtimer_run_queues+0x1c2/0x4d0 [10376.150296] ? watchdog+0x40/0x40 [10376.152136] ? hrtimer_interrupt+0x128/0x2c0 [10376.154008] ? smp_apic_timer_interrupt+0xa6/0x270 [10376.156470] ? apic_timer_interrupt+0xf/0x20 [10376.158842] [10376.159918] ? native_safe_halt+0x2e/0x50 [10376.162022] kvm_wait+0xcd/0xe0 [10376.163476] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10376.166468] do_raw_spin_lock+0x177/0x190 [10376.171168] _raw_spin_lock+0x21/0x30 [10376.172993] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10376.175765] process_one_work+0x2c8/0x700 [10376.178122] worker_thread+0x296/0x6e0 [10376.180190] ? rescuer_thread+0x570/0x570 [10376.181965] kthread+0x1d1/0x200 [10376.183444] ? set_kthread_struct+0x70/0x70 [10376.185310] ret_from_fork+0x1f/0x30 [10380.058036] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [10380.062359] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10380.123824] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10380.131383] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10380.138373] RIP: 0010:native_safe_halt+0x2e/0x50 [10380.145179] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10380.161472] RSP: 0018:ffffaf9fc1543ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10380.168451] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [10380.177071] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c20b4154 [10380.181834] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [10380.188160] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [10380.191485] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [10380.194924] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [10380.198853] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10380.201304] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [10380.204266] Call Trace: [10380.205300] [10380.206114] ? show_regs.cold.9+0x22/0x2f [10380.207995] ? watchdog_timer_fn.cold.6+0x62/0x10a [10380.209938] ? __hrtimer_run_queues+0x1c2/0x4d0 [10380.211553] ? watchdog+0x40/0x40 [10380.213058] ? hrtimer_interrupt+0x128/0x2c0 [10380.214847] ? smp_apic_timer_interrupt+0xa6/0x270 [10380.216911] ? apic_timer_interrupt+0xf/0x20 [10380.218776] [10380.219620] ? native_safe_halt+0x2e/0x50 [10380.221338] kvm_wait+0xcd/0xe0 [10380.222479] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10380.224206] do_raw_spin_lock+0x177/0x190 [10380.225805] _raw_spin_lock+0x21/0x30 [10380.227162] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10380.229446] ? do_raw_spin_unlock+0x75/0x190 [10380.231152] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10380.233241] ? do_wait_intr+0xf0/0xf0 [10380.234956] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10380.237450] mdc_close+0x53b/0xd70 [mdc] [10380.239240] lmv_close+0x18d/0x490 [lmv] [10380.241163] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [10380.243989] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10380.247038] ll_md_real_close+0xa5/0x2a0 [lustre] [10380.250268] ll_md_close+0x274/0x800 [lustre] [10380.252634] ll_file_release+0x128/0xc50 [lustre] [10380.254921] __fput+0x10d/0x440 [10380.257044] ____fput+0x12/0x20 [10380.258904] task_work_run+0xc6/0x110 [10380.260281] exit_to_usermode_loop+0x1dd/0x1f0 [10380.265699] do_syscall_64+0x3d6/0x440 [10380.267016] entry_SYSCALL_64_after_hwframe+0x49/0xae [10380.269524] RIP: 0033:0x7ff54af45cd8 [10380.271988] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10380.282660] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10380.286451] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [10380.289047] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [10380.295133] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [10380.298448] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [10380.301874] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10384.055027] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [10384.059051] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10384.079569] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10384.085099] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10384.089265] RIP: 0010:native_safe_halt+0x2e/0x50 [10384.091795] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10384.101176] RSP: 0000:ffffaf9fc1b97bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10384.107673] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [10384.111288] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff96848811cae0 [10384.115503] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [10384.119279] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [10384.124319] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10384.127076] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [10384.130351] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10384.132858] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [10384.135581] Call Trace: [10384.136950] [10384.138116] ? show_regs.cold.9+0x22/0x2f [10384.140054] ? watchdog_timer_fn.cold.6+0x62/0x10a [10384.142203] ? __hrtimer_run_queues+0x1c2/0x4d0 [10384.144158] ? watchdog+0x40/0x40 [10384.146354] ? hrtimer_interrupt+0x128/0x2c0 [10384.150222] ? smp_apic_timer_interrupt+0xa6/0x270 [10384.152773] ? apic_timer_interrupt+0xf/0x20 [10384.154766] [10384.155492] ? native_safe_halt+0x2e/0x50 [10384.157203] kvm_wait+0xcd/0xe0 [10384.158173] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10384.161142] do_raw_spin_lock+0x177/0x190 [10384.163008] _raw_spin_lock+0x21/0x30 [10384.165506] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10384.168818] obd_mod_free+0x46/0x140 [ptlrpc] [10384.173192] mdc_commit_open+0x91/0xb0 [mdc] [10384.174738] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10384.176969] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10384.179369] after_reply+0xe2c/0x1780 [ptlrpc] [10384.181305] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10384.182999] ptlrpcd+0xa82/0xe40 [ptlrpc] [10384.186453] ? do_wait_intr+0xf0/0xf0 [10384.187493] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10384.192008] kthread+0x1d1/0x200 [10384.193628] ? set_kthread_struct+0x70/0x70 [10384.199274] ret_from_fork+0x1f/0x30 [10387.937938] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8429s! [10387.946283] Showing busy workqueues and worker pools: [10387.949247] workqueue events: flags=0x0 [10387.951261] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10387.960933] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [10387.971277] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [10387.978404] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [10387.990479] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10387.992475] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [10388.011071] workqueue events_long: flags=0x0 [10388.016346] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10388.030028] pending: lu_objects_flush [obdclass] [10388.034901] workqueue events_unbound: flags=0x2 [10388.036522] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [10388.038964] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10388.043581] workqueue events_power_efficient: flags=0x80 [10388.045979] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10388.048828] pending: neigh_periodic_work [10388.050639] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10388.053173] pending: do_cache_clean, check_lifetime [10388.055041] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10388.057331] pending: neigh_periodic_work [10388.059014] workqueue writeback: flags=0x4a [10388.060614] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10388.062969] pending: wb_workfn, wb_workfn [10388.065034] workqueue xprtiod: flags=0xa [10388.066919] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10388.069426] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [10388.072391] workqueue ipv6_addrconf: flags=0x40008 [10388.074392] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10388.077155] pending: addrconf_verify_work [10388.079136] workqueue knbd0-recv: flags=0x1a [10388.081001] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10388.083699] in-flight: 657:recv_work [10388.085362] workqueue ptlrpc_pinger: flags=0x2 [10388.087152] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10388.089608] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [10388.092309] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8429s workers=4 idle: 20831 12642 28859 [10388.095279] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10378s workers=2 idle: 659 [10396.071866] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [10396.081078] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10396.138148] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10396.146411] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10396.150945] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10396.152668] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10396.166810] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10396.170319] RAX: 000000000000646d RBX: ffff96848811cae0 RCX: 0000000000000001 [10396.174203] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [10396.178157] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [10396.180955] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [10396.184326] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [10396.187691] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [10396.190808] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10396.193333] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [10396.196814] Call Trace: [10396.198334] [10396.199711] ? show_regs.cold.9+0x22/0x2f [10396.201285] ? watchdog_timer_fn.cold.6+0x62/0x10a [10396.203663] ? __hrtimer_run_queues+0x1c2/0x4d0 [10396.206265] ? watchdog+0x40/0x40 [10396.207639] ? hrtimer_interrupt+0x128/0x2c0 [10396.209678] ? smp_apic_timer_interrupt+0xa6/0x270 [10396.211637] ? apic_timer_interrupt+0xf/0x20 [10396.214230] [10396.215070] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10396.217710] do_raw_spin_lock+0x177/0x190 [10396.219250] _raw_spin_lock+0x21/0x30 [10396.220849] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10396.223571] ? do_raw_spin_unlock+0x75/0x190 [10396.225191] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10396.227733] ? do_wait_intr+0xf0/0xf0 [10396.229414] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10396.232181] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10396.235223] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10396.238395] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10396.240825] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10396.243385] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10396.246133] mdc_intent_lock+0x353/0x7e0 [mdc] [10396.248598] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10396.251296] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10396.254735] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10396.257453] lmv_intent_open+0x338/0x1150 [lmv] [10396.259652] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10396.262008] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10396.265201] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10396.268145] ll_intent_lock+0xc5/0xaa0 [lustre] [10396.270276] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10396.273495] ll_lookup_it+0xe0e/0x2270 [lustre] [10396.275227] ? do_raw_spin_unlock+0x75/0x190 [10396.278111] ? _raw_spin_unlock+0x12/0x30 [10396.281224] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10396.285042] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10396.290670] ll_atomic_open+0x36a/0xd70 [lustre] [10396.297101] lookup_open+0x356/0xb40 [10396.302547] do_last+0x313/0xfc0 [10396.303853] ? nd_jump_root+0xe5/0x160 [10396.306161] ? path_init+0x437/0x520 [10396.308027] path_openat+0xf7/0x500 [10396.310721] ? do_raw_spin_unlock+0x75/0x190 [10396.312951] do_filp_open+0x99/0x140 [10396.315623] ? getname_flags+0x6e/0x330 [10396.317810] ? __check_object_size+0xff/0x256 [10396.320017] ? do_raw_spin_unlock+0x75/0x190 [10396.323902] ? _raw_spin_unlock+0x12/0x30 [10396.325619] do_sys_openat2+0x2b4/0x410 [10396.328014] do_sys_open+0x73/0xa0 [10396.330247] __x64_sys_openat+0x24/0x30 [10396.332437] do_syscall_64+0xc1/0x440 [10396.333981] entry_SYSCALL_64_after_hwframe+0x49/0xae [10396.336513] RIP: 0033:0x7ff54af45332 [10396.338098] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10396.352834] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10396.356548] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [10396.360395] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [10396.366661] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [10396.370649] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [10396.374049] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [10404.059437] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [10404.063183] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10404.083932] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10404.089315] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10404.092955] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10404.096620] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10404.099752] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10404.117328] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10404.127138] RAX: 000000000000211f RBX: ffff96848811cae0 RCX: 0000000000000001 [10404.132094] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [10404.136803] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [10404.141093] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [10404.146909] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [10404.152186] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [10404.156874] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10404.161417] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [10404.168540] Call Trace: [10404.170360] [10404.177229] ? show_regs.cold.9+0x22/0x2f [10404.179966] ? watchdog_timer_fn.cold.6+0x62/0x10a [10404.189242] ? __hrtimer_run_queues+0x1c2/0x4d0 [10404.196883] ? watchdog+0x40/0x40 [10404.202887] ? hrtimer_interrupt+0x128/0x2c0 [10404.212473] ? smp_apic_timer_interrupt+0xa6/0x270 [10404.214171] ? apic_timer_interrupt+0xf/0x20 [10404.225785] [10404.228109] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10404.236484] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10404.251022] do_raw_spin_lock+0x177/0x190 [10404.254840] _raw_spin_lock+0x21/0x30 [10404.260283] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10404.271795] process_one_work+0x2c8/0x700 [10404.282207] worker_thread+0x296/0x6e0 [10404.283684] ? rescuer_thread+0x570/0x570 [10404.291387] kthread+0x1d1/0x200 [10404.294631] ? set_kthread_struct+0x70/0x70 [10404.297733] ret_from_fork+0x1f/0x30 [10408.060881] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [10408.066822] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10408.115593] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10408.132839] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10408.143539] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10408.148280] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10408.172559] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10408.182050] RAX: 0000000000006c1b RBX: ffff96848811cae0 RCX: 0000000000000001 [10408.191187] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [10408.202012] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [10408.215826] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [10408.224950] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [10408.235040] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [10408.242441] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10408.249780] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [10408.269116] Call Trace: [10408.277130] [10408.279790] ? show_regs.cold.9+0x22/0x2f [10408.283942] ? watchdog_timer_fn.cold.6+0x62/0x10a [10408.293093] ? __hrtimer_run_queues+0x1c2/0x4d0 [10408.301092] ? watchdog+0x40/0x40 [10408.302307] ? hrtimer_interrupt+0x128/0x2c0 [10408.308375] ? smp_apic_timer_interrupt+0xa6/0x270 [10408.316444] ? apic_timer_interrupt+0xf/0x20 [10408.322704] [10408.324549] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10408.329257] do_raw_spin_lock+0x177/0x190 [10408.332976] _raw_spin_lock+0x21/0x30 [10408.334240] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10408.338725] ? do_raw_spin_unlock+0x75/0x190 [10408.341639] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10408.349039] ? do_wait_intr+0xf0/0xf0 [10408.356778] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10408.365898] mdc_close+0x53b/0xd70 [mdc] [10408.373136] lmv_close+0x18d/0x490 [lmv] [10408.376402] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [10408.382256] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10408.384125] ll_md_real_close+0xa5/0x2a0 [lustre] [10408.392488] ll_md_close+0x274/0x800 [lustre] [10408.396798] ll_file_release+0x128/0xc50 [lustre] [10408.405744] __fput+0x10d/0x440 [10408.409927] ____fput+0x12/0x20 [10408.413599] task_work_run+0xc6/0x110 [10408.416835] exit_to_usermode_loop+0x1dd/0x1f0 [10408.422441] do_syscall_64+0x3d6/0x440 [10408.424933] entry_SYSCALL_64_after_hwframe+0x49/0xae [10408.429998] RIP: 0033:0x7ff54af45cd8 [10408.431251] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10408.451752] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10408.459905] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [10408.466506] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [10408.482604] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [10408.485062] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [10408.498795] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10416.055021] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [10416.058132] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10416.078349] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10416.082972] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10416.087052] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10416.089742] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10416.097485] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10416.101329] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [10416.105110] RDX: 000000000000288c RSI: 0000000000000003 RDI: ffff96848811cae0 [10416.108426] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [10416.111863] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [10416.114705] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10416.118110] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [10416.122773] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10416.124992] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [10416.128783] Call Trace: [10416.129694] [10416.130442] ? show_regs.cold.9+0x22/0x2f [10416.132071] ? watchdog_timer_fn.cold.6+0x62/0x10a [10416.134323] ? __hrtimer_run_queues+0x1c2/0x4d0 [10416.136026] ? watchdog+0x40/0x40 [10416.137928] ? hrtimer_interrupt+0x128/0x2c0 [10416.139519] ? smp_apic_timer_interrupt+0xa6/0x270 [10416.142150] ? apic_timer_interrupt+0xf/0x20 [10416.144489] [10416.145306] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10416.147897] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10416.150399] do_raw_spin_lock+0x177/0x190 [10416.152044] _raw_spin_lock+0x21/0x30 [10416.153898] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10416.156586] obd_mod_free+0x46/0x140 [ptlrpc] [10416.159508] mdc_commit_open+0x91/0xb0 [mdc] [10416.160809] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10416.162647] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10416.164610] after_reply+0xe2c/0x1780 [ptlrpc] [10416.166630] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10416.168571] ptlrpcd+0xa82/0xe40 [ptlrpc] [10416.170203] ? do_wait_intr+0xf0/0xf0 [10416.171730] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10416.174166] kthread+0x1d1/0x200 [10416.175208] ? set_kthread_struct+0x70/0x70 [10416.177003] ret_from_fork+0x1f/0x30 [10418.143082] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8459s! [10418.148737] Showing busy workqueues and worker pools: [10418.151187] workqueue events: flags=0x0 [10418.153517] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10418.157526] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [10418.162551] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [10418.165530] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [10418.171122] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10418.174008] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [10418.176893] workqueue events_long: flags=0x0 [10418.179147] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10418.182031] pending: lu_objects_flush [obdclass] [10418.185123] workqueue events_unbound: flags=0x2 [10418.187011] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [10418.189421] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10418.195288] workqueue events_power_efficient: flags=0x80 [10418.197889] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10418.201338] pending: neigh_periodic_work [10418.203542] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10418.206621] pending: do_cache_clean, check_lifetime [10418.208751] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10418.211691] pending: neigh_periodic_work [10418.214059] workqueue writeback: flags=0x4a [10418.215937] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10418.218529] pending: wb_workfn, wb_workfn [10418.220812] workqueue xprtiod: flags=0xa [10418.222484] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10418.225555] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [10418.229395] workqueue ipv6_addrconf: flags=0x40008 [10418.231768] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10418.234898] pending: addrconf_verify_work [10418.236983] workqueue knbd0-recv: flags=0x1a [10418.239167] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10418.242815] in-flight: 657:recv_work [10418.244765] workqueue ptlrpc_pinger: flags=0x2 [10418.247024] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10418.250470] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [10418.253711] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8459s workers=4 idle: 20831 12642 28859 [10418.257488] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10408s workers=2 idle: 659 [10424.062001] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [10424.065691] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10424.094599] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10424.103489] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10424.111012] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10424.116046] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10424.129779] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10424.135402] RAX: 000000000000440f RBX: ffff96848811cae0 RCX: 0000000000000001 [10424.139065] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [10424.142524] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [10424.145597] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [10424.148493] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [10424.155360] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [10424.161779] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10424.165001] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [10424.168392] Call Trace: [10424.169171] [10424.170062] ? show_regs.cold.9+0x22/0x2f [10424.171950] ? watchdog_timer_fn.cold.6+0x62/0x10a [10424.176755] ? __hrtimer_run_queues+0x1c2/0x4d0 [10424.178212] ? watchdog+0x40/0x40 [10424.181487] ? hrtimer_interrupt+0x128/0x2c0 [10424.184756] ? smp_apic_timer_interrupt+0xa6/0x270 [10424.188449] ? apic_timer_interrupt+0xf/0x20 [10424.190529] [10424.191675] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10424.194369] do_raw_spin_lock+0x177/0x190 [10424.196257] _raw_spin_lock+0x21/0x30 [10424.197713] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10424.201467] ? do_raw_spin_unlock+0x75/0x190 [10424.203456] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10424.210604] ? do_wait_intr+0xf0/0xf0 [10424.215438] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10424.219278] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10424.222540] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10424.225714] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10424.230938] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10424.233789] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10424.235575] mdc_intent_lock+0x353/0x7e0 [mdc] [10424.237940] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10424.240496] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10424.246926] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10424.253124] lmv_intent_open+0x338/0x1150 [lmv] [10424.256337] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10424.260025] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10424.262920] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10424.265582] ll_intent_lock+0xc5/0xaa0 [lustre] [10424.272245] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10424.275210] ll_lookup_it+0xe0e/0x2270 [lustre] [10424.277566] ? do_raw_spin_unlock+0x75/0x190 [10424.282666] ? _raw_spin_unlock+0x12/0x30 [10424.284778] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10424.287020] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10424.289378] ll_atomic_open+0x36a/0xd70 [lustre] [10424.292018] lookup_open+0x356/0xb40 [10424.293990] do_last+0x313/0xfc0 [10424.296013] ? nd_jump_root+0xe5/0x160 [10424.297592] ? path_init+0x437/0x520 [10424.299412] path_openat+0xf7/0x500 [10424.301089] ? do_raw_spin_unlock+0x75/0x190 [10424.303441] do_filp_open+0x99/0x140 [10424.305117] ? getname_flags+0x6e/0x330 [10424.306710] ? __check_object_size+0xff/0x256 [10424.309719] ? do_raw_spin_unlock+0x75/0x190 [10424.311444] ? _raw_spin_unlock+0x12/0x30 [10424.313296] do_sys_openat2+0x2b4/0x410 [10424.315056] do_sys_open+0x73/0xa0 [10424.316653] __x64_sys_openat+0x24/0x30 [10424.318429] do_syscall_64+0xc1/0x440 [10424.320380] entry_SYSCALL_64_after_hwframe+0x49/0xae [10424.322706] RIP: 0033:0x7ff54af45332 [10424.328687] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10424.355575] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10424.358680] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [10424.362561] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [10424.366439] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [10424.370102] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [10424.373332] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [10432.059356] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [10432.064336] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10432.100975] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10432.110703] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10432.128504] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10432.141858] RIP: 0010:native_safe_halt+0x2e/0x50 [10432.145611] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10432.166536] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10432.177974] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [10432.184654] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c2134154 [10432.191395] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [10432.197902] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [10432.203678] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [10432.211211] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [10432.218383] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10432.220915] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [10432.223893] Call Trace: [10432.225204] [10432.225971] ? show_regs.cold.9+0x22/0x2f [10432.227805] ? watchdog_timer_fn.cold.6+0x62/0x10a [10432.231945] ? __hrtimer_run_queues+0x1c2/0x4d0 [10432.237212] ? watchdog+0x40/0x40 [10432.239682] ? hrtimer_interrupt+0x128/0x2c0 [10432.241821] ? smp_apic_timer_interrupt+0xa6/0x270 [10432.243615] ? apic_timer_interrupt+0xf/0x20 [10432.245685] [10432.246739] ? native_safe_halt+0x2e/0x50 [10432.249739] kvm_wait+0xcd/0xe0 [10432.255482] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10432.260721] do_raw_spin_lock+0x177/0x190 [10432.268568] _raw_spin_lock+0x21/0x30 [10432.272602] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10432.286779] process_one_work+0x2c8/0x700 [10432.294920] worker_thread+0x296/0x6e0 [10432.297569] ? rescuer_thread+0x570/0x570 [10432.304680] kthread+0x1d1/0x200 [10432.313025] ? set_kthread_struct+0x70/0x70 [10432.322545] ret_from_fork+0x1f/0x30 [10436.058059] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [10436.068785] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10436.160200] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10436.179619] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10436.195012] RIP: 0010:native_safe_halt+0x2e/0x50 [10436.201834] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10436.239255] RSP: 0018:ffffaf9fc1543ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10436.241743] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [10436.248589] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c20b4154 [10436.252150] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [10436.257781] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [10436.264136] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [10436.276391] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [10436.279119] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10436.286822] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [10436.294255] Call Trace: [10436.296137] [10436.301862] ? show_regs.cold.9+0x22/0x2f [10436.308393] ? watchdog_timer_fn.cold.6+0x62/0x10a [10436.309864] ? __hrtimer_run_queues+0x1c2/0x4d0 [10436.319357] ? watchdog+0x40/0x40 [10436.326900] ? hrtimer_interrupt+0x128/0x2c0 [10436.328070] ? smp_apic_timer_interrupt+0xa6/0x270 [10436.338809] ? apic_timer_interrupt+0xf/0x20 [10436.351488] [10436.352272] ? native_safe_halt+0x2e/0x50 [10436.354659] kvm_wait+0xcd/0xe0 [10436.356850] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10436.362119] do_raw_spin_lock+0x177/0x190 [10436.363211] _raw_spin_lock+0x21/0x30 [10436.372244] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10436.374366] ? do_raw_spin_unlock+0x75/0x190 [10436.382664] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10436.388222] ? do_wait_intr+0xf0/0xf0 [10436.396011] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10436.404781] mdc_close+0x53b/0xd70 [mdc] [10436.408485] lmv_close+0x18d/0x490 [lmv] [10436.409849] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [10436.415843] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10436.425882] ll_md_real_close+0xa5/0x2a0 [lustre] [10436.436355] ll_md_close+0x274/0x800 [lustre] [10436.438021] ll_file_release+0x128/0xc50 [lustre] [10436.443176] __fput+0x10d/0x440 [10436.445970] ____fput+0x12/0x20 [10436.451786] task_work_run+0xc6/0x110 [10436.462689] exit_to_usermode_loop+0x1dd/0x1f0 [10436.464363] do_syscall_64+0x3d6/0x440 [10436.470337] entry_SYSCALL_64_after_hwframe+0x49/0xae [10436.477085] RIP: 0033:0x7ff54af45cd8 [10436.481804] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10436.497023] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10436.501143] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [10436.510552] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [10436.522356] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [10436.530143] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [10436.537814] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10444.055017] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [10444.057199] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10444.068527] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10444.071360] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10444.073602] RIP: 0010:native_safe_halt+0x2e/0x50 [10444.074788] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10444.079132] RSP: 0000:ffffaf9fc1b97bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10444.080894] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [10444.082563] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff96848811cae0 [10444.084716] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [10444.087130] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [10444.089039] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10444.090879] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [10444.093275] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10444.094651] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [10444.096531] Call Trace: [10444.097177] [10444.097594] ? show_regs.cold.9+0x22/0x2f [10444.098567] ? watchdog_timer_fn.cold.6+0x62/0x10a [10444.099675] ? __hrtimer_run_queues+0x1c2/0x4d0 [10444.100870] ? watchdog+0x40/0x40 [10444.101664] ? hrtimer_interrupt+0x128/0x2c0 [10444.102798] ? smp_apic_timer_interrupt+0xa6/0x270 [10444.104015] ? apic_timer_interrupt+0xf/0x20 [10444.105044] [10444.105505] ? native_safe_halt+0x2e/0x50 [10444.106391] kvm_wait+0xcd/0xe0 [10444.107140] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10444.108642] do_raw_spin_lock+0x177/0x190 [10444.109785] _raw_spin_lock+0x21/0x30 [10444.111095] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10444.112900] obd_mod_free+0x46/0x140 [ptlrpc] [10444.114741] mdc_commit_open+0x91/0xb0 [mdc] [10444.115928] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10444.117517] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10444.119076] after_reply+0xe2c/0x1780 [ptlrpc] [10444.120511] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10444.122741] ptlrpcd+0xa82/0xe40 [ptlrpc] [10444.124202] ? do_wait_intr+0xf0/0xf0 [10444.125140] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10444.126654] kthread+0x1d1/0x200 [10444.127507] ? set_kthread_struct+0x70/0x70 [10444.128634] ret_from_fork+0x1f/0x30 [10448.357058] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8489s! [10448.361474] Showing busy workqueues and worker pools: [10448.375797] workqueue events: flags=0x0 [10448.386240] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10448.398364] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [10448.414874] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [10448.431270] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [10448.458889] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10448.468695] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [10448.479689] workqueue events_long: flags=0x0 [10448.490257] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10448.505558] pending: lu_objects_flush [obdclass] [10448.518465] workqueue events_unbound: flags=0x2 [10448.520063] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [10448.533213] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10448.557708] workqueue events_power_efficient: flags=0x80 [10448.573456] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10448.581382] pending: neigh_periodic_work [10448.585320] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10448.598339] pending: do_cache_clean, check_lifetime [10448.605232] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10448.613380] pending: neigh_periodic_work [10448.619819] workqueue writeback: flags=0x4a [10448.624092] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10448.631395] pending: wb_workfn, wb_workfn [10448.637178] workqueue xprtiod: flags=0xa [10448.642750] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10448.654309] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [10448.673320] workqueue ipv6_addrconf: flags=0x40008 [10448.687832] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10448.710726] pending: addrconf_verify_work [10448.719436] workqueue knbd0-recv: flags=0x1a [10448.724670] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10448.737563] in-flight: 657:recv_work [10448.746702] workqueue ptlrpc_pinger: flags=0x2 [10448.754849] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10448.756856] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [10448.769032] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8489s workers=4 idle: 20831 12642 28859 [10448.780307] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10438s workers=2 idle: 659 [10452.060088] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [10452.063984] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10452.084118] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10452.089169] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10452.092988] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10452.095478] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10452.107574] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10452.119550] RAX: 0000000000000cb4 RBX: ffff96848811cae0 RCX: 0000000000000001 [10452.126500] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [10452.135288] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [10452.144198] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [10452.150808] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [10452.157400] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [10452.165983] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10452.174608] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [10452.193217] Call Trace: [10452.194699] [10452.196174] ? show_regs.cold.9+0x22/0x2f [10452.204242] ? watchdog_timer_fn.cold.6+0x62/0x10a [10452.209044] ? __hrtimer_run_queues+0x1c2/0x4d0 [10452.223864] ? watchdog+0x40/0x40 [10452.230924] ? hrtimer_interrupt+0x128/0x2c0 [10452.239939] ? smp_apic_timer_interrupt+0xa6/0x270 [10452.246602] ? apic_timer_interrupt+0xf/0x20 [10452.248881] [10452.250823] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10452.253820] do_raw_spin_lock+0x177/0x190 [10452.256100] _raw_spin_lock+0x21/0x30 [10452.260750] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10452.265845] ? do_raw_spin_unlock+0x75/0x190 [10452.267274] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10452.274747] ? do_wait_intr+0xf0/0xf0 [10452.277618] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10452.290187] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10452.300883] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10452.307169] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10452.315059] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10452.323402] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10452.327924] mdc_intent_lock+0x353/0x7e0 [mdc] [10452.337835] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10452.345650] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10452.357047] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10452.363103] lmv_intent_open+0x338/0x1150 [lmv] [10452.372554] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10452.378157] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10452.382775] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10452.385447] ll_intent_lock+0xc5/0xaa0 [lustre] [10452.387841] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10452.411811] ll_lookup_it+0xe0e/0x2270 [lustre] [10452.425742] ? do_raw_spin_unlock+0x75/0x190 [10452.428362] ? _raw_spin_unlock+0x12/0x30 [10452.434119] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10452.437310] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10452.441812] ll_atomic_open+0x36a/0xd70 [lustre] [10452.443797] lookup_open+0x356/0xb40 [10452.446787] do_last+0x313/0xfc0 [10452.448718] ? nd_jump_root+0xe5/0x160 [10452.450561] ? path_init+0x437/0x520 [10452.454418] path_openat+0xf7/0x500 [10452.458536] ? do_raw_spin_unlock+0x75/0x190 [10452.464661] do_filp_open+0x99/0x140 [10452.465782] ? getname_flags+0x6e/0x330 [10452.468294] ? __check_object_size+0xff/0x256 [10452.471615] ? do_raw_spin_unlock+0x75/0x190 [10452.480612] ? _raw_spin_unlock+0x12/0x30 [10452.482727] do_sys_openat2+0x2b4/0x410 [10452.486107] do_sys_open+0x73/0xa0 [10452.488135] __x64_sys_openat+0x24/0x30 [10452.491751] do_syscall_64+0xc1/0x440 [10452.493817] entry_SYSCALL_64_after_hwframe+0x49/0xae [10452.497039] RIP: 0033:0x7ff54af45332 [10452.505014] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10452.525341] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10452.533063] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [10452.543988] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [10452.552376] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [10452.568299] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [10452.577758] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [10460.061187] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [10460.073217] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10460.134963] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10460.148526] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10460.167368] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10460.178630] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10460.180896] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10460.219600] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10460.227302] RAX: 000000000000268b RBX: ffff96848811cae0 RCX: 0000000000000001 [10460.241292] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [10460.251940] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [10460.260816] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [10460.262903] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [10460.274383] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [10460.284374] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10460.293340] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [10460.302354] Call Trace: [10460.307764] [10460.312010] ? show_regs.cold.9+0x22/0x2f [10460.314506] ? watchdog_timer_fn.cold.6+0x62/0x10a [10460.317123] ? __hrtimer_run_queues+0x1c2/0x4d0 [10460.321163] ? watchdog+0x40/0x40 [10460.332321] ? hrtimer_interrupt+0x128/0x2c0 [10460.334509] ? smp_apic_timer_interrupt+0xa6/0x270 [10460.343071] ? apic_timer_interrupt+0xf/0x20 [10460.349587] [10460.351214] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10460.352909] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10460.364392] do_raw_spin_lock+0x177/0x190 [10460.370350] _raw_spin_lock+0x21/0x30 [10460.377222] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10460.383277] process_one_work+0x2c8/0x700 [10460.392983] worker_thread+0x296/0x6e0 [10460.397355] ? rescuer_thread+0x570/0x570 [10460.402291] kthread+0x1d1/0x200 [10460.404293] ? set_kthread_struct+0x70/0x70 [10460.405655] ret_from_fork+0x1f/0x30 [10464.061453] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [10464.068264] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10464.126626] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10464.140644] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10464.148506] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10464.153198] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10464.162754] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10464.174692] RAX: 000000000000596c RBX: ffff96848811cae0 RCX: 0000000000000001 [10464.190025] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [10464.198261] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [10464.201904] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [10464.207125] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [10464.213272] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [10464.219127] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10464.222667] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [10464.227228] Call Trace: [10464.228160] [10464.229667] ? show_regs.cold.9+0x22/0x2f [10464.234298] ? watchdog_timer_fn.cold.6+0x62/0x10a [10464.242266] ? __hrtimer_run_queues+0x1c2/0x4d0 [10464.248160] ? watchdog+0x40/0x40 [10464.252240] ? hrtimer_interrupt+0x128/0x2c0 [10464.263312] ? smp_apic_timer_interrupt+0xa6/0x270 [10464.266319] ? apic_timer_interrupt+0xf/0x20 [10464.273656] [10464.274733] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10464.281087] do_raw_spin_lock+0x177/0x190 [10464.286182] _raw_spin_lock+0x21/0x30 [10464.293059] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10464.299159] ? do_raw_spin_unlock+0x75/0x190 [10464.305412] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10464.314650] ? do_wait_intr+0xf0/0xf0 [10464.318074] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10464.320617] mdc_close+0x53b/0xd70 [mdc] [10464.322124] lmv_close+0x18d/0x490 [lmv] [10464.323640] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [10464.326054] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10464.330394] ll_md_real_close+0xa5/0x2a0 [lustre] [10464.332823] ll_md_close+0x274/0x800 [lustre] [10464.335218] ll_file_release+0x128/0xc50 [lustre] [10464.340478] __fput+0x10d/0x440 [10464.341387] ____fput+0x12/0x20 [10464.343380] task_work_run+0xc6/0x110 [10464.344421] exit_to_usermode_loop+0x1dd/0x1f0 [10464.345726] do_syscall_64+0x3d6/0x440 [10464.349232] entry_SYSCALL_64_after_hwframe+0x49/0xae [10464.352077] RIP: 0033:0x7ff54af45cd8 [10464.353527] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10464.359930] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10464.362394] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [10464.374876] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [10464.379691] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [10464.387204] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [10464.392348] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10474.083058] rcu: INFO: rcu_sched self-detected stall on CPU [10474.085753] rcu: 0-...!: (4477050 ticks this GP) idle=98a/1/0x4000000000000002 softirq=125154/125154 fqs=959 [10474.099676] (t=8520243 jiffies g=253585 q=5640) [10474.101399] rcu: rcu_sched kthread starved for 8515203 jiffies! g253585 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [10474.120293] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [10474.129561] rcu: RCU grace-period kthread stack dump: [10474.135316] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [10474.155732] Call Trace: [10474.158721] __schedule+0x351/0xcb0 [10474.165970] schedule+0xc0/0x180 [10474.172600] schedule_timeout+0xb4/0x190 [10474.183166] ? do_raw_spin_unlock+0x75/0x190 [10474.190632] ? __next_timer_interrupt+0x160/0x160 [10474.199201] ? prepare_to_swait_event+0xf3/0x140 [10474.210873] rcu_gp_kthread+0x8dd/0x15b0 [10474.217559] ? force_qs_rnp+0x2c0/0x2c0 [10474.225128] kthread+0x1d1/0x200 [10474.227033] ? set_kthread_struct+0x70/0x70 [10474.232560] ret_from_fork+0x1f/0x30 [10474.235338] rcu: Stack dump where RCU GP kthread last ran: [10474.238092] NMI backtrace for cpu 0 [10474.239377] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10474.250889] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10474.268360] Call Trace: [10474.275978] [10474.276940] ? dump_stack+0xbb/0x10e [10474.282234] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [10474.291323] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [10474.300846] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [10474.314832] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [10474.322308] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [10474.326445] ? rcu_check_gp_kthread_starvation+0x16f/0x188 [10474.334289] ? rcu_sched_clock_irq.cold.51+0xce/0x307 [10474.343650] ? update_process_times+0x6d/0xc0 [10474.350628] ? tick_sched_handle.isra.18+0x27/0x90 [10474.357335] ? tick_sched_timer+0x53/0xb0 [10474.366833] ? __hrtimer_run_queues+0x1c2/0x4d0 [10474.379136] ? tick_sched_handle.isra.18+0x90/0x90 [10474.388242] ? hrtimer_interrupt+0x128/0x2c0 [10474.391294] ? smp_apic_timer_interrupt+0xa6/0x270 [10474.403509] ? apic_timer_interrupt+0xf/0x20 [10474.409943] [10474.413745] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10474.420386] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10474.423916] ? do_raw_spin_lock+0x177/0x190 [10474.429260] ? _raw_spin_lock+0x21/0x30 [10474.434830] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10474.438589] ? obd_mod_free+0x46/0x140 [ptlrpc] [10474.448510] ? mdc_commit_open+0x91/0xb0 [mdc] [10474.457968] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10474.470365] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10474.479641] ? after_reply+0xe2c/0x1780 [ptlrpc] [10474.489431] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10474.499192] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [10474.501129] ? do_wait_intr+0xf0/0xf0 [10474.508532] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10474.520189] ? kthread+0x1d1/0x200 [10474.523636] ? set_kthread_struct+0x70/0x70 [10474.531006] ? ret_from_fork+0x1f/0x30 [10474.534878] NMI backtrace for cpu 0 [10474.538623] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10474.553770] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10474.563498] Call Trace: [10474.564545] [10474.565736] ? dump_stack+0xbb/0x10e [10474.569245] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [10474.574467] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [10474.583667] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [10474.588929] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [10474.592033] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [10474.596180] ? rcu_dump_cpu_stacks+0x11b/0x171 [10474.598870] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [10474.601630] ? update_process_times+0x6d/0xc0 [10474.604203] ? tick_sched_handle.isra.18+0x27/0x90 [10474.607498] ? tick_sched_timer+0x53/0xb0 [10474.609485] ? __hrtimer_run_queues+0x1c2/0x4d0 [10474.612291] ? tick_sched_handle.isra.18+0x90/0x90 [10474.619890] ? hrtimer_interrupt+0x128/0x2c0 [10474.626373] ? smp_apic_timer_interrupt+0xa6/0x270 [10474.630935] ? apic_timer_interrupt+0xf/0x20 [10474.636431] [10474.641254] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10474.646280] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10474.650674] ? do_raw_spin_lock+0x177/0x190 [10474.658410] ? _raw_spin_lock+0x21/0x30 [10474.661941] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10474.670516] ? obd_mod_free+0x46/0x140 [ptlrpc] [10474.681077] ? mdc_commit_open+0x91/0xb0 [mdc] [10474.688146] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10474.696365] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10474.699607] ? after_reply+0xe2c/0x1780 [ptlrpc] [10474.709900] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10474.718908] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [10474.721755] ? do_wait_intr+0xf0/0xf0 [10474.723493] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10474.731276] ? kthread+0x1d1/0x200 [10474.734033] ? set_kthread_struct+0x70/0x70 [10474.736285] ? ret_from_fork+0x1f/0x30 [10479.075148] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8520s! [10479.091473] Showing busy workqueues and worker pools: [10479.105985] workqueue events: flags=0x0 [10479.111437] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10479.123659] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [10479.142083] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [10479.158135] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [10479.173126] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10479.188985] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [10479.193442] workqueue events_long: flags=0x0 [10479.206116] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10479.220754] pending: lu_objects_flush [obdclass] [10479.236615] workqueue events_unbound: flags=0x2 [10479.242813] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [10479.253673] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10479.268720] workqueue events_power_efficient: flags=0x80 [10479.279834] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10479.284953] pending: neigh_periodic_work [10479.286464] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10479.305708] pending: do_cache_clean, check_lifetime [10479.313339] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10479.321474] pending: neigh_periodic_work [10479.330514] workqueue writeback: flags=0x4a [10479.339465] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10479.349809] pending: wb_workfn, wb_workfn [10479.355623] workqueue xprtiod: flags=0xa [10479.363470] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10479.374665] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [10479.387870] workqueue ipv6_addrconf: flags=0x40008 [10479.396533] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10479.409179] pending: addrconf_verify_work [10479.415863] workqueue knbd0-recv: flags=0x1a [10479.418324] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10479.421103] in-flight: 657:recv_work [10479.427569] workqueue ptlrpc_pinger: flags=0x2 [10479.434924] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10479.444052] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [10479.452328] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8520s workers=4 idle: 20831 12642 28859 [10479.458208] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10469s workers=2 idle: 659 [10480.064395] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [10480.074448] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10480.126061] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10480.140081] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10480.147424] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10480.152889] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10480.171679] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10480.174963] RAX: 000000000000721b RBX: ffff96848811cae0 RCX: 0000000000000001 [10480.184391] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [10480.189574] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [10480.198131] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [10480.209495] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [10480.216896] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [10480.229153] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10480.235102] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [10480.238868] Call Trace: [10480.240119] [10480.241310] ? show_regs.cold.9+0x22/0x2f [10480.244010] ? watchdog_timer_fn.cold.6+0x62/0x10a [10480.247413] ? __hrtimer_run_queues+0x1c2/0x4d0 [10480.252467] ? watchdog+0x40/0x40 [10480.254444] ? hrtimer_interrupt+0x128/0x2c0 [10480.256902] ? smp_apic_timer_interrupt+0xa6/0x270 [10480.260306] ? apic_timer_interrupt+0xf/0x20 [10480.264929] [10480.267714] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10480.271333] do_raw_spin_lock+0x177/0x190 [10480.278425] _raw_spin_lock+0x21/0x30 [10480.282094] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10480.289501] ? do_raw_spin_unlock+0x75/0x190 [10480.291921] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10480.300808] ? do_wait_intr+0xf0/0xf0 [10480.305831] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10480.312780] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10480.320531] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10480.328069] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10480.332428] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10480.337391] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10480.341098] mdc_intent_lock+0x353/0x7e0 [mdc] [10480.344527] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10480.351439] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10480.362138] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10480.367771] lmv_intent_open+0x338/0x1150 [lmv] [10480.372616] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10480.379098] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10480.386410] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10480.389310] ll_intent_lock+0xc5/0xaa0 [lustre] [10480.396085] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10480.403363] ll_lookup_it+0xe0e/0x2270 [lustre] [10480.409353] ? do_raw_spin_unlock+0x75/0x190 [10480.414489] ? _raw_spin_unlock+0x12/0x30 [10480.417732] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10480.420107] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10480.424368] ll_atomic_open+0x36a/0xd70 [lustre] [10480.426924] lookup_open+0x356/0xb40 [10480.428888] do_last+0x313/0xfc0 [10480.430510] ? nd_jump_root+0xe5/0x160 [10480.432484] ? path_init+0x437/0x520 [10480.434362] path_openat+0xf7/0x500 [10480.435911] ? do_raw_spin_unlock+0x75/0x190 [10480.437731] do_filp_open+0x99/0x140 [10480.439647] ? getname_flags+0x6e/0x330 [10480.441801] ? __check_object_size+0xff/0x256 [10480.444117] ? do_raw_spin_unlock+0x75/0x190 [10480.446281] ? _raw_spin_unlock+0x12/0x30 [10480.449904] do_sys_openat2+0x2b4/0x410 [10480.457706] do_sys_open+0x73/0xa0 [10480.463311] __x64_sys_openat+0x24/0x30 [10480.466400] do_syscall_64+0xc1/0x440 [10480.471754] entry_SYSCALL_64_after_hwframe+0x49/0xae [10480.476523] RIP: 0033:0x7ff54af45332 [10480.481407] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10480.493007] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10480.512501] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [10480.524553] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [10480.536332] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [10480.547397] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [10480.554684] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [10488.059016] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [10488.061433] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10488.089317] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10488.097020] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10488.102365] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10488.106058] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10488.109197] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10488.117893] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10488.122860] RAX: 0000000000001a0b RBX: ffff96848811cae0 RCX: 0000000000000001 [10488.128780] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [10488.133195] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [10488.136021] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [10488.138806] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [10488.140961] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [10488.144741] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10488.147113] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [10488.151176] Call Trace: [10488.152860] [10488.153746] ? show_regs.cold.9+0x22/0x2f [10488.154948] ? watchdog_timer_fn.cold.6+0x62/0x10a [10488.157246] ? __hrtimer_run_queues+0x1c2/0x4d0 [10488.159282] ? watchdog+0x40/0x40 [10488.161143] ? hrtimer_interrupt+0x128/0x2c0 [10488.162999] ? smp_apic_timer_interrupt+0xa6/0x270 [10488.165116] ? apic_timer_interrupt+0xf/0x20 [10488.168086] [10488.169016] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10488.171891] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10488.173401] do_raw_spin_lock+0x177/0x190 [10488.175150] _raw_spin_lock+0x21/0x30 [10488.179579] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10488.182115] process_one_work+0x2c8/0x700 [10488.183712] worker_thread+0x296/0x6e0 [10488.187902] ? rescuer_thread+0x570/0x570 [10488.191164] kthread+0x1d1/0x200 [10488.194882] ? set_kthread_struct+0x70/0x70 [10488.197004] ret_from_fork+0x1f/0x30 [10492.058544] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [10492.072361] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10492.144453] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10492.159747] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10492.168362] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10492.176086] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10492.197498] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10492.218603] RAX: 0000000000003b8f RBX: ffff96848811cae0 RCX: 0000000000000001 [10492.238228] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [10492.246412] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [10492.262710] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [10492.278100] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [10492.290824] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [10492.304300] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10492.322822] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [10492.337434] Call Trace: [10492.343288] [10492.350275] ? show_regs.cold.9+0x22/0x2f [10492.355099] ? watchdog_timer_fn.cold.6+0x62/0x10a [10492.367753] ? __hrtimer_run_queues+0x1c2/0x4d0 [10492.371295] ? watchdog+0x40/0x40 [10492.382681] ? hrtimer_interrupt+0x128/0x2c0 [10492.384163] ? smp_apic_timer_interrupt+0xa6/0x270 [10492.398601] ? apic_timer_interrupt+0xf/0x20 [10492.399986] [10492.410805] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10492.422953] do_raw_spin_lock+0x177/0x190 [10492.429342] _raw_spin_lock+0x21/0x30 [10492.431829] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10492.439139] ? do_raw_spin_unlock+0x75/0x190 [10492.443262] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10492.454868] ? do_wait_intr+0xf0/0xf0 [10492.460239] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10492.470972] mdc_close+0x53b/0xd70 [mdc] [10492.475123] lmv_close+0x18d/0x490 [lmv] [10492.486010] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [10492.492735] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10492.501160] ll_md_real_close+0xa5/0x2a0 [lustre] [10492.510555] ll_md_close+0x274/0x800 [lustre] [10492.517296] ll_file_release+0x128/0xc50 [lustre] [10492.524822] __fput+0x10d/0x440 [10492.529352] ____fput+0x12/0x20 [10492.530874] task_work_run+0xc6/0x110 [10492.538348] exit_to_usermode_loop+0x1dd/0x1f0 [10492.545055] do_syscall_64+0x3d6/0x440 [10492.555263] entry_SYSCALL_64_after_hwframe+0x49/0xae [10492.563533] RIP: 0033:0x7ff54af45cd8 [10492.565947] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10492.588976] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10492.602322] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [10492.614689] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [10492.618750] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [10492.628130] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [10492.642336] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10504.055068] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [10504.058257] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10504.096460] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10504.103297] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10504.112227] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10504.119067] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10504.132178] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10504.138341] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [10504.144492] RDX: 000000000000781d RSI: 0000000000000003 RDI: ffff96848811cae0 [10504.151258] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [10504.161329] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [10504.165822] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10504.170183] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [10504.177855] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10504.180849] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [10504.187255] Call Trace: [10504.189328] [10504.191139] ? show_regs.cold.9+0x22/0x2f [10504.192922] ? watchdog_timer_fn.cold.6+0x62/0x10a [10504.195259] ? __hrtimer_run_queues+0x1c2/0x4d0 [10504.197229] ? watchdog+0x40/0x40 [10504.198958] ? hrtimer_interrupt+0x128/0x2c0 [10504.201246] ? smp_apic_timer_interrupt+0xa6/0x270 [10504.203823] ? apic_timer_interrupt+0xf/0x20 [10504.207078] [10504.208310] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10504.211721] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10504.214815] do_raw_spin_lock+0x177/0x190 [10504.216998] _raw_spin_lock+0x21/0x30 [10504.219144] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10504.222256] obd_mod_free+0x46/0x140 [ptlrpc] [10504.225375] mdc_commit_open+0x91/0xb0 [mdc] [10504.227763] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10504.231178] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10504.234240] after_reply+0xe2c/0x1780 [ptlrpc] [10504.237114] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10504.240038] ptlrpcd+0xa82/0xe40 [ptlrpc] [10504.242303] ? do_wait_intr+0xf0/0xf0 [10504.244175] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10504.247242] kthread+0x1d1/0x200 [10504.249037] ? set_kthread_struct+0x70/0x70 [10504.251166] ret_from_fork+0x1f/0x30 [10508.060265] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [10508.063877] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10508.096893] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10508.102941] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10508.110265] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10508.113153] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10508.132589] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10508.139187] RAX: 000000000000037e RBX: ffff96848811cae0 RCX: 0000000000000001 [10508.145301] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [10508.148004] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [10508.151076] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [10508.152853] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [10508.157858] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [10508.161886] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10508.168528] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [10508.171900] Call Trace: [10508.174445] [10508.176220] ? show_regs.cold.9+0x22/0x2f [10508.179237] ? watchdog_timer_fn.cold.6+0x62/0x10a [10508.183562] ? __hrtimer_run_queues+0x1c2/0x4d0 [10508.184828] ? watchdog+0x40/0x40 [10508.185630] ? hrtimer_interrupt+0x128/0x2c0 [10508.187735] ? smp_apic_timer_interrupt+0xa6/0x270 [10508.189853] ? apic_timer_interrupt+0xf/0x20 [10508.191621] [10508.192701] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10508.195614] do_raw_spin_lock+0x177/0x190 [10508.197576] _raw_spin_lock+0x21/0x30 [10508.199172] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10508.201774] ? do_raw_spin_unlock+0x75/0x190 [10508.203629] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10508.209253] ? do_wait_intr+0xf0/0xf0 [10508.211206] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10508.216136] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10508.220282] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10508.222765] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10508.225360] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10508.230006] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10508.233055] mdc_intent_lock+0x353/0x7e0 [mdc] [10508.234671] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10508.237648] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10508.243161] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10508.246065] lmv_intent_open+0x338/0x1150 [lmv] [10508.248038] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10508.250729] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10508.254487] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10508.257568] ll_intent_lock+0xc5/0xaa0 [lustre] [10508.261269] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10508.263106] ll_lookup_it+0xe0e/0x2270 [lustre] [10508.265917] ? do_raw_spin_unlock+0x75/0x190 [10508.270185] ? _raw_spin_unlock+0x12/0x30 [10508.272403] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10508.274245] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10508.277278] ll_atomic_open+0x36a/0xd70 [lustre] [10508.279509] lookup_open+0x356/0xb40 [10508.281291] do_last+0x313/0xfc0 [10508.282939] ? nd_jump_root+0xe5/0x160 [10508.290156] ? path_init+0x437/0x520 [10508.291499] path_openat+0xf7/0x500 [10508.293055] ? do_raw_spin_unlock+0x75/0x190 [10508.295095] do_filp_open+0x99/0x140 [10508.304661] ? getname_flags+0x6e/0x330 [10508.306883] ? __check_object_size+0xff/0x256 [10508.309720] ? do_raw_spin_unlock+0x75/0x190 [10508.316622] ? _raw_spin_unlock+0x12/0x30 [10508.320878] do_sys_openat2+0x2b4/0x410 [10508.324398] do_sys_open+0x73/0xa0 [10508.328212] __x64_sys_openat+0x24/0x30 [10508.332050] do_syscall_64+0xc1/0x440 [10508.333316] entry_SYSCALL_64_after_hwframe+0x49/0xae [10508.341707] RIP: 0033:0x7ff54af45332 [10508.347449] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10508.358905] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10508.362143] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [10508.366456] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [10508.374198] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [10508.376542] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [10508.379065] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [10509.791102] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8550s! [10509.798273] Showing busy workqueues and worker pools: [10509.802525] workqueue events: flags=0x0 [10509.805298] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10509.809013] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [10509.816325] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [10509.823232] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [10509.833071] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10509.838134] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [10509.843216] workqueue events_long: flags=0x0 [10509.845408] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10509.850056] pending: lu_objects_flush [obdclass] [10509.853970] workqueue events_unbound: flags=0x2 [10509.856821] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [10509.861295] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10509.868309] workqueue events_power_efficient: flags=0x80 [10509.872265] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10509.877562] pending: neigh_periodic_work [10509.880490] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10509.884685] pending: do_cache_clean, check_lifetime [10509.888299] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10509.892921] pending: neigh_periodic_work [10509.895816] workqueue writeback: flags=0x4a [10509.898450] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10509.901627] pending: wb_workfn, wb_workfn [10509.904932] workqueue xprtiod: flags=0xa [10509.907437] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10509.910636] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [10509.914551] workqueue ipv6_addrconf: flags=0x40008 [10509.917609] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10509.922041] pending: addrconf_verify_work [10509.925535] workqueue knbd0-recv: flags=0x1a [10509.928934] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10509.933265] in-flight: 657:recv_work [10509.936062] workqueue ptlrpc_pinger: flags=0x2 [10509.939459] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10509.943827] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [10509.948140] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8551s workers=4 idle: 20831 12642 28859 [10509.954115] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10500s workers=2 idle: 659 [10516.059341] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [10516.067356] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10516.111556] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10516.124979] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10516.131181] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10516.134958] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10516.139951] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10516.165291] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [10516.172340] RAX: 00000000000033a0 RBX: ffff96848811cae0 RCX: 0000000000000001 [10516.182067] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [10516.187961] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [10516.193050] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [10516.199040] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [10516.205142] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [10516.220806] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10516.230553] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [10516.242249] Call Trace: [10516.243100] [10516.246490] ? show_regs.cold.9+0x22/0x2f [10516.263685] ? watchdog_timer_fn.cold.6+0x62/0x10a [10516.273127] ? __hrtimer_run_queues+0x1c2/0x4d0 [10516.274813] ? watchdog+0x40/0x40 [10516.276002] ? hrtimer_interrupt+0x128/0x2c0 [10516.283947] ? smp_apic_timer_interrupt+0xa6/0x270 [10516.289586] ? apic_timer_interrupt+0xf/0x20 [10516.291136] [10516.291932] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10516.296347] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10516.300514] do_raw_spin_lock+0x177/0x190 [10516.308452] _raw_spin_lock+0x21/0x30 [10516.314066] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10516.323062] process_one_work+0x2c8/0x700 [10516.327169] worker_thread+0x296/0x6e0 [10516.331082] ? rescuer_thread+0x570/0x570 [10516.339734] kthread+0x1d1/0x200 [10516.341448] ? set_kthread_struct+0x70/0x70 [10516.343006] ret_from_fork+0x1f/0x30 [10520.066175] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [10520.070050] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10520.099243] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10520.111070] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10520.119236] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10520.135601] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10520.171029] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10520.175981] RAX: 000000000000573d RBX: ffff96848811cae0 RCX: 0000000000000001 [10520.181979] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [10520.191075] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [10520.198009] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [10520.208192] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [10520.218282] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [10520.227230] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10520.235768] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [10520.246645] Call Trace: [10520.247586] [10520.248292] ? show_regs.cold.9+0x22/0x2f [10520.256166] ? watchdog_timer_fn.cold.6+0x62/0x10a [10520.262900] ? __hrtimer_run_queues+0x1c2/0x4d0 [10520.271602] ? watchdog+0x40/0x40 [10520.277932] ? hrtimer_interrupt+0x128/0x2c0 [10520.283934] ? smp_apic_timer_interrupt+0xa6/0x270 [10520.289470] ? apic_timer_interrupt+0xf/0x20 [10520.296335] [10520.297499] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10520.305263] do_raw_spin_lock+0x177/0x190 [10520.309103] _raw_spin_lock+0x21/0x30 [10520.311729] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10520.319014] ? do_raw_spin_unlock+0x75/0x190 [10520.325094] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10520.331953] ? do_wait_intr+0xf0/0xf0 [10520.336207] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10520.340261] mdc_close+0x53b/0xd70 [mdc] [10520.342701] lmv_close+0x18d/0x490 [lmv] [10520.344763] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [10520.349099] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10520.352947] ll_md_real_close+0xa5/0x2a0 [lustre] [10520.355062] ll_md_close+0x274/0x800 [lustre] [10520.356838] ll_file_release+0x128/0xc50 [lustre] [10520.359130] __fput+0x10d/0x440 [10520.360422] ____fput+0x12/0x20 [10520.362015] task_work_run+0xc6/0x110 [10520.363645] exit_to_usermode_loop+0x1dd/0x1f0 [10520.365189] do_syscall_64+0x3d6/0x440 [10520.367589] entry_SYSCALL_64_after_hwframe+0x49/0xae [10520.370012] RIP: 0033:0x7ff54af45cd8 [10520.371828] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10520.380212] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10520.383533] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [10520.387807] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [10520.391047] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [10520.394533] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [10520.398051] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10536.075214] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [10536.076059] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [10536.092418] Modules linked in: [10536.108084] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10536.108195] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10536.108205] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10536.108208] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10536.108242] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10536.108246] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10536.108251] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [10536.108254] RDX: 0000000000005125 RSI: 0000000000000003 RDI: ffff96848811cae0 [10536.108257] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [10536.108259] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [10536.108261] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10536.108265] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [10536.108268] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10536.108270] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [10536.108284] Call Trace: [10536.108292] [10536.108299] ? show_regs.cold.9+0x22/0x2f [10536.108311] ? watchdog_timer_fn.cold.6+0x62/0x10a [10536.108318] ? __hrtimer_run_queues+0x1c2/0x4d0 [10536.108322] ? watchdog+0x40/0x40 [10536.108327] ? hrtimer_interrupt+0x128/0x2c0 [10536.108331] ? smp_apic_timer_interrupt+0xa6/0x270 [10536.108340] ? apic_timer_interrupt+0xf/0x20 [10536.108345] [10536.108349] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10536.108354] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10536.108359] do_raw_spin_lock+0x177/0x190 [10536.108364] _raw_spin_lock+0x21/0x30 [10536.108371] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10536.108910] obd_mod_free+0x46/0x140 [ptlrpc] [10536.109123] mdc_commit_open+0x91/0xb0 [mdc] [10536.109270] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10536.109476] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10536.109750] after_reply+0xe2c/0x1780 [ptlrpc] [10536.110003] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10536.110320] ptlrpcd+0xa82/0xe40 [ptlrpc] [10536.111740] ? do_wait_intr+0xf0/0xf0 [10536.111760] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10536.113568] kthread+0x1d1/0x200 [10536.113580] ? set_kthread_struct+0x70/0x70 [10536.115216] ret_from_fork+0x1f/0x30 [10536.524556] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10536.604575] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10536.619473] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10536.625292] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10536.637538] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10536.680634] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10536.688207] RAX: 0000000000004584 RBX: ffff96848811cae0 RCX: 0000000000000001 [10536.698530] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [10536.709711] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [10536.724422] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [10536.733760] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [10536.740750] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [10536.754051] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10536.769261] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [10536.784825] Call Trace: [10536.787483] [10536.789437] ? show_regs.cold.9+0x22/0x2f [10536.797382] ? watchdog_timer_fn.cold.6+0x62/0x10a [10536.806896] ? __hrtimer_run_queues+0x1c2/0x4d0 [10536.811901] ? watchdog+0x40/0x40 [10536.815957] ? hrtimer_interrupt+0x128/0x2c0 [10536.820555] ? smp_apic_timer_interrupt+0xa6/0x270 [10536.834698] ? apic_timer_interrupt+0xf/0x20 [10536.843556] [10536.844581] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10536.853929] do_raw_spin_lock+0x177/0x190 [10536.859614] _raw_spin_lock+0x21/0x30 [10536.865961] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10536.876035] ? do_raw_spin_unlock+0x75/0x190 [10536.882789] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10536.884638] ? do_wait_intr+0xf0/0xf0 [10536.903185] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10536.910305] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10536.925597] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10536.950196] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10536.958934] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10536.973625] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10536.988386] mdc_intent_lock+0x353/0x7e0 [mdc] [10536.995339] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10536.998884] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10537.008967] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10537.018619] lmv_intent_open+0x338/0x1150 [lmv] [10537.026243] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10537.032626] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10537.041637] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10537.051316] ll_intent_lock+0xc5/0xaa0 [lustre] [10537.055347] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10537.067725] ll_lookup_it+0xe0e/0x2270 [lustre] [10537.070973] ? do_raw_spin_unlock+0x75/0x190 [10537.082268] ? _raw_spin_unlock+0x12/0x30 [10537.087550] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10537.093582] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10537.105572] ll_atomic_open+0x36a/0xd70 [lustre] [10537.110485] lookup_open+0x356/0xb40 [10537.118233] do_last+0x313/0xfc0 [10537.122951] ? nd_jump_root+0xe5/0x160 [10537.126223] ? path_init+0x437/0x520 [10537.134467] path_openat+0xf7/0x500 [10537.138530] ? do_raw_spin_unlock+0x75/0x190 [10537.144448] do_filp_open+0x99/0x140 [10537.152514] ? getname_flags+0x6e/0x330 [10537.163543] ? __check_object_size+0xff/0x256 [10537.167945] ? do_raw_spin_unlock+0x75/0x190 [10537.175428] ? _raw_spin_unlock+0x12/0x30 [10537.181543] do_sys_openat2+0x2b4/0x410 [10537.191538] do_sys_open+0x73/0xa0 [10537.200604] __x64_sys_openat+0x24/0x30 [10537.211762] do_syscall_64+0xc1/0x440 [10537.219830] entry_SYSCALL_64_after_hwframe+0x49/0xae [10537.231635] RIP: 0033:0x7ff54af45332 [10537.234248] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10537.258603] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10537.270061] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [10537.275237] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [10537.283985] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [10537.292239] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [10537.301855] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [10539.999146] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8581s! [10540.009045] Showing busy workqueues and worker pools: [10540.011647] workqueue events: flags=0x0 [10540.019313] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10540.031386] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [10540.044759] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [10540.059431] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [10540.076725] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10540.090584] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [10540.109454] workqueue events_long: flags=0x0 [10540.118941] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10540.125970] pending: lu_objects_flush [obdclass] [10540.131665] workqueue events_unbound: flags=0x2 [10540.136424] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [10540.143732] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10540.154433] workqueue events_power_efficient: flags=0x80 [10540.162972] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10540.170361] pending: neigh_periodic_work [10540.175978] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10540.182767] pending: do_cache_clean, check_lifetime [10540.189669] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10540.205936] pending: neigh_periodic_work [10540.211265] workqueue writeback: flags=0x4a [10540.217619] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10540.225697] pending: wb_workfn, wb_workfn [10540.233494] workqueue xprtiod: flags=0xa [10540.241539] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10540.250932] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [10540.259946] workqueue ipv6_addrconf: flags=0x40008 [10540.261550] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10540.281166] pending: addrconf_verify_work [10540.287167] workqueue knbd0-recv: flags=0x1a [10540.293475] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10540.310483] in-flight: 657:recv_work [10540.316649] workqueue ptlrpc_pinger: flags=0x2 [10540.321516] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10540.333757] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [10540.341068] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8581s workers=4 idle: 20831 12642 28859 [10540.356612] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10530s workers=2 idle: 659 [10544.069576] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [10544.092925] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10544.171330] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10544.179922] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10544.197693] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10544.209647] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10544.222813] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10544.253461] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10544.267290] RAX: 0000000000007a0b RBX: ffff96848811cae0 RCX: 0000000000000001 [10544.285911] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [10544.289154] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [10544.297218] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [10544.309209] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [10544.318709] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [10544.328152] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10544.333996] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [10544.344465] Call Trace: [10544.358299] [10544.362295] ? show_regs.cold.9+0x22/0x2f [10544.380530] ? watchdog_timer_fn.cold.6+0x62/0x10a [10544.391738] ? __hrtimer_run_queues+0x1c2/0x4d0 [10544.400788] ? watchdog+0x40/0x40 [10544.406657] ? hrtimer_interrupt+0x128/0x2c0 [10544.411664] ? smp_apic_timer_interrupt+0xa6/0x270 [10544.419694] ? apic_timer_interrupt+0xf/0x20 [10544.424585] [10544.427532] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10544.430793] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10544.450008] do_raw_spin_lock+0x177/0x190 [10544.459182] _raw_spin_lock+0x21/0x30 [10544.465793] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10544.472519] process_one_work+0x2c8/0x700 [10544.479770] worker_thread+0x296/0x6e0 [10544.491444] ? rescuer_thread+0x570/0x570 [10544.499064] kthread+0x1d1/0x200 [10544.501029] ? set_kthread_struct+0x70/0x70 [10544.512445] ret_from_fork+0x1f/0x30 [10548.062498] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [10548.069353] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10548.154515] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10548.184854] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10548.215112] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10548.216910] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10548.259911] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10548.272817] RAX: 0000000000006e93 RBX: ffff96848811cae0 RCX: 0000000000000001 [10548.278019] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [10548.283748] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [10548.291894] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [10548.309025] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [10548.317471] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [10548.323614] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10548.328466] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [10548.333603] Call Trace: [10548.334897] [10548.337266] ? show_regs.cold.9+0x22/0x2f [10548.339490] ? watchdog_timer_fn.cold.6+0x62/0x10a [10548.343524] ? __hrtimer_run_queues+0x1c2/0x4d0 [10548.345468] ? watchdog+0x40/0x40 [10548.351137] ? hrtimer_interrupt+0x128/0x2c0 [10548.359743] ? smp_apic_timer_interrupt+0xa6/0x270 [10548.365444] ? apic_timer_interrupt+0xf/0x20 [10548.367444] [10548.368376] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10548.376298] do_raw_spin_lock+0x177/0x190 [10548.381205] _raw_spin_lock+0x21/0x30 [10548.382807] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10548.386359] ? do_raw_spin_unlock+0x75/0x190 [10548.390924] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10548.396935] ? do_wait_intr+0xf0/0xf0 [10548.400698] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10548.406579] mdc_close+0x53b/0xd70 [mdc] [10548.412297] lmv_close+0x18d/0x490 [lmv] [10548.416609] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [10548.422756] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10548.430857] ll_md_real_close+0xa5/0x2a0 [lustre] [10548.432387] ll_md_close+0x274/0x800 [lustre] [10548.433826] ll_file_release+0x128/0xc50 [lustre] [10548.435900] __fput+0x10d/0x440 [10548.439192] ____fput+0x12/0x20 [10548.440409] task_work_run+0xc6/0x110 [10548.444384] exit_to_usermode_loop+0x1dd/0x1f0 [10548.448407] do_syscall_64+0x3d6/0x440 [10548.451251] entry_SYSCALL_64_after_hwframe+0x49/0xae [10548.460334] RIP: 0033:0x7ff54af45cd8 [10548.462500] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10548.478856] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10548.484219] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [10548.488558] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [10548.495756] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [10548.500108] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [10548.505126] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10564.061143] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [10564.070260] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10564.135093] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10564.146216] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10564.162081] RIP: 0010:native_safe_halt+0x2e/0x50 [10564.173194] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10564.198071] RSP: 0000:ffffaf9fc15234f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10564.205489] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [10564.209203] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c21b4154 [10564.212645] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [10564.216827] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [10564.223155] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [10564.226900] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [10564.234566] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10564.239644] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [10564.245213] Call Trace: [10564.251237] [10564.251943] ? show_regs.cold.9+0x22/0x2f [10564.258045] ? watchdog_timer_fn.cold.6+0x62/0x10a [10564.264668] ? __hrtimer_run_queues+0x1c2/0x4d0 [10564.269068] ? watchdog+0x40/0x40 [10564.273444] ? hrtimer_interrupt+0x128/0x2c0 [10564.277937] ? smp_apic_timer_interrupt+0xa6/0x270 [10564.283068] ? apic_timer_interrupt+0xf/0x20 [10564.294772] [10564.295708] ? native_safe_halt+0x2e/0x50 [10564.297381] kvm_wait+0xcd/0xe0 [10564.302885] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10564.312107] do_raw_spin_lock+0x177/0x190 [10564.318909] _raw_spin_lock+0x21/0x30 [10564.324720] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10564.330180] ? do_raw_spin_unlock+0x75/0x190 [10564.334195] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10564.342512] ? do_wait_intr+0xf0/0xf0 [10564.353199] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10564.360872] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10564.363291] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10564.368631] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10564.371148] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10564.375235] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10564.377031] mdc_intent_lock+0x353/0x7e0 [mdc] [10564.379722] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10564.383130] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10564.385988] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10564.389202] lmv_intent_open+0x338/0x1150 [lmv] [10564.393296] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10564.399633] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10564.402974] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10564.407307] ll_intent_lock+0xc5/0xaa0 [lustre] [10564.412629] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10564.423277] ll_lookup_it+0xe0e/0x2270 [lustre] [10564.430113] ? do_raw_spin_unlock+0x75/0x190 [10564.439242] ? _raw_spin_unlock+0x12/0x30 [10564.445052] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10564.453584] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10564.456336] ll_atomic_open+0x36a/0xd70 [lustre] [10564.459195] lookup_open+0x356/0xb40 [10564.460355] do_last+0x313/0xfc0 [10564.462006] ? nd_jump_root+0xe5/0x160 [10564.463320] ? path_init+0x437/0x520 [10564.464984] path_openat+0xf7/0x500 [10564.470826] ? do_raw_spin_unlock+0x75/0x190 [10564.476807] do_filp_open+0x99/0x140 [10564.478776] ? getname_flags+0x6e/0x330 [10564.480993] ? __check_object_size+0xff/0x256 [10564.483203] ? do_raw_spin_unlock+0x75/0x190 [10564.487039] ? _raw_spin_unlock+0x12/0x30 [10564.491528] do_sys_openat2+0x2b4/0x410 [10564.492936] do_sys_open+0x73/0xa0 [10564.494080] __x64_sys_openat+0x24/0x30 [10564.495977] do_syscall_64+0xc1/0x440 [10564.498006] entry_SYSCALL_64_after_hwframe+0x49/0xae [10564.500758] RIP: 0033:0x7ff54af45332 [10564.502447] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10564.513157] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10564.519100] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [10564.523510] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [10564.527306] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [10564.531335] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [10564.538238] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [10568.055021] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [10568.059046] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10568.084968] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10568.089765] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10568.094614] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10568.099020] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10568.111695] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10568.115608] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [10568.119234] RDX: 000000000000539f RSI: 0000000000000003 RDI: ffff96848811cae0 [10568.122280] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [10568.127167] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [10568.131744] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10568.136956] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [10568.143011] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10568.146906] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [10568.154496] Call Trace: [10568.155788] [10568.156883] ? show_regs.cold.9+0x22/0x2f [10568.158526] ? watchdog_timer_fn.cold.6+0x62/0x10a [10568.161286] ? __hrtimer_run_queues+0x1c2/0x4d0 [10568.164576] ? watchdog+0x40/0x40 [10568.166208] ? hrtimer_interrupt+0x128/0x2c0 [10568.168730] ? smp_apic_timer_interrupt+0xa6/0x270 [10568.170676] ? apic_timer_interrupt+0xf/0x20 [10568.173714] [10568.175370] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10568.177828] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10568.179864] do_raw_spin_lock+0x177/0x190 [10568.183357] _raw_spin_lock+0x21/0x30 [10568.184903] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10568.187049] obd_mod_free+0x46/0x140 [ptlrpc] [10568.189211] mdc_commit_open+0x91/0xb0 [mdc] [10568.190928] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10568.194370] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10568.197501] after_reply+0xe2c/0x1780 [ptlrpc] [10568.199686] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10568.202554] ptlrpcd+0xa82/0xe40 [ptlrpc] [10568.205151] ? do_wait_intr+0xf0/0xf0 [10568.206362] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10568.208476] kthread+0x1d1/0x200 [10568.209934] ? set_kthread_struct+0x70/0x70 [10568.212534] ret_from_fork+0x1f/0x30 [10570.719734] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8611s! [10570.728055] Showing busy workqueues and worker pools: [10570.730076] workqueue events: flags=0x0 [10570.732907] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10570.741550] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [10570.746728] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [10570.754591] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [10570.765131] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10570.770681] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [10570.775962] workqueue events_long: flags=0x0 [10570.778598] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10570.784054] pending: lu_objects_flush [obdclass] [10570.794430] workqueue events_unbound: flags=0x2 [10570.798256] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [10570.804228] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10570.811373] workqueue events_power_efficient: flags=0x80 [10570.814092] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10570.817777] pending: neigh_periodic_work [10570.819989] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10570.826498] pending: do_cache_clean, check_lifetime [10570.829697] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10570.836538] pending: neigh_periodic_work [10570.838689] workqueue writeback: flags=0x4a [10570.844033] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10570.847825] pending: wb_workfn, wb_workfn [10570.851984] workqueue xprtiod: flags=0xa [10570.854662] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10570.857323] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [10570.863879] workqueue ipv6_addrconf: flags=0x40008 [10570.867189] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10570.870842] pending: addrconf_verify_work [10570.873380] workqueue knbd0-recv: flags=0x1a [10570.879031] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10570.885413] in-flight: 657:recv_work [10570.888983] workqueue ptlrpc_pinger: flags=0x2 [10570.891657] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10570.894779] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [10570.900792] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8612s workers=4 idle: 20831 12642 28859 [10570.904529] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10561s workers=2 idle: 659 [10572.059989] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [10572.065526] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10572.103572] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10572.114455] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10572.123400] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10572.133206] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10572.141359] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10572.154386] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10572.157557] RAX: 0000000000007f82 RBX: ffff96848811cae0 RCX: 0000000000000001 [10572.163923] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [10572.167328] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [10572.174741] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [10572.182290] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [10572.194767] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [10572.205544] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10572.207376] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [10572.221167] Call Trace: [10572.223796] [10572.226000] ? show_regs.cold.9+0x22/0x2f [10572.230132] ? watchdog_timer_fn.cold.6+0x62/0x10a [10572.234235] ? __hrtimer_run_queues+0x1c2/0x4d0 [10572.236721] ? watchdog+0x40/0x40 [10572.238604] ? hrtimer_interrupt+0x128/0x2c0 [10572.241061] ? smp_apic_timer_interrupt+0xa6/0x270 [10572.243041] ? apic_timer_interrupt+0xf/0x20 [10572.245085] [10572.245991] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10572.248571] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10572.251091] do_raw_spin_lock+0x177/0x190 [10572.253079] _raw_spin_lock+0x21/0x30 [10572.255861] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10572.261078] process_one_work+0x2c8/0x700 [10572.263083] worker_thread+0x296/0x6e0 [10572.267144] ? rescuer_thread+0x570/0x570 [10572.270375] kthread+0x1d1/0x200 [10572.273892] ? set_kthread_struct+0x70/0x70 [10572.277610] ret_from_fork+0x1f/0x30 [10576.058070] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [10576.063540] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10576.087484] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10576.091735] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10576.094351] RIP: 0010:native_safe_halt+0x2e/0x50 [10576.096012] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10576.100952] RSP: 0018:ffffaf9fc1543ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10576.102561] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [10576.104369] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c20b4154 [10576.106391] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [10576.108151] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [10576.110376] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [10576.112231] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [10576.114244] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10576.115599] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [10576.117415] Call Trace: [10576.118260] [10576.118706] ? show_regs.cold.9+0x22/0x2f [10576.119630] ? watchdog_timer_fn.cold.6+0x62/0x10a [10576.120696] ? __hrtimer_run_queues+0x1c2/0x4d0 [10576.123411] ? watchdog+0x40/0x40 [10576.124259] ? hrtimer_interrupt+0x128/0x2c0 [10576.125241] ? smp_apic_timer_interrupt+0xa6/0x270 [10576.126823] ? apic_timer_interrupt+0xf/0x20 [10576.127974] [10576.128442] ? native_safe_halt+0x2e/0x50 [10576.129781] kvm_wait+0xcd/0xe0 [10576.131422] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10576.133469] do_raw_spin_lock+0x177/0x190 [10576.135514] _raw_spin_lock+0x21/0x30 [10576.136858] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10576.139932] ? do_raw_spin_unlock+0x75/0x190 [10576.141631] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10576.144453] ? do_wait_intr+0xf0/0xf0 [10576.146206] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10576.149769] mdc_close+0x53b/0xd70 [mdc] [10576.151795] lmv_close+0x18d/0x490 [lmv] [10576.153434] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [10576.156374] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10576.158967] ll_md_real_close+0xa5/0x2a0 [lustre] [10576.161209] ll_md_close+0x274/0x800 [lustre] [10576.164512] ll_file_release+0x128/0xc50 [lustre] [10576.166566] __fput+0x10d/0x440 [10576.167683] ____fput+0x12/0x20 [10576.170340] task_work_run+0xc6/0x110 [10576.173472] exit_to_usermode_loop+0x1dd/0x1f0 [10576.179415] do_syscall_64+0x3d6/0x440 [10576.182235] entry_SYSCALL_64_after_hwframe+0x49/0xae [10576.185389] RIP: 0033:0x7ff54af45cd8 [10576.189230] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10576.201146] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10576.204211] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [10576.208080] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [10576.211371] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [10576.214580] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [10576.218250] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10592.063903] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [10592.070695] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10592.141662] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10592.150426] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10592.155171] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10592.158183] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10592.172049] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10592.181261] RAX: 0000000000004383 RBX: ffff96848811cae0 RCX: 0000000000000001 [10592.187911] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [10592.192150] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [10592.198201] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [10592.204946] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [10592.212868] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [10592.221200] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10592.224581] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [10592.233690] Call Trace: [10592.236960] [10592.238184] ? show_regs.cold.9+0x22/0x2f [10592.242367] ? watchdog_timer_fn.cold.6+0x62/0x10a [10592.248308] ? __hrtimer_run_queues+0x1c2/0x4d0 [10592.257906] ? watchdog+0x40/0x40 [10592.260464] ? hrtimer_interrupt+0x128/0x2c0 [10592.262850] ? smp_apic_timer_interrupt+0xa6/0x270 [10592.265869] ? apic_timer_interrupt+0xf/0x20 [10592.267765] [10592.269709] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10592.274030] do_raw_spin_lock+0x177/0x190 [10592.275941] _raw_spin_lock+0x21/0x30 [10592.277915] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10592.281512] ? do_raw_spin_unlock+0x75/0x190 [10592.284477] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10592.287932] ? do_wait_intr+0xf0/0xf0 [10592.291552] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10592.298178] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10592.306054] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10592.311286] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10592.314269] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10592.318402] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10592.324477] mdc_intent_lock+0x353/0x7e0 [mdc] [10592.331384] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10592.340797] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10592.349900] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10592.353796] lmv_intent_open+0x338/0x1150 [lmv] [10592.359895] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10592.367385] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10592.375161] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10592.377836] ll_intent_lock+0xc5/0xaa0 [lustre] [10592.381318] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10592.388437] ll_lookup_it+0xe0e/0x2270 [lustre] [10592.391790] ? do_raw_spin_unlock+0x75/0x190 [10592.396206] ? _raw_spin_unlock+0x12/0x30 [10592.398879] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10592.401072] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10592.404171] ll_atomic_open+0x36a/0xd70 [lustre] [10592.406805] lookup_open+0x356/0xb40 [10592.412976] do_last+0x313/0xfc0 [10592.414329] ? nd_jump_root+0xe5/0x160 [10592.416871] ? path_init+0x437/0x520 [10592.418268] path_openat+0xf7/0x500 [10592.420178] ? do_raw_spin_unlock+0x75/0x190 [10592.422255] do_filp_open+0x99/0x140 [10592.424210] ? getname_flags+0x6e/0x330 [10592.429284] ? __check_object_size+0xff/0x256 [10592.435220] ? do_raw_spin_unlock+0x75/0x190 [10592.438266] ? _raw_spin_unlock+0x12/0x30 [10592.447405] do_sys_openat2+0x2b4/0x410 [10592.449281] do_sys_open+0x73/0xa0 [10592.452738] __x64_sys_openat+0x24/0x30 [10592.454346] do_syscall_64+0xc1/0x440 [10592.459479] entry_SYSCALL_64_after_hwframe+0x49/0xae [10592.463841] RIP: 0033:0x7ff54af45332 [10592.465410] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10592.476856] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10592.483743] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [10592.487230] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [10592.497134] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [10592.502585] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [10592.505907] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [10596.056268] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [10596.076368] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10596.128001] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10596.134854] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10596.141876] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10596.147378] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10596.159874] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10596.164812] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [10596.168939] RDX: 0000000000001e42 RSI: 0000000000000003 RDI: ffff96848811cae0 [10596.172388] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [10596.176615] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [10596.180084] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10596.184088] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [10596.188277] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10596.192003] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [10596.196837] Call Trace: [10596.198029] [10596.199964] ? show_regs.cold.9+0x22/0x2f [10596.201849] ? watchdog_timer_fn.cold.6+0x62/0x10a [10596.204739] ? __hrtimer_run_queues+0x1c2/0x4d0 [10596.207073] ? watchdog+0x40/0x40 [10596.209032] ? hrtimer_interrupt+0x128/0x2c0 [10596.212035] ? smp_apic_timer_interrupt+0xa6/0x270 [10596.214826] ? apic_timer_interrupt+0xf/0x20 [10596.216790] [10596.218455] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10596.221888] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10596.225646] do_raw_spin_lock+0x177/0x190 [10596.227760] _raw_spin_lock+0x21/0x30 [10596.229916] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10596.233880] obd_mod_free+0x46/0x140 [ptlrpc] [10596.236402] mdc_commit_open+0x91/0xb0 [mdc] [10596.238919] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10596.242481] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10596.246016] after_reply+0xe2c/0x1780 [ptlrpc] [10596.248877] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10596.252727] ptlrpcd+0xa82/0xe40 [ptlrpc] [10596.255872] ? do_wait_intr+0xf0/0xf0 [10596.262509] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10596.266015] kthread+0x1d1/0x200 [10596.267775] ? set_kthread_struct+0x70/0x70 [10596.270061] ret_from_fork+0x1f/0x30 [10600.059082] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [10600.067367] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10600.112771] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10600.121086] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10600.127112] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10600.133996] RIP: 0010:native_safe_halt+0x2e/0x50 [10600.136252] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10600.154863] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10600.164781] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [10600.167102] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c2134154 [10600.179525] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [10600.186875] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [10600.191733] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [10600.196166] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [10600.201948] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10600.206858] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [10600.216376] Call Trace: [10600.220478] [10600.222922] ? show_regs.cold.9+0x22/0x2f [10600.226391] ? watchdog_timer_fn.cold.6+0x62/0x10a [10600.232277] ? __hrtimer_run_queues+0x1c2/0x4d0 [10600.240250] ? watchdog+0x40/0x40 [10600.241937] ? hrtimer_interrupt+0x128/0x2c0 [10600.246696] ? smp_apic_timer_interrupt+0xa6/0x270 [10600.249776] ? apic_timer_interrupt+0xf/0x20 [10600.252377] [10600.253604] ? native_safe_halt+0x2e/0x50 [10600.255121] kvm_wait+0xcd/0xe0 [10600.260191] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10600.275505] do_raw_spin_lock+0x177/0x190 [10600.277986] _raw_spin_lock+0x21/0x30 [10600.280386] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10600.290110] process_one_work+0x2c8/0x700 [10600.291241] worker_thread+0x296/0x6e0 [10600.292278] ? rescuer_thread+0x570/0x570 [10600.293181] kthread+0x1d1/0x200 [10600.297842] ? set_kthread_struct+0x70/0x70 [10600.303420] ret_from_fork+0x1f/0x30 [10600.930968] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8642s! [10600.938066] Showing busy workqueues and worker pools: [10600.939930] workqueue events: flags=0x0 [10600.941189] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10600.943396] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [10600.949085] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [10600.954556] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [10600.962493] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10600.964976] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [10600.967227] workqueue events_long: flags=0x0 [10600.968853] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10600.977405] pending: lu_objects_flush [obdclass] [10600.984849] workqueue events_unbound: flags=0x2 [10600.988287] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [10600.998919] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10601.011746] workqueue events_power_efficient: flags=0x80 [10601.017273] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10601.025787] pending: neigh_periodic_work [10601.027933] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10601.035845] pending: do_cache_clean, check_lifetime [10601.044898] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10601.059846] pending: neigh_periodic_work [10601.065909] workqueue writeback: flags=0x4a [10601.069854] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10601.081959] pending: wb_workfn, wb_workfn [10601.093691] workqueue xprtiod: flags=0xa [10601.099277] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10601.109703] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [10601.124685] workqueue ipv6_addrconf: flags=0x40008 [10601.139699] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10601.147160] pending: addrconf_verify_work [10601.157235] workqueue knbd0-recv: flags=0x1a [10601.159830] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10601.177142] in-flight: 657:recv_work [10601.190219] workqueue ptlrpc_pinger: flags=0x2 [10601.193935] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10601.204001] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [10601.214621] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8642s workers=4 idle: 20831 12642 28859 [10601.226673] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10591s workers=2 idle: 659 [10604.062160] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [10604.070930] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10604.145292] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10604.169205] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10604.184184] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10604.193766] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10604.230310] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10604.240232] RAX: 0000000000007d91 RBX: ffff96848811cae0 RCX: 0000000000000001 [10604.256901] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [10604.262614] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [10604.272264] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [10604.281842] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [10604.290210] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [10604.294566] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10604.297793] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [10604.304847] Call Trace: [10604.308823] [10604.311733] ? show_regs.cold.9+0x22/0x2f [10604.321466] ? watchdog_timer_fn.cold.6+0x62/0x10a [10604.323226] ? __hrtimer_run_queues+0x1c2/0x4d0 [10604.327702] ? watchdog+0x40/0x40 [10604.330767] ? hrtimer_interrupt+0x128/0x2c0 [10604.343185] ? smp_apic_timer_interrupt+0xa6/0x270 [10604.345937] ? apic_timer_interrupt+0xf/0x20 [10604.355751] [10604.356878] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10604.373357] do_raw_spin_lock+0x177/0x190 [10604.377334] _raw_spin_lock+0x21/0x30 [10604.386794] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10604.394038] ? do_raw_spin_unlock+0x75/0x190 [10604.407048] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10604.414301] ? do_wait_intr+0xf0/0xf0 [10604.417997] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10604.422978] mdc_close+0x53b/0xd70 [mdc] [10604.433628] lmv_close+0x18d/0x490 [lmv] [10604.435135] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [10604.440975] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10604.445230] ll_md_real_close+0xa5/0x2a0 [lustre] [10604.447932] ll_md_close+0x274/0x800 [lustre] [10604.450575] ll_file_release+0x128/0xc50 [lustre] [10604.453723] __fput+0x10d/0x440 [10604.455307] ____fput+0x12/0x20 [10604.464432] task_work_run+0xc6/0x110 [10604.468245] exit_to_usermode_loop+0x1dd/0x1f0 [10604.469819] do_syscall_64+0x3d6/0x440 [10604.473022] entry_SYSCALL_64_after_hwframe+0x49/0xae [10604.475800] RIP: 0033:0x7ff54af45cd8 [10604.479667] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10604.496972] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10604.501150] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [10604.505972] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [10604.515330] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [10604.524161] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [10604.527394] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10620.060205] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [10620.064696] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10620.099474] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10620.106326] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10620.111356] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10620.114428] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10620.124485] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10620.127118] RAX: 0000000000003f86 RBX: ffff96848811cae0 RCX: 0000000000000001 [10620.130900] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [10620.135657] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [10620.139968] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [10620.144052] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [10620.148218] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [10620.153035] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10620.156556] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [10620.160580] Call Trace: [10620.162123] [10620.162903] ? show_regs.cold.9+0x22/0x2f [10620.165041] ? watchdog_timer_fn.cold.6+0x62/0x10a [10620.167999] ? __hrtimer_run_queues+0x1c2/0x4d0 [10620.170277] ? watchdog+0x40/0x40 [10620.172192] ? hrtimer_interrupt+0x128/0x2c0 [10620.174588] ? smp_apic_timer_interrupt+0xa6/0x270 [10620.177498] ? apic_timer_interrupt+0xf/0x20 [10620.179631] [10620.180768] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10620.183591] do_raw_spin_lock+0x177/0x190 [10620.185974] _raw_spin_lock+0x21/0x30 [10620.187942] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10620.191006] ? do_raw_spin_unlock+0x75/0x190 [10620.192568] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10620.195638] ? do_wait_intr+0xf0/0xf0 [10620.197861] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10620.200636] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10620.204381] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10620.207929] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10620.210825] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10620.213855] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10620.216342] mdc_intent_lock+0x353/0x7e0 [mdc] [10620.218638] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10620.221637] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10620.225242] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10620.228215] lmv_intent_open+0x338/0x1150 [lmv] [10620.230744] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10620.234557] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10620.238751] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10620.242517] ll_intent_lock+0xc5/0xaa0 [lustre] [10620.246567] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10620.251576] ll_lookup_it+0xe0e/0x2270 [lustre] [10620.253944] ? do_raw_spin_unlock+0x75/0x190 [10620.255986] ? _raw_spin_unlock+0x12/0x30 [10620.257989] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10620.261593] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10620.265452] ll_atomic_open+0x36a/0xd70 [lustre] [10620.273156] lookup_open+0x356/0xb40 [10620.276153] do_last+0x313/0xfc0 [10620.282081] ? nd_jump_root+0xe5/0x160 [10620.284311] ? path_init+0x437/0x520 [10620.285535] path_openat+0xf7/0x500 [10620.287097] ? do_raw_spin_unlock+0x75/0x190 [10620.289609] do_filp_open+0x99/0x140 [10620.291343] ? getname_flags+0x6e/0x330 [10620.294522] ? __check_object_size+0xff/0x256 [10620.296811] ? do_raw_spin_unlock+0x75/0x190 [10620.300792] ? _raw_spin_unlock+0x12/0x30 [10620.304204] do_sys_openat2+0x2b4/0x410 [10620.305733] do_sys_open+0x73/0xa0 [10620.309121] __x64_sys_openat+0x24/0x30 [10620.311529] do_syscall_64+0xc1/0x440 [10620.313940] entry_SYSCALL_64_after_hwframe+0x49/0xae [10620.316775] RIP: 0033:0x7ff54af45332 [10620.320774] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10620.341532] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10620.354731] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [10620.366273] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [10620.372206] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [10620.381900] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [10620.388973] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [10628.055023] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [10628.058419] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10628.059093] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [10628.090426] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10628.093527] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10628.093617] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10628.093627] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10628.093635] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10628.093929] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10628.093941] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10628.093945] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10628.093949] RAX: 0000000000002844 RBX: ffff96848811cae0 RCX: 0000000000000001 [10628.093952] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [10628.093954] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [10628.093957] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [10628.093959] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [10628.093963] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [10628.093966] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10628.093968] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [10628.093977] Call Trace: [10628.093983] [10628.093990] ? show_regs.cold.9+0x22/0x2f [10628.094000] ? watchdog_timer_fn.cold.6+0x62/0x10a [10628.094007] ? __hrtimer_run_queues+0x1c2/0x4d0 [10628.094012] ? watchdog+0x40/0x40 [10628.094016] ? hrtimer_interrupt+0x128/0x2c0 [10628.094019] ? smp_apic_timer_interrupt+0xa6/0x270 [10628.094027] ? apic_timer_interrupt+0xf/0x20 [10628.094031] [10628.094035] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10628.094039] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10628.094044] do_raw_spin_lock+0x177/0x190 [10628.094050] _raw_spin_lock+0x21/0x30 [10628.094054] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10628.094653] process_one_work+0x2c8/0x700 [10628.094662] worker_thread+0x296/0x6e0 [10628.094667] ? rescuer_thread+0x570/0x570 [10628.094671] kthread+0x1d1/0x200 [10628.094676] ? set_kthread_struct+0x70/0x70 [10628.094680] ret_from_fork+0x1f/0x30 [10628.229103] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10628.233741] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10628.236828] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10628.263000] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10628.267436] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [10628.272712] RDX: 000000000000785f RSI: 0000000000000003 RDI: ffff96848811cae0 [10628.275197] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [10628.284822] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [10628.297783] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10628.312045] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [10628.318970] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10628.324587] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [10628.328171] Call Trace: [10628.332569] [10628.335623] ? show_regs.cold.9+0x22/0x2f [10628.338739] ? watchdog_timer_fn.cold.6+0x62/0x10a [10628.341746] ? __hrtimer_run_queues+0x1c2/0x4d0 [10628.348837] ? watchdog+0x40/0x40 [10628.356365] ? hrtimer_interrupt+0x128/0x2c0 [10628.361627] ? smp_apic_timer_interrupt+0xa6/0x270 [10628.363801] ? apic_timer_interrupt+0xf/0x20 [10628.369837] [10628.371303] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10628.376367] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10628.379962] do_raw_spin_lock+0x177/0x190 [10628.381821] _raw_spin_lock+0x21/0x30 [10628.383825] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10628.391044] obd_mod_free+0x46/0x140 [ptlrpc] [10628.397454] mdc_commit_open+0x91/0xb0 [mdc] [10628.402661] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10628.408351] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10628.414207] after_reply+0xe2c/0x1780 [ptlrpc] [10628.419693] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10628.422404] ptlrpcd+0xa82/0xe40 [ptlrpc] [10628.431068] ? do_wait_intr+0xf0/0xf0 [10628.432728] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10628.435767] kthread+0x1d1/0x200 [10628.437525] ? set_kthread_struct+0x70/0x70 [10628.439799] ret_from_fork+0x1f/0x30 [10631.647048] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8672s! [10631.651032] Showing busy workqueues and worker pools: [10631.653454] workqueue events: flags=0x0 [10631.655395] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10631.658957] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [10631.670309] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [10631.674781] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [10631.682185] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10631.685529] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [10631.692943] workqueue events_long: flags=0x0 [10631.695428] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10631.699475] pending: lu_objects_flush [obdclass] [10631.703252] workqueue events_unbound: flags=0x2 [10631.705615] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [10631.709611] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10631.721289] workqueue events_power_efficient: flags=0x80 [10631.726727] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10631.731715] pending: neigh_periodic_work [10631.734596] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10631.740192] pending: do_cache_clean, check_lifetime [10631.744774] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10631.751513] pending: neigh_periodic_work [10631.755581] workqueue writeback: flags=0x4a [10631.758504] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10631.762770] pending: wb_workfn, wb_workfn [10631.767406] workqueue xprtiod: flags=0xa [10631.769486] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10631.774784] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [10631.780286] workqueue ipv6_addrconf: flags=0x40008 [10631.783538] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10631.787366] pending: addrconf_verify_work [10631.789722] workqueue knbd0-recv: flags=0x1a [10631.791280] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10631.795356] in-flight: 657:recv_work [10631.798562] workqueue ptlrpc_pinger: flags=0x2 [10631.801459] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10631.806652] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [10631.812386] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8672s workers=4 idle: 20831 12642 28859 [10631.817080] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10622s workers=2 idle: 659 [10632.058370] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [10632.069959] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10632.111332] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10632.117597] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10632.130235] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10632.136973] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10632.155284] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10632.164476] RAX: 000000000000532d RBX: ffff96848811cae0 RCX: 0000000000000001 [10632.171346] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [10632.175369] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [10632.183252] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [10632.190589] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [10632.198002] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [10632.204620] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10632.210955] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [10632.216365] Call Trace: [10632.217443] [10632.219419] ? show_regs.cold.9+0x22/0x2f [10632.221389] ? watchdog_timer_fn.cold.6+0x62/0x10a [10632.226119] ? __hrtimer_run_queues+0x1c2/0x4d0 [10632.228011] ? watchdog+0x40/0x40 [10632.231096] ? hrtimer_interrupt+0x128/0x2c0 [10632.233283] ? smp_apic_timer_interrupt+0xa6/0x270 [10632.235435] ? apic_timer_interrupt+0xf/0x20 [10632.237512] [10632.238499] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10632.243777] do_raw_spin_lock+0x177/0x190 [10632.248483] _raw_spin_lock+0x21/0x30 [10632.253123] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10632.257000] ? do_raw_spin_unlock+0x75/0x190 [10632.263576] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10632.272428] ? do_wait_intr+0xf0/0xf0 [10632.274503] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10632.280471] mdc_close+0x53b/0xd70 [mdc] [10632.282381] lmv_close+0x18d/0x490 [lmv] [10632.290063] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [10632.293860] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10632.297846] ll_md_real_close+0xa5/0x2a0 [lustre] [10632.300551] ll_md_close+0x274/0x800 [lustre] [10632.303197] ll_file_release+0x128/0xc50 [lustre] [10632.308203] __fput+0x10d/0x440 [10632.312258] ____fput+0x12/0x20 [10632.314007] task_work_run+0xc6/0x110 [10632.317658] exit_to_usermode_loop+0x1dd/0x1f0 [10632.320204] do_syscall_64+0x3d6/0x440 [10632.322628] entry_SYSCALL_64_after_hwframe+0x49/0xae [10632.326985] RIP: 0033:0x7ff54af45cd8 [10632.328850] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10632.342015] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10632.346519] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [10632.350750] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [10632.355580] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [10632.360336] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [10632.366437] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10648.066058] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [10648.072688] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10648.106751] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10648.114170] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10648.120832] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10648.125188] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10648.142732] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10648.149822] RAX: 0000000000005590 RBX: ffff96848811cae0 RCX: 0000000000000001 [10648.168575] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [10648.174920] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [10648.185357] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [10648.193831] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [10648.198956] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [10648.203530] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10648.208185] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [10648.214046] Call Trace: [10648.216155] [10648.217751] ? show_regs.cold.9+0x22/0x2f [10648.221309] ? watchdog_timer_fn.cold.6+0x62/0x10a [10648.225561] ? __hrtimer_run_queues+0x1c2/0x4d0 [10648.229406] ? watchdog+0x40/0x40 [10648.231991] ? hrtimer_interrupt+0x128/0x2c0 [10648.262767] ? smp_apic_timer_interrupt+0xa6/0x270 [10648.266438] ? apic_timer_interrupt+0xf/0x20 [10648.269603] [10648.271323] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10648.274865] do_raw_spin_lock+0x177/0x190 [10648.277595] _raw_spin_lock+0x21/0x30 [10648.279576] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10648.282177] ? do_raw_spin_unlock+0x75/0x190 [10648.284140] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10648.288035] ? do_wait_intr+0xf0/0xf0 [10648.289783] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10648.292421] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10648.295052] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10648.298176] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10648.300564] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10648.303103] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10648.305379] mdc_intent_lock+0x353/0x7e0 [mdc] [10648.330868] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10648.334999] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10648.345690] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10648.349655] lmv_intent_open+0x338/0x1150 [lmv] [10648.352546] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10648.354814] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10648.357758] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10648.359548] ll_intent_lock+0xc5/0xaa0 [lustre] [10648.363283] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10648.364747] ll_lookup_it+0xe0e/0x2270 [lustre] [10648.365913] ? do_raw_spin_unlock+0x75/0x190 [10648.367386] ? _raw_spin_unlock+0x12/0x30 [10648.368609] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10648.370868] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10648.373026] ll_atomic_open+0x36a/0xd70 [lustre] [10648.375861] lookup_open+0x356/0xb40 [10648.377582] do_last+0x313/0xfc0 [10648.378819] ? nd_jump_root+0xe5/0x160 [10648.380373] ? path_init+0x437/0x520 [10648.381714] path_openat+0xf7/0x500 [10648.382851] ? do_raw_spin_unlock+0x75/0x190 [10648.384065] do_filp_open+0x99/0x140 [10648.386397] ? getname_flags+0x6e/0x330 [10648.387951] ? __check_object_size+0xff/0x256 [10648.391662] ? do_raw_spin_unlock+0x75/0x190 [10648.393928] ? _raw_spin_unlock+0x12/0x30 [10648.396292] do_sys_openat2+0x2b4/0x410 [10648.397959] do_sys_open+0x73/0xa0 [10648.399729] __x64_sys_openat+0x24/0x30 [10648.401600] do_syscall_64+0xc1/0x440 [10648.403189] entry_SYSCALL_64_after_hwframe+0x49/0xae [10648.404975] RIP: 0033:0x7ff54af45332 [10648.410347] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10648.416428] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10648.420016] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [10648.422625] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [10648.425161] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [10648.427815] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [10648.430500] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [10654.086018] rcu: INFO: rcu_sched self-detected stall on CPU [10654.088887] rcu: 0-...!: (4569284 ticks this GP) idle=98a/1/0x4000000000000002 softirq=125154/125154 fqs=959 [10654.094008] (t=8700246 jiffies g=253585 q=5734) [10654.096166] rcu: rcu_sched kthread starved for 8695206 jiffies! g253585 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=3 [10654.101936] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [10654.106316] rcu: RCU grace-period kthread stack dump: [10654.108843] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [10654.113684] Call Trace: [10654.115142] __schedule+0x351/0xcb0 [10654.116875] schedule+0xc0/0x180 [10654.119279] schedule_timeout+0xb4/0x190 [10654.121409] ? do_raw_spin_unlock+0x75/0x190 [10654.123821] ? __next_timer_interrupt+0x160/0x160 [10654.126156] ? prepare_to_swait_event+0xf3/0x140 [10654.127978] rcu_gp_kthread+0x8dd/0x15b0 [10654.129895] ? force_qs_rnp+0x2c0/0x2c0 [10654.131541] kthread+0x1d1/0x200 [10654.133300] ? set_kthread_struct+0x70/0x70 [10654.135238] ret_from_fork+0x1f/0x30 [10654.137126] rcu: Stack dump where RCU GP kthread last ran: [10654.139547] Sending NMI from CPU 0 to CPUs 3: [10654.147983] NMI backtrace for cpu 3 [10654.147993] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10654.147995] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10654.147997] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10654.147999] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10654.148001] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 [10654.148004] RAX: 0000000000004117 RBX: ffff96848811cae0 RCX: 0000000000000001 [10654.148005] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [10654.148007] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [10654.148008] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [10654.148009] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [10654.148010] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [10654.148012] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10654.148013] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [10654.148014] Call Trace: [10654.148015] [10654.148016] ? show_regs.cold.9+0x22/0x2f [10654.148017] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [10654.148018] ? nmi_cpu_backtrace_handler+0x15/0x40 [10654.148019] ? nmi_handle+0xbd/0x210 [10654.148020] ? default_do_nmi+0x4b/0x1b0 [10654.148021] ? do_nmi+0x280/0x380 [10654.148022] ? end_repeat_nmi+0x16/0x1f [10654.148023] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10654.148025] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10654.148245] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10654.148247] [10654.148248] do_raw_spin_lock+0x177/0x190 [10654.148248] _raw_spin_lock+0x21/0x30 [10654.148250] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10654.148251] ? do_raw_spin_unlock+0x75/0x190 [10654.148252] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10654.148253] ? do_wait_intr+0xf0/0xf0 [10654.148255] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10654.148256] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10654.148257] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10654.148258] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10654.148260] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10654.148261] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10654.148262] mdc_intent_lock+0x353/0x7e0 [mdc] [10654.148264] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10654.148265] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10654.148266] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10654.148268] lmv_intent_open+0x338/0x1150 [lmv] [10654.148269] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10654.148270] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10654.148271] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10654.148272] ll_intent_lock+0xc5/0xaa0 [lustre] [10654.148274] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10654.148275] ll_lookup_it+0xe0e/0x2270 [lustre] [10654.148276] ? do_raw_spin_unlock+0x75/0x190 [10654.148278] ? _raw_spin_unlock+0x12/0x30 [10654.148279] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10654.148280] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10654.148281] ll_atomic_open+0x36a/0xd70 [lustre] [10654.148283] lookup_open+0x356/0xb40 [10654.148284] do_last+0x313/0xfc0 [10654.148285] ? nd_jump_root+0xe5/0x160 [10654.148287] ? path_init+0x437/0x520 [10654.148288] path_openat+0xf7/0x500 [10654.148289] ? do_raw_spin_unlock+0x75/0x190 [10654.148290] do_filp_open+0x99/0x140 [10654.148291] ? getname_flags+0x6e/0x330 [10654.148292] ? __check_object_size+0xff/0x256 [10654.148294] ? do_raw_spin_unlock+0x75/0x190 [10654.148295] ? _raw_spin_unlock+0x12/0x30 [10654.148296] do_sys_openat2+0x2b4/0x410 [10654.148297] do_sys_open+0x73/0xa0 [10654.148298] __x64_sys_openat+0x24/0x30 [10654.148299] do_syscall_64+0xc1/0x440 [10654.148300] entry_SYSCALL_64_after_hwframe+0x49/0xae [10654.148302] RIP: 0033:0x7ff54af45332 [10654.148303] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10654.148306] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10654.148309] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [10654.148311] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [10654.148312] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [10654.148314] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [10654.148315] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [10654.149683] NMI backtrace for cpu 0 [10654.444754] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10654.458659] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10654.468192] Call Trace: [10654.469753] [10654.474409] ? dump_stack+0xbb/0x10e [10654.478593] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [10654.484647] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [10654.486710] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [10654.492438] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [10654.496590] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [10654.502609] ? rcu_dump_cpu_stacks+0x11b/0x171 [10654.506402] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [10654.513210] ? update_process_times+0x6d/0xc0 [10654.518529] ? tick_sched_handle.isra.18+0x27/0x90 [10654.520739] ? tick_sched_timer+0x53/0xb0 [10654.526099] ? __hrtimer_run_queues+0x1c2/0x4d0 [10654.529618] ? tick_sched_handle.isra.18+0x90/0x90 [10654.531862] ? hrtimer_interrupt+0x128/0x2c0 [10654.533878] ? smp_apic_timer_interrupt+0xa6/0x270 [10654.535821] ? apic_timer_interrupt+0xf/0x20 [10654.537568] [10654.538488] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10654.541060] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10654.543017] ? do_raw_spin_lock+0x177/0x190 [10654.544381] ? _raw_spin_lock+0x21/0x30 [10654.545991] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10654.548985] ? obd_mod_free+0x46/0x140 [ptlrpc] [10654.551142] ? mdc_commit_open+0x91/0xb0 [mdc] [10654.553553] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10654.556180] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10654.559169] ? after_reply+0xe2c/0x1780 [ptlrpc] [10654.561734] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10654.563811] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [10654.565974] ? do_wait_intr+0xf0/0xf0 [10654.567373] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10654.569524] ? kthread+0x1d1/0x200 [10654.571561] ? set_kthread_struct+0x70/0x70 [10654.573610] ? ret_from_fork+0x1f/0x30 [10656.059007] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [10656.071745] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10656.129358] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10656.141187] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10656.146343] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10656.151406] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10656.156415] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10656.174469] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10656.184711] RAX: 0000000000003718 RBX: ffff96848811cae0 RCX: 0000000000000001 [10656.198671] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [10656.207583] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [10656.214384] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [10656.222165] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [10656.231377] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [10656.242248] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10656.248303] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [10656.251246] Call Trace: [10656.261359] [10656.262445] ? show_regs.cold.9+0x22/0x2f [10656.266572] ? watchdog_timer_fn.cold.6+0x62/0x10a [10656.270337] ? __hrtimer_run_queues+0x1c2/0x4d0 [10656.277526] ? watchdog+0x40/0x40 [10656.279428] ? hrtimer_interrupt+0x128/0x2c0 [10656.286962] ? smp_apic_timer_interrupt+0xa6/0x270 [10656.290721] ? apic_timer_interrupt+0xf/0x20 [10656.294578] [10656.296381] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10656.301225] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10656.304313] do_raw_spin_lock+0x177/0x190 [10656.305741] _raw_spin_lock+0x21/0x30 [10656.307391] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10656.310365] process_one_work+0x2c8/0x700 [10656.314124] worker_thread+0x296/0x6e0 [10656.320346] ? rescuer_thread+0x570/0x570 [10656.324495] kthread+0x1d1/0x200 [10656.325662] ? set_kthread_struct+0x70/0x70 [10656.330941] ret_from_fork+0x1f/0x30 [10660.058051] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [10660.062678] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10660.112279] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10660.126179] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10660.139243] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10660.146599] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10660.166594] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10660.176888] RAX: 0000000000002d19 RBX: ffff96848811cae0 RCX: 0000000000000001 [10660.182517] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [10660.185630] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [10660.188808] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [10660.192198] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [10660.195223] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [10660.198789] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10660.206696] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [10660.220731] Call Trace: [10660.225227] [10660.227602] ? show_regs.cold.9+0x22/0x2f [10660.234729] ? watchdog_timer_fn.cold.6+0x62/0x10a [10660.247960] ? __hrtimer_run_queues+0x1c2/0x4d0 [10660.255964] ? watchdog+0x40/0x40 [10660.263608] ? hrtimer_interrupt+0x128/0x2c0 [10660.269179] ? smp_apic_timer_interrupt+0xa6/0x270 [10660.273655] ? apic_timer_interrupt+0xf/0x20 [10660.278447] [10660.279570] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10660.285590] do_raw_spin_lock+0x177/0x190 [10660.287788] _raw_spin_lock+0x21/0x30 [10660.293709] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10660.307452] ? do_raw_spin_unlock+0x75/0x190 [10660.313481] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10660.318436] ? do_wait_intr+0xf0/0xf0 [10660.325557] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10660.327520] mdc_close+0x53b/0xd70 [mdc] [10660.333470] lmv_close+0x18d/0x490 [lmv] [10660.339444] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [10660.347901] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10660.354460] ll_md_real_close+0xa5/0x2a0 [lustre] [10660.360236] ll_md_close+0x274/0x800 [lustre] [10660.362799] ll_file_release+0x128/0xc50 [lustre] [10660.366465] __fput+0x10d/0x440 [10660.369129] ____fput+0x12/0x20 [10660.371214] task_work_run+0xc6/0x110 [10660.373399] exit_to_usermode_loop+0x1dd/0x1f0 [10660.376222] do_syscall_64+0x3d6/0x440 [10660.379144] entry_SYSCALL_64_after_hwframe+0x49/0xae [10660.384457] RIP: 0033:0x7ff54af45cd8 [10660.388789] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10660.408189] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10660.412434] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [10660.416716] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [10660.420667] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [10660.424496] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [10660.427676] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10661.859026] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8702s! [10661.871070] Showing busy workqueues and worker pools: [10661.876318] workqueue events: flags=0x0 [10661.880770] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10661.894118] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [10661.908202] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [10661.912862] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [10661.923123] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10661.927617] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [10661.936483] workqueue events_long: flags=0x0 [10661.944445] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10661.956110] pending: lu_objects_flush [obdclass] [10661.962143] workqueue events_unbound: flags=0x2 [10661.970420] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [10661.982968] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10661.999360] workqueue events_power_efficient: flags=0x80 [10662.012264] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10662.021250] pending: neigh_periodic_work [10662.025490] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10662.033106] pending: do_cache_clean, check_lifetime [10662.040490] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10662.045311] pending: neigh_periodic_work [10662.048481] workqueue writeback: flags=0x4a [10662.053481] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10662.057129] pending: wb_workfn, wb_workfn [10662.059970] workqueue xprtiod: flags=0xa [10662.063403] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10662.072614] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [10662.076475] workqueue ipv6_addrconf: flags=0x40008 [10662.080920] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10662.088430] pending: addrconf_verify_work [10662.093430] workqueue knbd0-recv: flags=0x1a [10662.099322] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10662.110767] in-flight: 657:recv_work [10662.121078] workqueue ptlrpc_pinger: flags=0x2 [10662.128417] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10662.135612] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [10662.142134] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8703s workers=4 idle: 20831 12642 28859 [10662.149214] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10652s workers=2 idle: 659 [10680.060007] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [10680.065516] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10680.100043] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10680.109430] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10680.116329] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10680.122066] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10680.137269] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10680.144617] RAX: 0000000000003998 RBX: ffff96848811cae0 RCX: 0000000000000001 [10680.147578] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [10680.151614] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [10680.166030] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [10680.171548] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [10680.174384] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [10680.179839] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10680.182512] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [10680.185334] Call Trace: [10680.186190] [10680.187077] ? show_regs.cold.9+0x22/0x2f [10680.189896] ? watchdog_timer_fn.cold.6+0x62/0x10a [10680.192800] ? __hrtimer_run_queues+0x1c2/0x4d0 [10680.195084] ? watchdog+0x40/0x40 [10680.196463] ? hrtimer_interrupt+0x128/0x2c0 [10680.198815] ? smp_apic_timer_interrupt+0xa6/0x270 [10680.200879] ? apic_timer_interrupt+0xf/0x20 [10680.205024] [10680.205871] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10680.208069] do_raw_spin_lock+0x177/0x190 [10680.214986] _raw_spin_lock+0x21/0x30 [10680.219767] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10680.222553] ? do_raw_spin_unlock+0x75/0x190 [10680.224172] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10680.228765] ? do_wait_intr+0xf0/0xf0 [10680.233302] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10680.237087] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10680.240490] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10680.248154] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10680.254975] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10680.265561] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10680.268250] mdc_intent_lock+0x353/0x7e0 [mdc] [10680.271629] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10680.277382] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10680.283383] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10680.290963] lmv_intent_open+0x338/0x1150 [lmv] [10680.296689] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10680.301539] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10680.304552] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10680.309139] ll_intent_lock+0xc5/0xaa0 [lustre] [10680.313216] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10680.316641] ll_lookup_it+0xe0e/0x2270 [lustre] [10680.320018] ? do_raw_spin_unlock+0x75/0x190 [10680.322355] ? _raw_spin_unlock+0x12/0x30 [10680.323953] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10680.326354] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10680.329497] ll_atomic_open+0x36a/0xd70 [lustre] [10680.332996] lookup_open+0x356/0xb40 [10680.335190] do_last+0x313/0xfc0 [10680.337630] ? nd_jump_root+0xe5/0x160 [10680.339725] ? path_init+0x437/0x520 [10680.341539] path_openat+0xf7/0x500 [10680.343552] ? do_raw_spin_unlock+0x75/0x190 [10680.345880] do_filp_open+0x99/0x140 [10680.347968] ? getname_flags+0x6e/0x330 [10680.350135] ? __check_object_size+0xff/0x256 [10680.353847] ? do_raw_spin_unlock+0x75/0x190 [10680.355708] ? _raw_spin_unlock+0x12/0x30 [10680.357106] do_sys_openat2+0x2b4/0x410 [10680.359982] do_sys_open+0x73/0xa0 [10680.361999] __x64_sys_openat+0x24/0x30 [10680.364199] do_syscall_64+0xc1/0x440 [10680.366200] entry_SYSCALL_64_after_hwframe+0x49/0xae [10680.369461] RIP: 0033:0x7ff54af45332 [10680.371532] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10680.382594] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10680.385885] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [10680.389153] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [10680.393006] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [10680.397563] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [10680.402913] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [10684.059026] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [10684.062904] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10684.104873] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10684.111294] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10684.116816] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10684.121306] RIP: 0010:native_safe_halt+0x2e/0x50 [10684.123728] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10684.137159] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10684.144961] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [10684.152367] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c2134154 [10684.159052] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [10684.165452] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [10684.169812] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [10684.173472] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [10684.177610] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10684.180565] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [10684.184053] Call Trace: [10684.185129] [10684.186300] ? show_regs.cold.9+0x22/0x2f [10684.188508] ? watchdog_timer_fn.cold.6+0x62/0x10a [10684.190406] ? __hrtimer_run_queues+0x1c2/0x4d0 [10684.191769] ? watchdog+0x40/0x40 [10684.193541] ? hrtimer_interrupt+0x128/0x2c0 [10684.195708] ? smp_apic_timer_interrupt+0xa6/0x270 [10684.198455] ? apic_timer_interrupt+0xf/0x20 [10684.200528] [10684.201519] ? native_safe_halt+0x2e/0x50 [10684.203918] kvm_wait+0xcd/0xe0 [10684.205467] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10684.208445] do_raw_spin_lock+0x177/0x190 [10684.210537] _raw_spin_lock+0x21/0x30 [10684.212362] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10684.216723] process_one_work+0x2c8/0x700 [10684.218885] worker_thread+0x296/0x6e0 [10684.220690] ? rescuer_thread+0x570/0x570 [10684.222977] kthread+0x1d1/0x200 [10684.224817] ? set_kthread_struct+0x70/0x70 [10684.226600] ret_from_fork+0x1f/0x30 [10688.055051] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [10688.058039] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [10688.058181] Modules linked in: [10688.061608] Modules linked in: [10688.061608] lustre(O) [10688.064916] lustre(O) [10688.065478] osp(O) [10688.066522] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10688.066610] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10688.067347] ofd(O) [10688.068334] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10688.093329] lod(O) [10688.098679] RIP: 0010:native_safe_halt+0x2e/0x50 [10688.099281] mdt(O) [10688.106998] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10688.107620] mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata [10688.112110] RSP: 0018:ffffaf9fc1543ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10688.112128] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [10688.112130] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c20b4154 [10688.112133] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [10688.112135] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [10688.112138] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [10688.112142] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [10688.112145] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10688.112148] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [10688.112158] Call Trace: [10688.112166] [10688.112173] ? show_regs.cold.9+0x22/0x2f [10688.112194] ? watchdog_timer_fn.cold.6+0x62/0x10a [10688.112202] ? __hrtimer_run_queues+0x1c2/0x4d0 [10688.112207] ? watchdog+0x40/0x40 [10688.112212] ? hrtimer_interrupt+0x128/0x2c0 [10688.112217] ? smp_apic_timer_interrupt+0xa6/0x270 [10688.112229] ? apic_timer_interrupt+0xf/0x20 [10688.112234] [10688.112238] ? native_safe_halt+0x2e/0x50 [10688.112243] kvm_wait+0xcd/0xe0 [10688.112253] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10688.112264] do_raw_spin_lock+0x177/0x190 [10688.112269] _raw_spin_lock+0x21/0x30 [10688.112274] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10688.114901] ? do_raw_spin_unlock+0x75/0x190 [10688.114917] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10688.115221] ? do_wait_intr+0xf0/0xf0 [10688.115230] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10688.115420] mdc_close+0x53b/0xd70 [mdc] [10688.115468] lmv_close+0x18d/0x490 [lmv] [10688.115497] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [10688.115790] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10688.115932] ll_md_real_close+0xa5/0x2a0 [lustre] [10688.116014] ll_md_close+0x274/0x800 [lustre] [10688.116096] ll_file_release+0x128/0xc50 [lustre] [10688.116178] __fput+0x10d/0x440 [10688.116190] ____fput+0x12/0x20 [10688.116195] task_work_run+0xc6/0x110 [10688.116203] exit_to_usermode_loop+0x1dd/0x1f0 [10688.116211] do_syscall_64+0x3d6/0x440 [10688.116216] entry_SYSCALL_64_after_hwframe+0x49/0xae [10688.116228] RIP: 0033:0x7ff54af45cd8 [10688.116239] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10688.116244] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10688.116250] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [10688.116252] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [10688.116255] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [10688.116257] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [10688.116259] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10688.280323] dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10688.283471] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10688.288445] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10688.292268] RIP: 0010:native_safe_halt+0x2e/0x50 [10688.294254] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10688.302556] RSP: 0000:ffffaf9fc1b97bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10688.305393] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [10688.307737] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff96848811cae0 [10688.310391] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [10688.312899] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [10688.315721] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10688.318563] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [10688.321673] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10688.324473] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [10688.327792] Call Trace: [10688.328874] [10688.329820] ? show_regs.cold.9+0x22/0x2f [10688.331433] ? watchdog_timer_fn.cold.6+0x62/0x10a [10688.333505] ? __hrtimer_run_queues+0x1c2/0x4d0 [10688.335403] ? watchdog+0x40/0x40 [10688.336899] ? hrtimer_interrupt+0x128/0x2c0 [10688.338751] ? smp_apic_timer_interrupt+0xa6/0x270 [10688.340943] ? apic_timer_interrupt+0xf/0x20 [10688.342725] [10688.343696] ? native_safe_halt+0x2e/0x50 [10688.345490] kvm_wait+0xcd/0xe0 [10688.347492] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10688.350358] do_raw_spin_lock+0x177/0x190 [10688.352448] _raw_spin_lock+0x21/0x30 [10688.354436] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10688.357477] obd_mod_free+0x46/0x140 [ptlrpc] [10688.359802] mdc_commit_open+0x91/0xb0 [mdc] [10688.361533] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10688.363859] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10688.367693] after_reply+0xe2c/0x1780 [ptlrpc] [10688.371293] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10688.376470] ptlrpcd+0xa82/0xe40 [ptlrpc] [10688.377963] ? do_wait_intr+0xf0/0xf0 [10688.381639] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10688.386142] kthread+0x1d1/0x200 [10688.387702] ? set_kthread_struct+0x70/0x70 [10688.389450] ret_from_fork+0x1f/0x30 [10692.575055] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8733s! [10692.578101] Showing busy workqueues and worker pools: [10692.581284] workqueue events: flags=0x0 [10692.583059] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10692.591238] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [10692.595355] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [10692.598310] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [10692.602413] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10692.604518] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [10692.606909] workqueue events_long: flags=0x0 [10692.608484] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10692.611170] pending: lu_objects_flush [obdclass] [10692.613586] workqueue events_unbound: flags=0x2 [10692.615463] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [10692.618183] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10692.622371] workqueue events_power_efficient: flags=0x80 [10692.624962] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10692.627924] pending: neigh_periodic_work [10692.629725] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10692.632909] pending: do_cache_clean, check_lifetime [10692.635021] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10692.637694] pending: neigh_periodic_work [10692.639352] workqueue writeback: flags=0x4a [10692.641012] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10692.643478] pending: wb_workfn, wb_workfn [10692.645420] workqueue xprtiod: flags=0xa [10692.647079] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10692.649909] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [10692.653260] workqueue ipv6_addrconf: flags=0x40008 [10692.655876] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10692.658941] pending: addrconf_verify_work [10692.662036] workqueue knbd0-recv: flags=0x1a [10692.664042] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10692.668166] in-flight: 657:recv_work [10692.670124] workqueue ptlrpc_pinger: flags=0x2 [10692.672054] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10692.674972] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [10692.678325] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8733s workers=4 idle: 20831 12642 28859 [10692.681365] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10682s workers=2 idle: 659 [10708.060772] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [10708.069073] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10708.104666] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10708.113627] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10708.117911] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10708.123883] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10708.146669] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10708.149887] RAX: 00000000000038b6 RBX: ffff96848811cae0 RCX: 0000000000000001 [10708.160310] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [10708.172178] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [10708.178925] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [10708.185682] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [10708.191301] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [10708.198915] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10708.201895] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [10708.209813] Call Trace: [10708.211943] [10708.213932] ? show_regs.cold.9+0x22/0x2f [10708.216053] ? watchdog_timer_fn.cold.6+0x62/0x10a [10708.218210] ? __hrtimer_run_queues+0x1c2/0x4d0 [10708.219370] ? watchdog+0x40/0x40 [10708.224499] ? hrtimer_interrupt+0x128/0x2c0 [10708.226074] ? smp_apic_timer_interrupt+0xa6/0x270 [10708.228702] ? apic_timer_interrupt+0xf/0x20 [10708.233776] [10708.234651] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10708.239269] do_raw_spin_lock+0x177/0x190 [10708.241779] _raw_spin_lock+0x21/0x30 [10708.243442] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10708.248664] ? do_raw_spin_unlock+0x75/0x190 [10708.253500] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10708.263995] ? do_wait_intr+0xf0/0xf0 [10708.265286] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10708.268198] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10708.271362] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10708.275311] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10708.281373] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10708.285264] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10708.287585] mdc_intent_lock+0x353/0x7e0 [mdc] [10708.289936] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10708.299592] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10708.309287] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10708.315114] lmv_intent_open+0x338/0x1150 [lmv] [10708.318320] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10708.325160] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10708.332531] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10708.335796] ll_intent_lock+0xc5/0xaa0 [lustre] [10708.338177] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10708.341775] ll_lookup_it+0xe0e/0x2270 [lustre] [10708.344674] ? do_raw_spin_unlock+0x75/0x190 [10708.346232] ? _raw_spin_unlock+0x12/0x30 [10708.351080] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10708.357783] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10708.363740] ll_atomic_open+0x36a/0xd70 [lustre] [10708.365254] lookup_open+0x356/0xb40 [10708.366770] do_last+0x313/0xfc0 [10708.368854] ? nd_jump_root+0xe5/0x160 [10708.370688] ? path_init+0x437/0x520 [10708.373152] path_openat+0xf7/0x500 [10708.374683] ? do_raw_spin_unlock+0x75/0x190 [10708.376654] do_filp_open+0x99/0x140 [10708.378956] ? getname_flags+0x6e/0x330 [10708.380782] ? __check_object_size+0xff/0x256 [10708.383138] ? do_raw_spin_unlock+0x75/0x190 [10708.385696] ? _raw_spin_unlock+0x12/0x30 [10708.387711] do_sys_openat2+0x2b4/0x410 [10708.390468] do_sys_open+0x73/0xa0 [10708.391918] __x64_sys_openat+0x24/0x30 [10708.393644] do_syscall_64+0xc1/0x440 [10708.395129] entry_SYSCALL_64_after_hwframe+0x49/0xae [10708.397488] RIP: 0033:0x7ff54af45332 [10708.399511] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10708.409632] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10708.413430] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [10708.417159] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [10708.421813] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [10708.425443] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [10708.428179] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [10712.060273] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [10712.064276] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10712.084508] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10712.088832] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10712.092907] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10712.096257] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10712.100006] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10712.108736] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10712.112033] RAX: 0000000000007f0f RBX: ffff96848811cae0 RCX: 0000000000000001 [10712.115290] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [10712.118304] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [10712.121324] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [10712.124960] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [10712.128328] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [10712.132091] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10712.134591] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [10712.137201] Call Trace: [10712.138028] [10712.138876] ? show_regs.cold.9+0x22/0x2f [10712.140471] ? watchdog_timer_fn.cold.6+0x62/0x10a [10712.142481] ? __hrtimer_run_queues+0x1c2/0x4d0 [10712.144281] ? watchdog+0x40/0x40 [10712.145911] ? hrtimer_interrupt+0x128/0x2c0 [10712.147772] ? smp_apic_timer_interrupt+0xa6/0x270 [10712.149999] ? apic_timer_interrupt+0xf/0x20 [10712.152885] [10712.153721] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10712.157297] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10712.159247] do_raw_spin_lock+0x177/0x190 [10712.161081] _raw_spin_lock+0x21/0x30 [10712.162860] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10712.166011] process_one_work+0x2c8/0x700 [10712.168088] worker_thread+0x296/0x6e0 [10712.170415] ? rescuer_thread+0x570/0x570 [10712.171731] kthread+0x1d1/0x200 [10712.173222] ? set_kthread_struct+0x70/0x70 [10712.174484] ret_from_fork+0x1f/0x30 [10716.058064] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [10716.063990] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10716.096076] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10716.116205] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10716.127344] RIP: 0010:native_safe_halt+0x2e/0x50 [10716.129636] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10716.140545] RSP: 0018:ffffaf9fc1543ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10716.144644] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [10716.149215] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c20b4154 [10716.154891] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [10716.160897] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [10716.165754] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [10716.168467] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [10716.171925] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10716.178734] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [10716.183775] Call Trace: [10716.184775] [10716.185983] ? show_regs.cold.9+0x22/0x2f [10716.188524] ? watchdog_timer_fn.cold.6+0x62/0x10a [10716.194246] ? __hrtimer_run_queues+0x1c2/0x4d0 [10716.196799] ? watchdog+0x40/0x40 [10716.198970] ? hrtimer_interrupt+0x128/0x2c0 [10716.202292] ? smp_apic_timer_interrupt+0xa6/0x270 [10716.205945] ? apic_timer_interrupt+0xf/0x20 [10716.207461] [10716.209653] ? native_safe_halt+0x2e/0x50 [10716.211406] kvm_wait+0xcd/0xe0 [10716.213109] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10716.216830] do_raw_spin_lock+0x177/0x190 [10716.222177] _raw_spin_lock+0x21/0x30 [10716.224729] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10716.227976] ? do_raw_spin_unlock+0x75/0x190 [10716.232693] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10716.241244] ? do_wait_intr+0xf0/0xf0 [10716.243275] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10716.245819] mdc_close+0x53b/0xd70 [mdc] [10716.247836] lmv_close+0x18d/0x490 [lmv] [10716.249654] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [10716.252678] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10716.255825] ll_md_real_close+0xa5/0x2a0 [lustre] [10716.257728] ll_md_close+0x274/0x800 [lustre] [10716.259805] ll_file_release+0x128/0xc50 [lustre] [10716.264983] __fput+0x10d/0x440 [10716.267756] ____fput+0x12/0x20 [10716.271240] task_work_run+0xc6/0x110 [10716.277137] exit_to_usermode_loop+0x1dd/0x1f0 [10716.281981] do_syscall_64+0x3d6/0x440 [10716.285237] entry_SYSCALL_64_after_hwframe+0x49/0xae [10716.290952] RIP: 0033:0x7ff54af45cd8 [10716.292810] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10716.302725] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10716.310125] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [10716.315164] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [10716.320550] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [10716.324806] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [10716.328646] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10720.056559] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [10720.063702] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10720.095585] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10720.102592] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10720.108104] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10720.110424] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10720.119166] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10720.122842] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [10720.128032] RDX: 0000000000005f3c RSI: 0000000000000003 RDI: ffff96848811cae0 [10720.132862] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [10720.138577] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [10720.143866] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10720.147801] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [10720.151717] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10720.154043] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [10720.158058] Call Trace: [10720.159220] [10720.160410] ? show_regs.cold.9+0x22/0x2f [10720.162151] ? watchdog_timer_fn.cold.6+0x62/0x10a [10720.164251] ? __hrtimer_run_queues+0x1c2/0x4d0 [10720.166487] ? watchdog+0x40/0x40 [10720.168153] ? hrtimer_interrupt+0x128/0x2c0 [10720.170529] ? smp_apic_timer_interrupt+0xa6/0x270 [10720.172949] ? apic_timer_interrupt+0xf/0x20 [10720.175082] [10720.176213] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10720.180124] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10720.184064] do_raw_spin_lock+0x177/0x190 [10720.186078] _raw_spin_lock+0x21/0x30 [10720.190583] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10720.195116] obd_mod_free+0x46/0x140 [ptlrpc] [10720.198874] mdc_commit_open+0x91/0xb0 [mdc] [10720.201428] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10720.204875] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10720.208095] after_reply+0xe2c/0x1780 [ptlrpc] [10720.209981] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10720.212516] ptlrpcd+0xa82/0xe40 [ptlrpc] [10720.215155] ? do_wait_intr+0xf0/0xf0 [10720.216702] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10720.219868] kthread+0x1d1/0x200 [10720.220972] ? set_kthread_struct+0x70/0x70 [10720.223025] ret_from_fork+0x1f/0x30 [10722.783710] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8763s! [10722.791994] Showing busy workqueues and worker pools: [10722.797656] workqueue events: flags=0x0 [10722.804062] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10722.806432] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [10722.814105] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [10722.817307] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [10722.822452] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10722.825075] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [10722.827620] workqueue events_long: flags=0x0 [10722.829765] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10722.832381] pending: lu_objects_flush [obdclass] [10722.835047] workqueue events_unbound: flags=0x2 [10722.836921] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [10722.839128] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10722.843617] workqueue events_power_efficient: flags=0x80 [10722.845494] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10722.848757] pending: neigh_periodic_work [10722.850844] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10722.853988] pending: do_cache_clean, check_lifetime [10722.856666] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10722.860123] pending: neigh_periodic_work [10722.862689] workqueue writeback: flags=0x4a [10722.864767] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10722.867682] pending: wb_workfn, wb_workfn [10722.869901] workqueue xprtiod: flags=0xa [10722.871882] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10722.874729] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [10722.878256] workqueue ipv6_addrconf: flags=0x40008 [10722.880059] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10722.882483] pending: addrconf_verify_work [10722.883902] workqueue knbd0-recv: flags=0x1a [10722.885494] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10722.888610] in-flight: 657:recv_work [10722.890570] workqueue ptlrpc_pinger: flags=0x2 [10722.892461] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10722.895002] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [10722.898020] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8764s workers=4 idle: 20831 12642 28859 [10722.901874] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10713s workers=2 idle: 659 [10736.062778] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [10736.069720] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10736.105232] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10736.115336] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10736.121688] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10736.127198] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10736.141336] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10736.147268] RAX: 0000000000007b1a RBX: ffff96848811cae0 RCX: 0000000000000001 [10736.151757] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [10736.155576] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [10736.158914] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [10736.162622] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [10736.167523] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [10736.173744] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10736.178251] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [10736.181740] Call Trace: [10736.182865] [10736.184103] ? show_regs.cold.9+0x22/0x2f [10736.186120] ? watchdog_timer_fn.cold.6+0x62/0x10a [10736.189525] ? __hrtimer_run_queues+0x1c2/0x4d0 [10736.191860] ? watchdog+0x40/0x40 [10736.193476] ? hrtimer_interrupt+0x128/0x2c0 [10736.194688] ? smp_apic_timer_interrupt+0xa6/0x270 [10736.199507] ? apic_timer_interrupt+0xf/0x20 [10736.202307] [10736.202877] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10736.207910] do_raw_spin_lock+0x177/0x190 [10736.209505] _raw_spin_lock+0x21/0x30 [10736.213472] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10736.217361] ? do_raw_spin_unlock+0x75/0x190 [10736.218655] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10736.223803] ? do_wait_intr+0xf0/0xf0 [10736.225439] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10736.228688] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10736.231610] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10736.236918] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10736.241440] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10736.244520] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10736.246750] mdc_intent_lock+0x353/0x7e0 [mdc] [10736.248926] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10736.251480] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10736.254007] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10736.256836] lmv_intent_open+0x338/0x1150 [lmv] [10736.259069] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10736.261800] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10736.264523] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10736.270313] ll_intent_lock+0xc5/0xaa0 [lustre] [10736.272449] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10736.274851] ll_lookup_it+0xe0e/0x2270 [lustre] [10736.276864] ? do_raw_spin_unlock+0x75/0x190 [10736.278961] ? _raw_spin_unlock+0x12/0x30 [10736.280628] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10736.283346] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10736.289537] ll_atomic_open+0x36a/0xd70 [lustre] [10736.291653] lookup_open+0x356/0xb40 [10736.293507] do_last+0x313/0xfc0 [10736.295631] ? nd_jump_root+0xe5/0x160 [10736.298358] ? path_init+0x437/0x520 [10736.299968] path_openat+0xf7/0x500 [10736.301158] ? do_raw_spin_unlock+0x75/0x190 [10736.303100] do_filp_open+0x99/0x140 [10736.304804] ? getname_flags+0x6e/0x330 [10736.308960] ? __check_object_size+0xff/0x256 [10736.312783] ? do_raw_spin_unlock+0x75/0x190 [10736.315739] ? _raw_spin_unlock+0x12/0x30 [10736.317807] do_sys_openat2+0x2b4/0x410 [10736.319785] do_sys_open+0x73/0xa0 [10736.320940] __x64_sys_openat+0x24/0x30 [10736.322100] do_syscall_64+0xc1/0x440 [10736.323340] entry_SYSCALL_64_after_hwframe+0x49/0xae [10736.325007] RIP: 0033:0x7ff54af45332 [10736.326364] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10736.332302] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10736.335697] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [10736.337977] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [10736.340783] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [10736.344029] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [10736.347691] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [10740.059377] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [10740.063986] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10740.098160] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10740.102608] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10740.106417] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10740.109186] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10740.111451] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10740.119112] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10740.121850] RAX: 0000000000007051 RBX: ffff96848811cae0 RCX: 0000000000000001 [10740.124880] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [10740.136891] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [10740.141788] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [10740.147620] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [10740.155818] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [10740.161542] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10740.166693] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [10740.170564] Call Trace: [10740.177618] [10740.179857] ? show_regs.cold.9+0x22/0x2f [10740.186226] ? watchdog_timer_fn.cold.6+0x62/0x10a [10740.192653] ? __hrtimer_run_queues+0x1c2/0x4d0 [10740.196624] ? watchdog+0x40/0x40 [10740.200563] ? hrtimer_interrupt+0x128/0x2c0 [10740.204913] ? smp_apic_timer_interrupt+0xa6/0x270 [10740.212444] ? apic_timer_interrupt+0xf/0x20 [10740.219442] [10740.220750] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10740.225861] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10740.229599] do_raw_spin_lock+0x177/0x190 [10740.231925] _raw_spin_lock+0x21/0x30 [10740.233918] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10740.237775] process_one_work+0x2c8/0x700 [10740.246419] worker_thread+0x296/0x6e0 [10740.250188] ? rescuer_thread+0x570/0x570 [10740.251653] kthread+0x1d1/0x200 [10740.252622] ? set_kthread_struct+0x70/0x70 [10740.264565] ret_from_fork+0x1f/0x30 [10744.058779] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [10744.063374] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10744.097516] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10744.104813] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10744.111606] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10744.116333] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10744.126566] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10744.128917] RAX: 0000000000004381 RBX: ffff96848811cae0 RCX: 0000000000000001 [10744.135805] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [10744.153780] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [10744.157942] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [10744.164046] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [10744.167945] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [10744.173821] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10744.178517] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [10744.182944] Call Trace: [10744.183807] [10744.184937] ? show_regs.cold.9+0x22/0x2f [10744.192945] ? watchdog_timer_fn.cold.6+0x62/0x10a [10744.204506] ? __hrtimer_run_queues+0x1c2/0x4d0 [10744.218031] ? watchdog+0x40/0x40 [10744.222049] ? hrtimer_interrupt+0x128/0x2c0 [10744.222951] ? smp_apic_timer_interrupt+0xa6/0x270 [10744.224022] ? apic_timer_interrupt+0xf/0x20 [10744.229451] [10744.230096] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10744.232756] do_raw_spin_lock+0x177/0x190 [10744.233950] _raw_spin_lock+0x21/0x30 [10744.238425] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10744.248719] ? do_raw_spin_unlock+0x75/0x190 [10744.249674] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10744.250894] ? do_wait_intr+0xf0/0xf0 [10744.260731] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10744.262182] mdc_close+0x53b/0xd70 [mdc] [10744.266151] lmv_close+0x18d/0x490 [lmv] [10744.267907] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [10744.271460] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10744.276519] ll_md_real_close+0xa5/0x2a0 [lustre] [10744.279386] ll_md_close+0x274/0x800 [lustre] [10744.283303] ll_file_release+0x128/0xc50 [lustre] [10744.284995] __fput+0x10d/0x440 [10744.288334] ____fput+0x12/0x20 [10744.290415] task_work_run+0xc6/0x110 [10744.294521] exit_to_usermode_loop+0x1dd/0x1f0 [10744.298435] do_syscall_64+0x3d6/0x440 [10744.300658] entry_SYSCALL_64_after_hwframe+0x49/0xae [10744.308784] RIP: 0033:0x7ff54af45cd8 [10744.310967] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10744.335658] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10744.342459] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [10744.356748] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [10744.364047] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [10744.369714] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [10744.376876] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10748.055025] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [10748.060030] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10748.090145] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10748.101701] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10748.114175] RIP: 0010:native_safe_halt+0x2e/0x50 [10748.118680] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10748.129187] RSP: 0000:ffffaf9fc1b97bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10748.134490] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [10748.137844] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff96848811cae0 [10748.141524] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [10748.145263] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [10748.149556] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10748.153104] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [10748.159799] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10748.162952] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [10748.169770] Call Trace: [10748.173191] [10748.177333] ? show_regs.cold.9+0x22/0x2f [10748.183448] ? watchdog_timer_fn.cold.6+0x62/0x10a [10748.189595] ? __hrtimer_run_queues+0x1c2/0x4d0 [10748.191893] ? watchdog+0x40/0x40 [10748.196852] ? hrtimer_interrupt+0x128/0x2c0 [10748.202957] ? smp_apic_timer_interrupt+0xa6/0x270 [10748.209988] ? apic_timer_interrupt+0xf/0x20 [10748.216940] [10748.221508] ? native_safe_halt+0x2e/0x50 [10748.226103] kvm_wait+0xcd/0xe0 [10748.232160] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10748.240931] do_raw_spin_lock+0x177/0x190 [10748.246001] _raw_spin_lock+0x21/0x30 [10748.247839] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10748.253092] obd_mod_free+0x46/0x140 [ptlrpc] [10748.258276] mdc_commit_open+0x91/0xb0 [mdc] [10748.263844] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10748.272226] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10748.284459] after_reply+0xe2c/0x1780 [ptlrpc] [10748.286412] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10748.295010] ptlrpcd+0xa82/0xe40 [ptlrpc] [10748.302072] ? do_wait_intr+0xf0/0xf0 [10748.307747] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10748.317902] kthread+0x1d1/0x200 [10748.319841] ? set_kthread_struct+0x70/0x70 [10748.321137] ret_from_fork+0x1f/0x30 [10752.992750] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8794s! [10753.002645] Showing busy workqueues and worker pools: [10753.007197] workqueue events: flags=0x0 [10753.009568] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10753.020333] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [10753.029891] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [10753.038229] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [10753.046917] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10753.052067] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [10753.055232] workqueue events_long: flags=0x0 [10753.056750] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10753.066460] pending: lu_objects_flush [obdclass] [10753.080812] workqueue events_unbound: flags=0x2 [10753.087647] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [10753.093639] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10753.099075] workqueue events_power_efficient: flags=0x80 [10753.103475] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10753.109895] pending: neigh_periodic_work [10753.113803] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10753.117371] pending: do_cache_clean, check_lifetime [10753.119378] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10753.130952] pending: neigh_periodic_work [10753.132544] workqueue writeback: flags=0x4a [10753.133987] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10753.141914] pending: wb_workfn, wb_workfn [10753.144226] workqueue xprtiod: flags=0xa [10753.146099] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10753.149969] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [10753.153310] workqueue ipv6_addrconf: flags=0x40008 [10753.157246] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10753.164248] pending: addrconf_verify_work [10753.166293] workqueue knbd0-recv: flags=0x1a [10753.168622] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10753.175586] in-flight: 657:recv_work [10753.176922] workqueue ptlrpc_pinger: flags=0x2 [10753.182930] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10753.188664] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [10753.192337] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8794s workers=4 idle: 20831 12642 28859 [10753.198226] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10743s workers=2 idle: 659 [10764.063770] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [10764.068184] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10764.118843] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10764.131278] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10764.143347] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10764.144749] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10764.160828] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10764.169191] RAX: 000000000000750e RBX: ffff96848811cae0 RCX: 0000000000000001 [10764.178532] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [10764.188917] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [10764.200822] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [10764.210070] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [10764.214658] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [10764.220764] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10764.229609] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [10764.238877] Call Trace: [10764.239674] [10764.248479] ? show_regs.cold.9+0x22/0x2f [10764.251661] ? watchdog_timer_fn.cold.6+0x62/0x10a [10764.256863] ? __hrtimer_run_queues+0x1c2/0x4d0 [10764.262464] ? watchdog+0x40/0x40 [10764.269226] ? hrtimer_interrupt+0x128/0x2c0 [10764.273504] ? smp_apic_timer_interrupt+0xa6/0x270 [10764.276079] ? apic_timer_interrupt+0xf/0x20 [10764.277736] [10764.279487] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10764.283255] do_raw_spin_lock+0x177/0x190 [10764.288500] _raw_spin_lock+0x21/0x30 [10764.291392] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10764.302644] ? do_raw_spin_unlock+0x75/0x190 [10764.309069] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10764.327354] ? do_wait_intr+0xf0/0xf0 [10764.331491] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10764.349062] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10764.359403] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10764.361608] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10764.380421] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10764.390518] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10764.399508] mdc_intent_lock+0x353/0x7e0 [mdc] [10764.404663] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10764.419219] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10764.427365] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10764.431336] lmv_intent_open+0x338/0x1150 [lmv] [10764.434409] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10764.438312] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10764.443418] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10764.445987] ll_intent_lock+0xc5/0xaa0 [lustre] [10764.448944] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10764.452031] ll_lookup_it+0xe0e/0x2270 [lustre] [10764.455003] ? do_raw_spin_unlock+0x75/0x190 [10764.457920] ? _raw_spin_unlock+0x12/0x30 [10764.460207] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10764.463255] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10764.466308] ll_atomic_open+0x36a/0xd70 [lustre] [10764.473002] lookup_open+0x356/0xb40 [10764.477862] do_last+0x313/0xfc0 [10764.481589] ? nd_jump_root+0xe5/0x160 [10764.482593] ? path_init+0x437/0x520 [10764.483497] path_openat+0xf7/0x500 [10764.488440] ? do_raw_spin_unlock+0x75/0x190 [10764.499510] do_filp_open+0x99/0x140 [10764.505461] ? getname_flags+0x6e/0x330 [10764.506966] ? __check_object_size+0xff/0x256 [10764.516319] ? do_raw_spin_unlock+0x75/0x190 [10764.519423] ? _raw_spin_unlock+0x12/0x30 [10764.525348] do_sys_openat2+0x2b4/0x410 [10764.528561] do_sys_open+0x73/0xa0 [10764.531394] __x64_sys_openat+0x24/0x30 [10764.540279] do_syscall_64+0xc1/0x440 [10764.549235] entry_SYSCALL_64_after_hwframe+0x49/0xae [10764.552876] RIP: 0033:0x7ff54af45332 [10764.557494] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10764.590821] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10764.601729] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [10764.609421] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [10764.625616] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [10764.641083] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [10764.643588] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [10768.060519] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [10768.069384] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10768.119686] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10768.130574] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10768.142240] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10768.154597] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10768.160599] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10768.179654] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10768.187432] RAX: 00000000000037d3 RBX: ffff96848811cae0 RCX: 0000000000000001 [10768.193530] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [10768.199273] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [10768.205150] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [10768.209420] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [10768.214522] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [10768.218363] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10768.223337] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [10768.228187] Call Trace: [10768.230369] [10768.231486] ? show_regs.cold.9+0x22/0x2f [10768.236147] ? watchdog_timer_fn.cold.6+0x62/0x10a [10768.239638] ? __hrtimer_run_queues+0x1c2/0x4d0 [10768.253401] ? watchdog+0x40/0x40 [10768.259761] ? hrtimer_interrupt+0x128/0x2c0 [10768.263640] ? smp_apic_timer_interrupt+0xa6/0x270 [10768.269571] ? apic_timer_interrupt+0xf/0x20 [10768.274845] [10768.275613] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10768.292708] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10768.302437] do_raw_spin_lock+0x177/0x190 [10768.307464] _raw_spin_lock+0x21/0x30 [10768.309561] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10768.318181] process_one_work+0x2c8/0x700 [10768.323686] worker_thread+0x296/0x6e0 [10768.329273] ? rescuer_thread+0x570/0x570 [10768.333543] kthread+0x1d1/0x200 [10768.340443] ? set_kthread_struct+0x70/0x70 [10768.344579] ret_from_fork+0x1f/0x30 [10772.059703] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [10772.072626] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10772.129395] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10772.146355] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10772.162727] RIP: 0010:native_safe_halt+0x2e/0x50 [10772.168486] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10772.187661] RSP: 0018:ffffaf9fc1543ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10772.193750] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [10772.200494] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c20b4154 [10772.215385] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [10772.232076] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [10772.234863] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [10772.236909] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [10772.239573] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10772.242310] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [10772.251371] Call Trace: [10772.253683] [10772.255139] ? show_regs.cold.9+0x22/0x2f [10772.257864] ? watchdog_timer_fn.cold.6+0x62/0x10a [10772.270696] ? __hrtimer_run_queues+0x1c2/0x4d0 [10772.278382] ? watchdog+0x40/0x40 [10772.283126] ? hrtimer_interrupt+0x128/0x2c0 [10772.285881] ? smp_apic_timer_interrupt+0xa6/0x270 [10772.288820] ? apic_timer_interrupt+0xf/0x20 [10772.291635] [10772.293032] ? native_safe_halt+0x2e/0x50 [10772.294570] kvm_wait+0xcd/0xe0 [10772.300134] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10772.304495] do_raw_spin_lock+0x177/0x190 [10772.308382] _raw_spin_lock+0x21/0x30 [10772.314253] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10772.320689] ? do_raw_spin_unlock+0x75/0x190 [10772.332736] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10772.342384] ? do_wait_intr+0xf0/0xf0 [10772.346140] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10772.353533] mdc_close+0x53b/0xd70 [mdc] [10772.358772] lmv_close+0x18d/0x490 [lmv] [10772.362561] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [10772.366322] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10772.375379] ll_md_real_close+0xa5/0x2a0 [lustre] [10772.381648] ll_md_close+0x274/0x800 [lustre] [10772.385521] ll_file_release+0x128/0xc50 [lustre] [10772.392579] __fput+0x10d/0x440 [10772.397199] ____fput+0x12/0x20 [10772.398514] task_work_run+0xc6/0x110 [10772.401581] exit_to_usermode_loop+0x1dd/0x1f0 [10772.403510] do_syscall_64+0x3d6/0x440 [10772.407867] entry_SYSCALL_64_after_hwframe+0x49/0xae [10772.411033] RIP: 0033:0x7ff54af45cd8 [10772.413172] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10772.426509] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10772.430613] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [10772.439008] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [10772.447180] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [10772.455120] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [10772.459503] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10780.055129] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [10780.059292] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10780.087334] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10780.095557] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10780.101742] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10780.106090] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10780.126216] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10780.141091] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [10780.147169] RDX: 000000000000368c RSI: 0000000000000003 RDI: ffff96848811cae0 [10780.154507] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [10780.163354] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [10780.172940] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10780.182673] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [10780.193502] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10780.198765] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [10780.210084] Call Trace: [10780.214341] [10780.215158] ? show_regs.cold.9+0x22/0x2f [10780.221527] ? watchdog_timer_fn.cold.6+0x62/0x10a [10780.234788] ? __hrtimer_run_queues+0x1c2/0x4d0 [10780.240604] ? watchdog+0x40/0x40 [10780.246486] ? hrtimer_interrupt+0x128/0x2c0 [10780.249634] ? smp_apic_timer_interrupt+0xa6/0x270 [10780.252718] ? apic_timer_interrupt+0xf/0x20 [10780.257757] [10780.259165] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10780.262408] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10780.265699] do_raw_spin_lock+0x177/0x190 [10780.268180] _raw_spin_lock+0x21/0x30 [10780.270427] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10780.273937] obd_mod_free+0x46/0x140 [ptlrpc] [10780.286114] mdc_commit_open+0x91/0xb0 [mdc] [10780.292826] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10780.297626] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10780.301786] after_reply+0xe2c/0x1780 [ptlrpc] [10780.305287] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10780.311096] ptlrpcd+0xa82/0xe40 [ptlrpc] [10780.315632] ? do_wait_intr+0xf0/0xf0 [10780.318274] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10780.321671] kthread+0x1d1/0x200 [10780.323807] ? set_kthread_struct+0x70/0x70 [10780.327356] ret_from_fork+0x1f/0x30 [10783.711105] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8824s! [10783.715021] Showing busy workqueues and worker pools: [10783.717500] workqueue events: flags=0x0 [10783.719256] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10783.723147] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [10783.728211] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [10783.731770] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [10783.737972] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10783.741624] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [10783.745208] workqueue events_long: flags=0x0 [10783.747484] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10783.751334] pending: lu_objects_flush [obdclass] [10783.754403] workqueue events_unbound: flags=0x2 [10783.756955] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [10783.760942] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10783.769292] workqueue events_power_efficient: flags=0x80 [10783.772466] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10783.776288] pending: neigh_periodic_work [10783.778573] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10783.782266] pending: do_cache_clean, check_lifetime [10783.785410] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10783.789274] pending: neigh_periodic_work [10783.791904] workqueue writeback: flags=0x4a [10783.794340] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10783.798002] pending: wb_workfn, wb_workfn [10783.800376] workqueue xprtiod: flags=0xa [10783.802608] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10783.806301] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [10783.810484] workqueue ipv6_addrconf: flags=0x40008 [10783.813447] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10783.817398] pending: addrconf_verify_work [10783.819864] workqueue knbd0-recv: flags=0x1a [10783.822324] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10783.826304] in-flight: 657:recv_work [10783.831430] workqueue ptlrpc_pinger: flags=0x2 [10783.835769] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10783.839927] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [10783.843622] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8824s workers=4 idle: 20831 12642 28859 [10783.848828] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10774s workers=2 idle: 659 [10792.061565] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [10792.069467] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10792.141853] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10792.152847] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10792.166413] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10792.172771] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10792.193133] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10792.216055] RAX: 0000000000001a05 RBX: ffff96848811cae0 RCX: 0000000000000001 [10792.229901] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [10792.239240] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [10792.244913] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [10792.252675] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [10792.261692] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [10792.274899] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10792.282421] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [10792.288018] Call Trace: [10792.292442] [10792.298432] ? show_regs.cold.9+0x22/0x2f [10792.302983] ? watchdog_timer_fn.cold.6+0x62/0x10a [10792.309067] ? __hrtimer_run_queues+0x1c2/0x4d0 [10792.311131] ? watchdog+0x40/0x40 [10792.319047] ? hrtimer_interrupt+0x128/0x2c0 [10792.322442] ? smp_apic_timer_interrupt+0xa6/0x270 [10792.332137] ? apic_timer_interrupt+0xf/0x20 [10792.340912] [10792.343780] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10792.349461] do_raw_spin_lock+0x177/0x190 [10792.352554] _raw_spin_lock+0x21/0x30 [10792.357912] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10792.370215] ? do_raw_spin_unlock+0x75/0x190 [10792.374225] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10792.383274] ? do_wait_intr+0xf0/0xf0 [10792.387716] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10792.396310] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10792.403223] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10792.408076] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10792.411687] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10792.417324] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10792.419798] mdc_intent_lock+0x353/0x7e0 [mdc] [10792.422153] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10792.429769] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10792.436048] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10792.438212] lmv_intent_open+0x338/0x1150 [lmv] [10792.444446] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10792.454837] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10792.461155] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10792.462758] ll_intent_lock+0xc5/0xaa0 [lustre] [10792.470635] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10792.483805] ll_lookup_it+0xe0e/0x2270 [lustre] [10792.493560] ? do_raw_spin_unlock+0x75/0x190 [10792.496416] ? _raw_spin_unlock+0x12/0x30 [10792.508531] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10792.514497] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10792.519912] ll_atomic_open+0x36a/0xd70 [lustre] [10792.528866] lookup_open+0x356/0xb40 [10792.543328] do_last+0x313/0xfc0 [10792.548282] ? nd_jump_root+0xe5/0x160 [10792.553389] ? path_init+0x437/0x520 [10792.564175] path_openat+0xf7/0x500 [10792.571433] ? do_raw_spin_unlock+0x75/0x190 [10792.576282] do_filp_open+0x99/0x140 [10792.587541] ? getname_flags+0x6e/0x330 [10792.601414] ? __check_object_size+0xff/0x256 [10792.610518] ? do_raw_spin_unlock+0x75/0x190 [10792.619408] ? _raw_spin_unlock+0x12/0x30 [10792.627343] do_sys_openat2+0x2b4/0x410 [10792.638082] do_sys_open+0x73/0xa0 [10792.640442] __x64_sys_openat+0x24/0x30 [10792.646251] do_syscall_64+0xc1/0x440 [10792.655266] entry_SYSCALL_64_after_hwframe+0x49/0xae [10792.667491] RIP: 0033:0x7ff54af45332 [10792.669725] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10792.694092] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10792.705880] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [10792.715660] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [10792.725518] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [10792.735068] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [10792.741412] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [10796.059029] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [10796.069205] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10796.137370] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10796.153199] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10796.174418] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10796.191583] RIP: 0010:native_safe_halt+0x2e/0x50 [10796.205036] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10796.245539] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10796.252026] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [10796.258966] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c2134154 [10796.265116] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [10796.276469] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [10796.287235] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [10796.291188] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [10796.297499] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10796.305931] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [10796.312941] Call Trace: [10796.314594] [10796.317048] ? show_regs.cold.9+0x22/0x2f [10796.322938] ? watchdog_timer_fn.cold.6+0x62/0x10a [10796.332049] ? __hrtimer_run_queues+0x1c2/0x4d0 [10796.335113] ? watchdog+0x40/0x40 [10796.339059] ? hrtimer_interrupt+0x128/0x2c0 [10796.340562] ? smp_apic_timer_interrupt+0xa6/0x270 [10796.351890] ? apic_timer_interrupt+0xf/0x20 [10796.362948] [10796.367022] ? native_safe_halt+0x2e/0x50 [10796.378218] kvm_wait+0xcd/0xe0 [10796.386052] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10796.398844] do_raw_spin_lock+0x177/0x190 [10796.403830] _raw_spin_lock+0x21/0x30 [10796.408425] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10796.416538] process_one_work+0x2c8/0x700 [10796.421160] worker_thread+0x296/0x6e0 [10796.429890] ? rescuer_thread+0x570/0x570 [10796.434034] kthread+0x1d1/0x200 [10796.439586] ? set_kthread_struct+0x70/0x70 [10796.445309] ret_from_fork+0x1f/0x30 [10800.061042] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [10800.066908] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10800.128398] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10800.146073] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10800.150386] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10800.158481] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10800.191675] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [10800.202511] RAX: 000000000000791b RBX: ffff96848811cae0 RCX: 0000000000000001 [10800.210025] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [10800.217464] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [10800.224360] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [10800.244129] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [10800.257292] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [10800.269403] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10800.277589] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [10800.297759] Call Trace: [10800.303760] [10800.309354] ? show_regs.cold.9+0x22/0x2f [10800.318312] ? watchdog_timer_fn.cold.6+0x62/0x10a [10800.319741] ? __hrtimer_run_queues+0x1c2/0x4d0 [10800.328482] ? watchdog+0x40/0x40 [10800.331053] ? hrtimer_interrupt+0x128/0x2c0 [10800.335222] ? smp_apic_timer_interrupt+0xa6/0x270 [10800.341186] ? apic_timer_interrupt+0xf/0x20 [10800.343205] [10800.343831] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10800.346789] do_raw_spin_lock+0x177/0x190 [10800.348247] _raw_spin_lock+0x21/0x30 [10800.351588] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10800.359461] ? do_raw_spin_unlock+0x75/0x190 [10800.364211] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10800.368697] ? do_wait_intr+0xf0/0xf0 [10800.378261] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10800.387156] mdc_close+0x53b/0xd70 [mdc] [10800.391024] lmv_close+0x18d/0x490 [lmv] [10800.398206] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [10800.403719] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10800.417943] ll_md_real_close+0xa5/0x2a0 [lustre] [10800.419456] ll_md_close+0x274/0x800 [lustre] [10800.427457] ll_file_release+0x128/0xc50 [lustre] [10800.437692] __fput+0x10d/0x440 [10800.440596] ____fput+0x12/0x20 [10800.442277] task_work_run+0xc6/0x110 [10800.446719] exit_to_usermode_loop+0x1dd/0x1f0 [10800.455737] do_syscall_64+0x3d6/0x440 [10800.459738] entry_SYSCALL_64_after_hwframe+0x49/0xae [10800.464312] RIP: 0033:0x7ff54af45cd8 [10800.470139] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10800.488796] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10800.494461] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [10800.501436] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [10800.520313] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [10800.538350] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [10800.543458] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10808.055022] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [10808.058300] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10808.084241] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10808.092769] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10808.097316] RIP: 0010:native_safe_halt+0x2e/0x50 [10808.099947] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10808.109246] RSP: 0000:ffffaf9fc1b97bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10808.112329] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [10808.114577] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff96848811cae0 [10808.116794] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [10808.119306] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [10808.121164] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10808.123385] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [10808.125499] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10808.127382] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [10808.129678] Call Trace: [10808.130676] [10808.131270] ? show_regs.cold.9+0x22/0x2f [10808.132592] ? watchdog_timer_fn.cold.6+0x62/0x10a [10808.134350] ? __hrtimer_run_queues+0x1c2/0x4d0 [10808.135708] ? watchdog+0x40/0x40 [10808.136627] ? hrtimer_interrupt+0x128/0x2c0 [10808.137817] ? smp_apic_timer_interrupt+0xa6/0x270 [10808.139396] ? apic_timer_interrupt+0xf/0x20 [10808.140577] [10808.141228] ? native_safe_halt+0x2e/0x50 [10808.142578] kvm_wait+0xcd/0xe0 [10808.143466] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10808.145127] do_raw_spin_lock+0x177/0x190 [10808.146315] _raw_spin_lock+0x21/0x30 [10808.147317] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10808.149095] obd_mod_free+0x46/0x140 [ptlrpc] [10808.150799] mdc_commit_open+0x91/0xb0 [mdc] [10808.152023] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10808.154011] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10808.155722] after_reply+0xe2c/0x1780 [ptlrpc] [10808.159731] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10808.162349] ptlrpcd+0xa82/0xe40 [ptlrpc] [10808.165192] ? do_wait_intr+0xf0/0xf0 [10808.169746] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10808.174067] kthread+0x1d1/0x200 [10808.177335] ? set_kthread_struct+0x70/0x70 [10808.178557] ret_from_fork+0x1f/0x30 [10813.919153] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8855s! [10813.924514] Showing busy workqueues and worker pools: [10813.926826] workqueue events: flags=0x0 [10813.928860] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10813.932869] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [10813.940330] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [10813.944390] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [10813.950776] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10813.953938] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [10813.957517] workqueue events_long: flags=0x0 [10813.961635] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10813.967391] pending: lu_objects_flush [obdclass] [10813.970622] workqueue events_unbound: flags=0x2 [10813.973271] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [10813.977094] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10813.981875] workqueue events_power_efficient: flags=0x80 [10813.984336] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10813.988028] pending: neigh_periodic_work [10813.991152] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10813.995484] pending: do_cache_clean, check_lifetime [10813.999049] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10814.003468] pending: neigh_periodic_work [10814.006052] workqueue writeback: flags=0x4a [10814.008685] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10814.011192] pending: wb_workfn, wb_workfn [10814.013053] workqueue xprtiod: flags=0xa [10814.014879] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10814.018102] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [10814.021922] workqueue ipv6_addrconf: flags=0x40008 [10814.024099] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10814.027468] pending: addrconf_verify_work [10814.029342] workqueue knbd0-recv: flags=0x1a [10814.031154] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10814.033938] in-flight: 657:recv_work [10814.035859] workqueue ptlrpc_pinger: flags=0x2 [10814.037815] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10814.040469] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [10814.043253] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8855s workers=4 idle: 20831 12642 28859 [10814.046968] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10804s workers=2 idle: 659 [10820.062254] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [10820.067037] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10820.102865] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10820.110808] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10820.124243] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10820.127973] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10820.137820] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10820.142725] RAX: 0000000000003984 RBX: ffff96848811cae0 RCX: 0000000000000001 [10820.147691] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [10820.152931] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [10820.156869] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [10820.159840] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [10820.162798] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [10820.170142] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10820.172083] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [10820.174703] Call Trace: [10820.176420] [10820.177893] ? show_regs.cold.9+0x22/0x2f [10820.180869] ? watchdog_timer_fn.cold.6+0x62/0x10a [10820.184081] ? __hrtimer_run_queues+0x1c2/0x4d0 [10820.186259] ? watchdog+0x40/0x40 [10820.187815] ? hrtimer_interrupt+0x128/0x2c0 [10820.189417] ? smp_apic_timer_interrupt+0xa6/0x270 [10820.193563] ? apic_timer_interrupt+0xf/0x20 [10820.194962] [10820.195681] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10820.198184] do_raw_spin_lock+0x177/0x190 [10820.203817] _raw_spin_lock+0x21/0x30 [10820.205040] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10820.210922] ? do_raw_spin_unlock+0x75/0x190 [10820.217031] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10820.223717] ? do_wait_intr+0xf0/0xf0 [10820.227681] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10820.231956] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10820.234818] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10820.239088] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10820.241030] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10820.242944] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10820.245691] mdc_intent_lock+0x353/0x7e0 [mdc] [10820.247536] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10820.255618] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10820.259248] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10820.266599] lmv_intent_open+0x338/0x1150 [lmv] [10820.268924] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10820.275510] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10820.278074] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10820.281027] ll_intent_lock+0xc5/0xaa0 [lustre] [10820.284091] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10820.287645] ll_lookup_it+0xe0e/0x2270 [lustre] [10820.289134] ? do_raw_spin_unlock+0x75/0x190 [10820.290372] ? _raw_spin_unlock+0x12/0x30 [10820.293972] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10820.299916] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10820.305657] ll_atomic_open+0x36a/0xd70 [lustre] [10820.309580] lookup_open+0x356/0xb40 [10820.310962] do_last+0x313/0xfc0 [10820.321857] ? nd_jump_root+0xe5/0x160 [10820.323187] ? path_init+0x437/0x520 [10820.324416] path_openat+0xf7/0x500 [10820.336831] ? do_raw_spin_unlock+0x75/0x190 [10820.339971] do_filp_open+0x99/0x140 [10820.344374] ? getname_flags+0x6e/0x330 [10820.345727] ? __check_object_size+0xff/0x256 [10820.351055] ? do_raw_spin_unlock+0x75/0x190 [10820.354869] ? _raw_spin_unlock+0x12/0x30 [10820.359240] do_sys_openat2+0x2b4/0x410 [10820.364446] do_sys_open+0x73/0xa0 [10820.366748] __x64_sys_openat+0x24/0x30 [10820.368811] do_syscall_64+0xc1/0x440 [10820.371256] entry_SYSCALL_64_after_hwframe+0x49/0xae [10820.377807] RIP: 0033:0x7ff54af45332 [10820.380764] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10820.403080] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10820.418565] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [10820.423621] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [10820.429767] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [10820.433616] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [10820.440091] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [10824.061182] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [10824.072097] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10824.135127] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10824.148544] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10824.156662] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10824.162134] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10824.167800] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10824.189602] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10824.201547] RAX: 0000000000007694 RBX: ffff96848811cae0 RCX: 0000000000000001 [10824.206574] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [10824.211649] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [10824.216578] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [10824.221606] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [10824.226539] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [10824.232102] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10824.236118] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [10824.241192] Call Trace: [10824.242911] [10824.244512] ? show_regs.cold.9+0x22/0x2f [10824.247078] ? watchdog_timer_fn.cold.6+0x62/0x10a [10824.249358] ? __hrtimer_run_queues+0x1c2/0x4d0 [10824.252783] ? watchdog+0x40/0x40 [10824.255283] ? hrtimer_interrupt+0x128/0x2c0 [10824.258205] ? smp_apic_timer_interrupt+0xa6/0x270 [10824.261734] ? apic_timer_interrupt+0xf/0x20 [10824.264600] [10824.265939] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10824.269379] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10824.273383] do_raw_spin_lock+0x177/0x190 [10824.276077] _raw_spin_lock+0x21/0x30 [10824.278465] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10824.282227] process_one_work+0x2c8/0x700 [10824.285045] worker_thread+0x296/0x6e0 [10824.287812] ? rescuer_thread+0x570/0x570 [10824.290843] kthread+0x1d1/0x200 [10824.293071] ? set_kthread_struct+0x70/0x70 [10824.295851] ret_from_fork+0x1f/0x30 [10828.058513] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [10828.062249] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10828.093578] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10828.099801] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10828.104676] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10828.107071] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10828.118279] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10828.124359] RAX: 0000000000006185 RBX: ffff96848811cae0 RCX: 0000000000000001 [10828.133358] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [10828.135169] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [10828.136910] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [10828.142719] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [10828.145336] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [10828.154770] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10828.159438] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [10828.164137] Call Trace: [10828.165284] [10828.166165] ? show_regs.cold.9+0x22/0x2f [10828.169774] ? watchdog_timer_fn.cold.6+0x62/0x10a [10828.172692] ? __hrtimer_run_queues+0x1c2/0x4d0 [10828.179051] ? watchdog+0x40/0x40 [10828.180135] ? hrtimer_interrupt+0x128/0x2c0 [10828.181748] ? smp_apic_timer_interrupt+0xa6/0x270 [10828.186760] ? apic_timer_interrupt+0xf/0x20 [10828.188049] [10828.195497] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10828.197360] do_raw_spin_lock+0x177/0x190 [10828.201588] _raw_spin_lock+0x21/0x30 [10828.204528] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10828.209422] ? do_raw_spin_unlock+0x75/0x190 [10828.211339] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10828.215824] ? do_wait_intr+0xf0/0xf0 [10828.224261] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10828.226255] mdc_close+0x53b/0xd70 [mdc] [10828.235125] lmv_close+0x18d/0x490 [lmv] [10828.239017] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [10828.243362] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10828.247800] ll_md_real_close+0xa5/0x2a0 [lustre] [10828.250588] ll_md_close+0x274/0x800 [lustre] [10828.258675] ll_file_release+0x128/0xc50 [lustre] [10828.260588] __fput+0x10d/0x440 [10828.269493] ____fput+0x12/0x20 [10828.270493] task_work_run+0xc6/0x110 [10828.273511] exit_to_usermode_loop+0x1dd/0x1f0 [10828.275484] do_syscall_64+0x3d6/0x440 [10828.276906] entry_SYSCALL_64_after_hwframe+0x49/0xae [10828.282999] RIP: 0033:0x7ff54af45cd8 [10828.286345] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10828.299899] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10828.303045] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [10828.307752] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [10828.310973] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [10828.314343] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [10828.318760] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10834.092591] rcu: INFO: rcu_sched self-detected stall on CPU [10834.099502] rcu: 0-...!: (4687742 ticks this GP) idle=98a/1/0x4000000000000002 softirq=125154/125154 fqs=959 [10834.108491] (t=8880252 jiffies g=253585 q=5782) [10834.109963] rcu: rcu_sched kthread starved for 8875212 jiffies! g253585 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [10834.122356] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [10834.126429] rcu: RCU grace-period kthread stack dump: [10834.128742] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [10834.133502] Call Trace: [10834.134465] __schedule+0x351/0xcb0 [10834.135449] schedule+0xc0/0x180 [10834.136463] schedule_timeout+0xb4/0x190 [10834.138439] ? do_raw_spin_unlock+0x75/0x190 [10834.139976] ? __next_timer_interrupt+0x160/0x160 [10834.142305] ? prepare_to_swait_event+0xf3/0x140 [10834.143793] rcu_gp_kthread+0x8dd/0x15b0 [10834.145521] ? force_qs_rnp+0x2c0/0x2c0 [10834.147738] kthread+0x1d1/0x200 [10834.149453] ? set_kthread_struct+0x70/0x70 [10834.150717] ret_from_fork+0x1f/0x30 [10834.152050] rcu: Stack dump where RCU GP kthread last ran: [10834.158498] NMI backtrace for cpu 0 [10834.159501] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10834.165235] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10834.169768] Call Trace: [10834.170582] [10834.171102] ? dump_stack+0xbb/0x10e [10834.172445] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [10834.177878] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [10834.180937] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [10834.183893] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [10834.190513] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [10834.192700] ? rcu_check_gp_kthread_starvation+0x16f/0x188 [10834.195882] ? rcu_sched_clock_irq.cold.51+0xce/0x307 [10834.206638] ? update_process_times+0x6d/0xc0 [10834.208268] ? tick_sched_handle.isra.18+0x27/0x90 [10834.211514] ? tick_sched_timer+0x53/0xb0 [10834.213808] ? __hrtimer_run_queues+0x1c2/0x4d0 [10834.216029] ? tick_sched_handle.isra.18+0x90/0x90 [10834.217781] ? hrtimer_interrupt+0x128/0x2c0 [10834.222734] ? smp_apic_timer_interrupt+0xa6/0x270 [10834.226354] ? apic_timer_interrupt+0xf/0x20 [10834.234834] [10834.235836] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10834.238990] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10834.242884] ? do_raw_spin_lock+0x177/0x190 [10834.249780] ? _raw_spin_lock+0x21/0x30 [10834.252748] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10834.257150] ? obd_mod_free+0x46/0x140 [ptlrpc] [10834.260903] ? mdc_commit_open+0x91/0xb0 [mdc] [10834.262705] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10834.265393] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10834.269071] ? after_reply+0xe2c/0x1780 [ptlrpc] [10834.276103] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10834.283925] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [10834.287856] ? do_wait_intr+0xf0/0xf0 [10834.292813] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10834.297321] ? kthread+0x1d1/0x200 [10834.300304] ? set_kthread_struct+0x70/0x70 [10834.305308] ? ret_from_fork+0x1f/0x30 [10834.309465] NMI backtrace for cpu 0 [10834.312236] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10834.319090] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10834.323951] Call Trace: [10834.325426] [10834.326718] ? dump_stack+0xbb/0x10e [10834.328118] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [10834.332770] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [10834.337171] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [10834.338907] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [10834.345359] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [10834.347609] ? rcu_dump_cpu_stacks+0x11b/0x171 [10834.350952] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [10834.355913] ? update_process_times+0x6d/0xc0 [10834.358425] ? tick_sched_handle.isra.18+0x27/0x90 [10834.360756] ? tick_sched_timer+0x53/0xb0 [10834.363322] ? __hrtimer_run_queues+0x1c2/0x4d0 [10834.365730] ? tick_sched_handle.isra.18+0x90/0x90 [10834.368320] ? hrtimer_interrupt+0x128/0x2c0 [10834.371389] ? smp_apic_timer_interrupt+0xa6/0x270 [10834.373638] ? apic_timer_interrupt+0xf/0x20 [10834.378717] [10834.381286] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10834.384237] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10834.390346] ? do_raw_spin_lock+0x177/0x190 [10834.392135] ? _raw_spin_lock+0x21/0x30 [10834.393963] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10834.404356] ? obd_mod_free+0x46/0x140 [ptlrpc] [10834.406241] ? mdc_commit_open+0x91/0xb0 [mdc] [10834.415304] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10834.418588] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10834.426317] ? after_reply+0xe2c/0x1780 [ptlrpc] [10834.430659] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10834.441435] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [10834.445939] ? do_wait_intr+0xf0/0xf0 [10834.451912] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10834.454658] ? kthread+0x1d1/0x200 [10834.456965] ? set_kthread_struct+0x70/0x70 [10834.459508] ? ret_from_fork+0x1f/0x30 [10844.128393] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8885s! [10844.136156] Showing busy workqueues and worker pools: [10844.141872] workqueue events: flags=0x0 [10844.146744] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10844.156850] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [10844.166613] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [10844.172746] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [10844.183799] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10844.187447] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [10844.190959] workqueue events_long: flags=0x0 [10844.192968] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10844.200015] pending: lu_objects_flush [obdclass] [10844.206012] workqueue events_unbound: flags=0x2 [10844.209017] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [10844.212489] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10844.220560] workqueue events_power_efficient: flags=0x80 [10844.225743] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10844.241589] pending: neigh_periodic_work [10844.244811] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10844.248186] pending: do_cache_clean, check_lifetime [10844.251247] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10844.254711] pending: neigh_periodic_work [10844.257220] workqueue writeback: flags=0x4a [10844.259197] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10844.263328] pending: wb_workfn, wb_workfn [10844.266033] workqueue xprtiod: flags=0xa [10844.268279] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10844.271809] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [10844.275885] workqueue ipv6_addrconf: flags=0x40008 [10844.278392] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10844.286034] pending: addrconf_verify_work [10844.289345] workqueue knbd0-recv: flags=0x1a [10844.290870] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10844.293437] in-flight: 657:recv_work [10844.294881] workqueue ptlrpc_pinger: flags=0x2 [10844.296392] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10844.299247] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [10844.301774] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8885s workers=4 idle: 20831 12642 28859 [10844.305197] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10834s workers=2 idle: 659 [10848.060010] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [10848.064077] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10848.085893] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10848.092293] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10848.096922] RIP: 0010:__pv_queued_spin_lock_slowpath+0x328/0x6d0 [10848.099887] Code: d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 05 18 d4 dc 02 01 e0 48 83 05 06 d4 dc 02 01 48 83 05 be d5 dc 02 01 8b 45 08 85 [10848.109863] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10848.113387] RAX: 0000000000006435 RBX: ffff96848811cae0 RCX: 0000000000000001 [10848.116497] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [10848.119965] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [10848.123107] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [10848.125240] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [10848.128002] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [10848.133387] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10848.136477] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [10848.140530] Call Trace: [10848.142401] [10848.143464] ? show_regs.cold.9+0x22/0x2f [10848.145914] ? watchdog_timer_fn.cold.6+0x62/0x10a [10848.148227] ? __hrtimer_run_queues+0x1c2/0x4d0 [10848.150471] ? watchdog+0x40/0x40 [10848.152247] ? hrtimer_interrupt+0x128/0x2c0 [10848.154284] ? smp_apic_timer_interrupt+0xa6/0x270 [10848.158341] ? apic_timer_interrupt+0xf/0x20 [10848.161173] [10848.161929] ? __pv_queued_spin_lock_slowpath+0x328/0x6d0 [10848.167330] do_raw_spin_lock+0x177/0x190 [10848.171349] _raw_spin_lock+0x21/0x30 [10848.174328] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10848.176749] ? do_raw_spin_unlock+0x75/0x190 [10848.180025] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10848.182354] ? do_wait_intr+0xf0/0xf0 [10848.184478] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10848.187746] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10848.189823] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10848.198433] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10848.205640] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10848.208965] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10848.211724] mdc_intent_lock+0x353/0x7e0 [mdc] [10848.214432] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10848.218246] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10848.221372] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10848.224423] lmv_intent_open+0x338/0x1150 [lmv] [10848.227366] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10848.229511] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10848.231482] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10848.233447] ll_intent_lock+0xc5/0xaa0 [lustre] [10848.235339] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10848.237612] ll_lookup_it+0xe0e/0x2270 [lustre] [10848.239347] ? do_raw_spin_unlock+0x75/0x190 [10848.240564] ? _raw_spin_unlock+0x12/0x30 [10848.241826] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10848.243847] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10848.245705] ll_atomic_open+0x36a/0xd70 [lustre] [10848.247709] lookup_open+0x356/0xb40 [10848.248874] do_last+0x313/0xfc0 [10848.250210] ? nd_jump_root+0xe5/0x160 [10848.251584] ? path_init+0x437/0x520 [10848.252908] path_openat+0xf7/0x500 [10848.254519] ? do_raw_spin_unlock+0x75/0x190 [10848.260470] do_filp_open+0x99/0x140 [10848.261678] ? getname_flags+0x6e/0x330 [10848.264486] ? __check_object_size+0xff/0x256 [10848.265932] ? do_raw_spin_unlock+0x75/0x190 [10848.269188] ? _raw_spin_unlock+0x12/0x30 [10848.272274] do_sys_openat2+0x2b4/0x410 [10848.273605] do_sys_open+0x73/0xa0 [10848.274354] __x64_sys_openat+0x24/0x30 [10848.275457] do_syscall_64+0xc1/0x440 [10848.277141] entry_SYSCALL_64_after_hwframe+0x49/0xae [10848.279210] RIP: 0033:0x7ff54af45332 [10848.280884] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10848.288927] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10848.291779] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [10848.295665] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [10848.298871] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [10848.301827] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [10848.305247] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [10852.066088] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [10852.071514] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10852.120130] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10852.126175] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10852.131449] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10852.134700] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10852.137426] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10852.150442] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [10852.155637] RAX: 0000000000006f05 RBX: ffff96848811cae0 RCX: 0000000000000001 [10852.165792] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [10852.168802] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [10852.172714] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [10852.180410] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [10852.186116] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [10852.189658] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10852.192203] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [10852.200169] Call Trace: [10852.201461] [10852.204099] ? show_regs.cold.9+0x22/0x2f [10852.206793] ? watchdog_timer_fn.cold.6+0x62/0x10a [10852.213441] ? __hrtimer_run_queues+0x1c2/0x4d0 [10852.215056] ? watchdog+0x40/0x40 [10852.219859] ? hrtimer_interrupt+0x128/0x2c0 [10852.229774] ? smp_apic_timer_interrupt+0xa6/0x270 [10852.236597] ? apic_timer_interrupt+0xf/0x20 [10852.238046] [10852.238745] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10852.244587] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10852.249824] do_raw_spin_lock+0x177/0x190 [10852.253850] _raw_spin_lock+0x21/0x30 [10852.256633] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10852.263178] process_one_work+0x2c8/0x700 [10852.267383] worker_thread+0x296/0x6e0 [10852.271147] ? rescuer_thread+0x570/0x570 [10852.276247] kthread+0x1d1/0x200 [10852.277829] ? set_kthread_struct+0x70/0x70 [10852.280683] ret_from_fork+0x1f/0x30 [10856.064194] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [10856.070873] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10856.120638] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10856.133931] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10856.150836] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10856.153273] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10856.165091] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10856.168329] RAX: 00000000000060a2 RBX: ffff96848811cae0 RCX: 0000000000000001 [10856.171420] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [10856.178132] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [10856.185371] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [10856.193134] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [10856.200001] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [10856.205887] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10856.210581] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [10856.217004] Call Trace: [10856.217818] [10856.223631] ? show_regs.cold.9+0x22/0x2f [10856.228579] ? watchdog_timer_fn.cold.6+0x62/0x10a [10856.234848] ? __hrtimer_run_queues+0x1c2/0x4d0 [10856.236218] ? watchdog+0x40/0x40 [10856.239244] ? hrtimer_interrupt+0x128/0x2c0 [10856.240647] ? smp_apic_timer_interrupt+0xa6/0x270 [10856.254075] ? apic_timer_interrupt+0xf/0x20 [10856.255515] [10856.256224] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10856.272574] do_raw_spin_lock+0x177/0x190 [10856.273912] _raw_spin_lock+0x21/0x30 [10856.282998] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10856.290399] ? do_raw_spin_unlock+0x75/0x190 [10856.297868] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10856.309147] ? do_wait_intr+0xf0/0xf0 [10856.317673] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10856.326966] mdc_close+0x53b/0xd70 [mdc] [10856.333215] lmv_close+0x18d/0x490 [lmv] [10856.336491] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [10856.349336] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10856.356303] ll_md_real_close+0xa5/0x2a0 [lustre] [10856.363345] ll_md_close+0x274/0x800 [lustre] [10856.369940] ll_file_release+0x128/0xc50 [lustre] [10856.372817] __fput+0x10d/0x440 [10856.376751] ____fput+0x12/0x20 [10856.379732] task_work_run+0xc6/0x110 [10856.385471] exit_to_usermode_loop+0x1dd/0x1f0 [10856.388476] do_syscall_64+0x3d6/0x440 [10856.393554] entry_SYSCALL_64_after_hwframe+0x49/0xae [10856.399981] RIP: 0033:0x7ff54af45cd8 [10856.401205] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10856.428440] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10856.437884] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [10856.442320] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [10856.447507] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [10856.458391] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [10856.466150] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10872.057308] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [10872.060581] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10872.081228] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10872.087636] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10872.090904] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10872.108320] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10872.134535] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10872.141705] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [10872.147513] RDX: 0000000000003d94 RSI: 0000000000000003 RDI: ffff96848811cae0 [10872.154873] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [10872.164213] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [10872.166736] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10872.174150] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [10872.178130] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10872.182871] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [10872.186704] Call Trace: [10872.187950] [10872.189250] ? show_regs.cold.9+0x22/0x2f [10872.196569] ? watchdog_timer_fn.cold.6+0x62/0x10a [10872.200091] ? __hrtimer_run_queues+0x1c2/0x4d0 [10872.213233] ? watchdog+0x40/0x40 [10872.214848] ? hrtimer_interrupt+0x128/0x2c0 [10872.217766] ? smp_apic_timer_interrupt+0xa6/0x270 [10872.220685] ? apic_timer_interrupt+0xf/0x20 [10872.223818] [10872.224817] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10872.227910] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10872.231284] do_raw_spin_lock+0x177/0x190 [10872.233531] _raw_spin_lock+0x21/0x30 [10872.235709] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10872.237934] obd_mod_free+0x46/0x140 [ptlrpc] [10872.240487] mdc_commit_open+0x91/0xb0 [mdc] [10872.242079] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10872.244020] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10872.246026] after_reply+0xe2c/0x1780 [ptlrpc] [10872.248275] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10872.250566] ptlrpcd+0xa82/0xe40 [ptlrpc] [10872.252292] ? do_wait_intr+0xf0/0xf0 [10872.253604] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10872.255563] kthread+0x1d1/0x200 [10872.257130] ? set_kthread_struct+0x70/0x70 [10872.259053] ret_from_fork+0x1f/0x30 [10874.336878] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8915s! [10874.340567] Showing busy workqueues and worker pools: [10874.342758] workqueue events: flags=0x0 [10874.344908] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10874.348481] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [10874.353241] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [10874.355921] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [10874.360530] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10874.363475] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [10874.366101] workqueue events_long: flags=0x0 [10874.368034] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10874.371539] pending: lu_objects_flush [obdclass] [10874.374037] workqueue events_unbound: flags=0x2 [10874.377097] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [10874.380469] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10874.385496] workqueue events_power_efficient: flags=0x80 [10874.388857] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10874.392450] pending: neigh_periodic_work [10874.394707] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10874.397895] pending: do_cache_clean, check_lifetime [10874.400269] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10874.402907] pending: neigh_periodic_work [10874.404539] workqueue writeback: flags=0x4a [10874.406494] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10874.409542] pending: wb_workfn, wb_workfn [10874.411462] workqueue xprtiod: flags=0xa [10874.413033] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10874.415067] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [10874.418063] workqueue ipv6_addrconf: flags=0x40008 [10874.419876] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10874.422077] pending: addrconf_verify_work [10874.424033] workqueue knbd0-recv: flags=0x1a [10874.425483] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10874.427586] in-flight: 657:recv_work [10874.428728] workqueue ptlrpc_pinger: flags=0x2 [10874.430836] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10874.433693] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [10874.436828] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8915s workers=4 idle: 20831 12642 28859 [10874.440911] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10864s workers=2 idle: 659 [10876.060027] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [10876.066137] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10876.105450] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10876.113072] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10876.117352] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10876.120191] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10876.129066] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10876.136526] RAX: 0000000000007f87 RBX: ffff96848811cae0 RCX: 0000000000000001 [10876.142048] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [10876.148337] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [10876.160140] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [10876.167011] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [10876.177052] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [10876.180570] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10876.182787] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [10876.185245] Call Trace: [10876.186158] [10876.187036] ? show_regs.cold.9+0x22/0x2f [10876.188347] ? watchdog_timer_fn.cold.6+0x62/0x10a [10876.192148] ? __hrtimer_run_queues+0x1c2/0x4d0 [10876.197131] ? watchdog+0x40/0x40 [10876.201473] ? hrtimer_interrupt+0x128/0x2c0 [10876.203445] ? smp_apic_timer_interrupt+0xa6/0x270 [10876.205009] ? apic_timer_interrupt+0xf/0x20 [10876.209317] [10876.213095] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10876.219382] do_raw_spin_lock+0x177/0x190 [10876.224763] _raw_spin_lock+0x21/0x30 [10876.229332] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10876.233735] ? do_raw_spin_unlock+0x75/0x190 [10876.239397] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10876.243323] ? do_wait_intr+0xf0/0xf0 [10876.247578] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10876.254086] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10876.257190] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10876.263393] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10876.266467] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10876.270528] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10876.272670] mdc_intent_lock+0x353/0x7e0 [mdc] [10876.279143] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10876.282490] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10876.288669] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10876.294270] lmv_intent_open+0x338/0x1150 [lmv] [10876.296651] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10876.299714] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10876.304514] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10876.307509] ll_intent_lock+0xc5/0xaa0 [lustre] [10876.310496] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10876.312326] ll_lookup_it+0xe0e/0x2270 [lustre] [10876.314273] ? do_raw_spin_unlock+0x75/0x190 [10876.316349] ? _raw_spin_unlock+0x12/0x30 [10876.318195] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10876.320723] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10876.323350] ll_atomic_open+0x36a/0xd70 [lustre] [10876.325619] lookup_open+0x356/0xb40 [10876.327466] do_last+0x313/0xfc0 [10876.328617] ? nd_jump_root+0xe5/0x160 [10876.329898] ? path_init+0x437/0x520 [10876.332692] path_openat+0xf7/0x500 [10876.334361] ? do_raw_spin_unlock+0x75/0x190 [10876.336311] do_filp_open+0x99/0x140 [10876.337568] ? getname_flags+0x6e/0x330 [10876.340698] ? __check_object_size+0xff/0x256 [10876.342373] ? do_raw_spin_unlock+0x75/0x190 [10876.344411] ? _raw_spin_unlock+0x12/0x30 [10876.346179] do_sys_openat2+0x2b4/0x410 [10876.348202] do_sys_open+0x73/0xa0 [10876.350608] __x64_sys_openat+0x24/0x30 [10876.354102] do_syscall_64+0xc1/0x440 [10876.355372] entry_SYSCALL_64_after_hwframe+0x49/0xae [10876.358438] RIP: 0033:0x7ff54af45332 [10876.361303] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10876.370364] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10876.376587] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [10876.379789] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [10876.385352] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [10876.389173] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [10876.400681] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [10880.059010] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [10880.064814] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10880.093525] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10880.098591] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10880.103924] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10880.107439] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10880.110083] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10880.117614] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10880.121537] RAX: 0000000000006f33 RBX: ffff96848811cae0 RCX: 0000000000000001 [10880.125901] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [10880.130104] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [10880.134294] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [10880.138303] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [10880.143225] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [10880.147314] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10880.151069] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [10880.158283] Call Trace: [10880.159445] [10880.163154] ? show_regs.cold.9+0x22/0x2f [10880.169450] ? watchdog_timer_fn.cold.6+0x62/0x10a [10880.172177] ? __hrtimer_run_queues+0x1c2/0x4d0 [10880.177851] ? watchdog+0x40/0x40 [10880.180304] ? hrtimer_interrupt+0x128/0x2c0 [10880.182892] ? smp_apic_timer_interrupt+0xa6/0x270 [10880.186361] ? apic_timer_interrupt+0xf/0x20 [10880.189273] [10880.190651] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10880.194510] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10880.198328] do_raw_spin_lock+0x177/0x190 [10880.201848] _raw_spin_lock+0x21/0x30 [10880.207145] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10880.211038] process_one_work+0x2c8/0x700 [10880.215219] worker_thread+0x296/0x6e0 [10880.218957] ? rescuer_thread+0x570/0x570 [10880.221551] kthread+0x1d1/0x200 [10880.223418] ? set_kthread_struct+0x70/0x70 [10880.227182] ret_from_fork+0x1f/0x30 [10884.058135] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [10884.062409] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10884.088645] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10884.096596] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10884.104628] RIP: 0010:native_safe_halt+0x2e/0x50 [10884.110079] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10884.122210] RSP: 0018:ffffaf9fc1543ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10884.127163] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [10884.133506] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [10884.139082] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [10884.146108] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [10884.157542] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [10884.167526] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [10884.174578] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10884.181341] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [10884.191426] Call Trace: [10884.192264] [10884.193061] ? show_regs.cold.9+0x22/0x2f [10884.199124] ? watchdog_timer_fn.cold.6+0x62/0x10a [10884.207475] ? __hrtimer_run_queues+0x1c2/0x4d0 [10884.215415] ? watchdog+0x40/0x40 [10884.220420] ? hrtimer_interrupt+0x128/0x2c0 [10884.226352] ? smp_apic_timer_interrupt+0xa6/0x270 [10884.230496] ? apic_timer_interrupt+0xf/0x20 [10884.231981] [10884.232695] ? native_safe_halt+0x2e/0x50 [10884.240569] kvm_wait+0xcd/0xe0 [10884.243516] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10884.254712] do_raw_spin_lock+0x177/0x190 [10884.263568] _raw_spin_lock+0x21/0x30 [10884.271263] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10884.275447] ? do_raw_spin_unlock+0x75/0x190 [10884.280959] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10884.288023] ? do_wait_intr+0xf0/0xf0 [10884.289513] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10884.292760] mdc_close+0x53b/0xd70 [mdc] [10884.294350] lmv_close+0x18d/0x490 [lmv] [10884.295993] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [10884.301417] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10884.304002] ll_md_real_close+0xa5/0x2a0 [lustre] [10884.306204] ll_md_close+0x274/0x800 [lustre] [10884.308521] ll_file_release+0x128/0xc50 [lustre] [10884.311450] __fput+0x10d/0x440 [10884.315326] ____fput+0x12/0x20 [10884.317801] task_work_run+0xc6/0x110 [10884.319318] exit_to_usermode_loop+0x1dd/0x1f0 [10884.327283] do_syscall_64+0x3d6/0x440 [10884.329373] entry_SYSCALL_64_after_hwframe+0x49/0xae [10884.336071] RIP: 0033:0x7ff54af45cd8 [10884.337377] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10884.356040] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10884.363532] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [10884.367778] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [10884.373600] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [10884.376720] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [10884.382286] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10900.056318] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [10900.066851] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10900.092922] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10900.100678] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10900.107828] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10900.111043] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10900.122327] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10900.126247] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [10900.129129] RDX: 0000000000005f50 RSI: 0000000000000003 RDI: ffff96848811cae0 [10900.131805] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [10900.134826] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [10900.137660] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10900.140808] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [10900.145953] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10900.149470] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [10900.156167] Call Trace: [10900.157487] [10900.158736] ? show_regs.cold.9+0x22/0x2f [10900.161114] ? watchdog_timer_fn.cold.6+0x62/0x10a [10900.163421] ? __hrtimer_run_queues+0x1c2/0x4d0 [10900.170296] ? watchdog+0x40/0x40 [10900.174339] ? hrtimer_interrupt+0x128/0x2c0 [10900.180553] ? smp_apic_timer_interrupt+0xa6/0x270 [10900.184149] ? apic_timer_interrupt+0xf/0x20 [10900.188498] [10900.191109] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10900.195607] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10900.199381] do_raw_spin_lock+0x177/0x190 [10900.202129] _raw_spin_lock+0x21/0x30 [10900.204341] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10900.209969] obd_mod_free+0x46/0x140 [ptlrpc] [10900.214094] mdc_commit_open+0x91/0xb0 [mdc] [10900.217711] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10900.221340] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10900.224358] after_reply+0xe2c/0x1780 [ptlrpc] [10900.227150] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10900.229745] ptlrpcd+0xa82/0xe40 [ptlrpc] [10900.232321] ? do_wait_intr+0xf0/0xf0 [10900.234291] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10900.237258] kthread+0x1d1/0x200 [10900.238523] ? set_kthread_struct+0x70/0x70 [10900.241277] ret_from_fork+0x1f/0x30 [10904.060576] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [10904.072632] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10904.106878] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10904.114446] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10904.123874] RIP: 0010:__pv_queued_spin_lock_slowpath+0x328/0x6d0 [10904.130207] Code: d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 05 18 d4 dc 02 01 e0 48 83 05 06 d4 dc 02 01 48 83 05 be d5 dc 02 01 8b 45 08 85 [10904.142262] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10904.145754] RAX: 0000000000005dd6 RBX: ffff96848811cae0 RCX: 0000000000000001 [10904.149219] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [10904.152914] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [10904.155776] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [10904.161466] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [10904.170754] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [10904.177453] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10904.182140] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [10904.189422] Call Trace: [10904.191239] [10904.194610] ? show_regs.cold.9+0x22/0x2f [10904.197710] ? watchdog_timer_fn.cold.6+0x62/0x10a [10904.203762] ? __hrtimer_run_queues+0x1c2/0x4d0 [10904.209456] ? watchdog+0x40/0x40 [10904.214385] ? hrtimer_interrupt+0x128/0x2c0 [10904.219068] ? smp_apic_timer_interrupt+0xa6/0x270 [10904.222692] ? apic_timer_interrupt+0xf/0x20 [10904.228135] [10904.228950] ? __pv_queued_spin_lock_slowpath+0x328/0x6d0 [10904.237146] do_raw_spin_lock+0x177/0x190 [10904.240121] _raw_spin_lock+0x21/0x30 [10904.243333] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10904.246948] ? do_raw_spin_unlock+0x75/0x190 [10904.252060] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10904.256687] ? do_wait_intr+0xf0/0xf0 [10904.258044] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10904.271270] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10904.277931] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10904.279930] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10904.288581] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10904.292223] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10904.299390] mdc_intent_lock+0x353/0x7e0 [mdc] [10904.302754] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10904.307475] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10904.317659] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10904.323275] lmv_intent_open+0x338/0x1150 [lmv] [10904.334599] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10904.342623] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10904.348735] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10904.353359] ll_intent_lock+0xc5/0xaa0 [lustre] [10904.356186] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10904.358958] ll_lookup_it+0xe0e/0x2270 [lustre] [10904.364080] ? do_raw_spin_unlock+0x75/0x190 [10904.373177] ? _raw_spin_unlock+0x12/0x30 [10904.376634] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10904.381044] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10904.385026] ll_atomic_open+0x36a/0xd70 [lustre] [10904.397186] lookup_open+0x356/0xb40 [10904.398891] do_last+0x313/0xfc0 [10904.400198] ? nd_jump_root+0xe5/0x160 [10904.406293] ? path_init+0x437/0x520 [10904.410213] path_openat+0xf7/0x500 [10904.411886] ? do_raw_spin_unlock+0x75/0x190 [10904.417075] do_filp_open+0x99/0x140 [10904.420780] ? getname_flags+0x6e/0x330 [10904.426231] ? __check_object_size+0xff/0x256 [10904.430856] ? do_raw_spin_unlock+0x75/0x190 [10904.435337] ? _raw_spin_unlock+0x12/0x30 [10904.442281] do_sys_openat2+0x2b4/0x410 [10904.444665] do_sys_open+0x73/0xa0 [10904.445794] __x64_sys_openat+0x24/0x30 [10904.447038] do_syscall_64+0xc1/0x440 [10904.457185] entry_SYSCALL_64_after_hwframe+0x49/0xae [10904.467223] RIP: 0033:0x7ff54af45332 [10904.475467] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10904.504050] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10904.511011] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [10904.515756] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [10904.525208] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [10904.530666] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [10904.534797] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [10904.544658] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8945s! [10904.551376] Showing busy workqueues and worker pools: [10904.562957] workqueue events: flags=0x0 [10904.570834] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10904.574269] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [10904.578706] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [10904.581663] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [10904.586131] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10904.591553] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [10904.594488] workqueue events_long: flags=0x0 [10904.607118] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10904.613797] pending: lu_objects_flush [obdclass] [10904.620450] workqueue events_unbound: flags=0x2 [10904.627840] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [10904.639226] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10904.646262] workqueue events_power_efficient: flags=0x80 [10904.649307] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10904.653468] pending: neigh_periodic_work [10904.655318] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10904.659216] pending: do_cache_clean, check_lifetime [10904.662340] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10904.665956] pending: neigh_periodic_work [10904.668951] workqueue writeback: flags=0x4a [10904.671303] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10904.676002] pending: wb_workfn, wb_workfn [10904.678887] workqueue xprtiod: flags=0xa [10904.681226] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10904.685003] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [10904.689353] workqueue ipv6_addrconf: flags=0x40008 [10904.692155] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10904.696300] pending: addrconf_verify_work [10904.699325] workqueue knbd0-recv: flags=0x1a [10904.701585] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10904.705274] in-flight: 657:recv_work [10904.707267] workqueue ptlrpc_pinger: flags=0x2 [10904.709983] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10904.713731] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [10904.718282] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8945s workers=4 idle: 20831 12642 28859 [10904.723443] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10894s workers=2 idle: 659 [10908.059019] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [10908.062520] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10908.100285] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10908.107725] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10908.117142] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10908.127527] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10908.133132] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10908.156867] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10908.161955] RAX: 00000000000079a5 RBX: ffff96848811cae0 RCX: 0000000000000001 [10908.166337] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [10908.169893] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [10908.178033] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [10908.186304] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [10908.196291] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [10908.207666] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10908.211788] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [10908.221315] Call Trace: [10908.222795] [10908.226369] ? show_regs.cold.9+0x22/0x2f [10908.231228] ? watchdog_timer_fn.cold.6+0x62/0x10a [10908.236421] ? __hrtimer_run_queues+0x1c2/0x4d0 [10908.241586] ? watchdog+0x40/0x40 [10908.248506] ? hrtimer_interrupt+0x128/0x2c0 [10908.251810] ? smp_apic_timer_interrupt+0xa6/0x270 [10908.253059] ? apic_timer_interrupt+0xf/0x20 [10908.254212] [10908.262748] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10908.275282] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10908.281651] do_raw_spin_lock+0x177/0x190 [10908.285676] _raw_spin_lock+0x21/0x30 [10908.289901] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10908.294270] process_one_work+0x2c8/0x700 [10908.295678] worker_thread+0x296/0x6e0 [10908.298054] ? rescuer_thread+0x570/0x570 [10908.301762] kthread+0x1d1/0x200 [10908.304996] ? set_kthread_struct+0x70/0x70 [10908.308184] ret_from_fork+0x1f/0x30 [10912.059377] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [10912.071959] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10912.122119] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10912.130124] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10912.142872] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10912.152742] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10912.180739] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10912.188047] RAX: 0000000000007297 RBX: ffff96848811cae0 RCX: 0000000000000001 [10912.190319] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [10912.206255] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [10912.210993] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [10912.224331] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [10912.235276] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [10912.246669] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10912.248456] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [10912.263055] Call Trace: [10912.263792] [10912.264407] ? show_regs.cold.9+0x22/0x2f [10912.280226] ? watchdog_timer_fn.cold.6+0x62/0x10a [10912.281768] ? __hrtimer_run_queues+0x1c2/0x4d0 [10912.288393] ? watchdog+0x40/0x40 [10912.290102] ? hrtimer_interrupt+0x128/0x2c0 [10912.292603] ? smp_apic_timer_interrupt+0xa6/0x270 [10912.294698] ? apic_timer_interrupt+0xf/0x20 [10912.297292] [10912.298034] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10912.302822] do_raw_spin_lock+0x177/0x190 [10912.310445] _raw_spin_lock+0x21/0x30 [10912.313333] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10912.326659] ? do_raw_spin_unlock+0x75/0x190 [10912.328155] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10912.341323] ? do_wait_intr+0xf0/0xf0 [10912.344042] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10912.352847] mdc_close+0x53b/0xd70 [mdc] [10912.355005] lmv_close+0x18d/0x490 [lmv] [10912.360047] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [10912.363671] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10912.366242] ll_md_real_close+0xa5/0x2a0 [lustre] [10912.370385] ll_md_close+0x274/0x800 [lustre] [10912.375442] ll_file_release+0x128/0xc50 [lustre] [10912.377794] __fput+0x10d/0x440 [10912.379113] ____fput+0x12/0x20 [10912.380285] task_work_run+0xc6/0x110 [10912.381996] exit_to_usermode_loop+0x1dd/0x1f0 [10912.387036] do_syscall_64+0x3d6/0x440 [10912.389673] entry_SYSCALL_64_after_hwframe+0x49/0xae [10912.396290] RIP: 0033:0x7ff54af45cd8 [10912.401793] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10912.418844] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10912.422835] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [10912.435817] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [10912.446688] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [10912.457839] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [10912.466052] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10932.055093] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [10932.058576] Modules linked in: lustre(O) osp(O) [10932.060285] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [10932.065710] ofd(O) [10932.066821] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) [10932.070698] lod(O) [10932.074025] ptlrpc_gss(O) [10932.081358] mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) [10932.083309] ptlrpc(O) obdclass(O) [10932.085760] lov(O) osc(O) lmv(O) ec(O) fid(O) [10932.087082] ksocklnd(O) [10932.087087] fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 [10932.090321] lnet(O) [10932.090329] squashfs [10932.095778] libcfs(O) [10932.096646] ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror [10932.098255] rpcsec_gss_krb5 [10932.098932] dm_region_hash [10932.102388] auth_rpcgss [10932.103103] dm_log dm_mod sha512_ssse3 sha512_generic [10932.103115] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10932.103121] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10932.103124] RIP: 0010:native_safe_halt+0x2e/0x50 [10932.103188] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10932.103192] RSP: 0000:ffffaf9fc1b97bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10932.103197] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [10932.103199] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff96848811cae0 [10932.103201] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [10932.103203] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [10932.103205] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10932.103888] nfsv4 [10932.104725] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [10932.104732] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10932.104734] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [10932.104742] Call Trace: [10932.104781] [10932.106454] dns_resolver [10932.110040] ? show_regs.cold.9+0x22/0x2f [10932.112762] intel_rapl_msr [10932.113956] ? watchdog_timer_fn.cold.6+0x62/0x10a [10932.119798] intel_rapl_common [10932.121808] ? __hrtimer_run_queues+0x1c2/0x4d0 [10932.121831] ? watchdog+0x40/0x40 [10932.121838] ? hrtimer_interrupt+0x128/0x2c0 [10932.121843] ? smp_apic_timer_interrupt+0xa6/0x270 [10932.124089] sb_edac [10932.125996] ? apic_timer_interrupt+0xf/0x20 [10932.126015] [10932.126021] ? native_safe_halt+0x2e/0x50 [10932.126027] kvm_wait+0xcd/0xe0 [10932.126038] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10932.126046] do_raw_spin_lock+0x177/0x190 [10932.126050] _raw_spin_lock+0x21/0x30 [10932.126063] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10932.128563] rapl [10932.130576] obd_mod_free+0x46/0x140 [ptlrpc] [10932.132634] pcspkr [10932.133338] mdc_commit_open+0x91/0xb0 [mdc] [10932.133394] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10932.136043] i2c_piix4 [10932.137869] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10932.139779] squashfs [10932.140569] after_reply+0xe2c/0x1780 [ptlrpc] [10932.141214] ata_generic [10932.141913] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10932.143306] crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10932.143333] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10932.143340] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10932.143343] RIP: 0010:native_safe_halt+0x2e/0x50 [10932.143362] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10932.144187] ptlrpcd+0xa82/0xe40 [ptlrpc] [10932.145554] RSP: 0000:ffffaf9fc15234f0 EFLAGS: 00000202 [10932.146450] ? do_wait_intr+0xf0/0xf0 [10932.146468] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10932.148338] ORIG_RAX: ffffffffffffff13 [10932.149156] kthread+0x1d1/0x200 [10932.150286] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [10932.150294] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c21b4154 [10932.150296] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [10932.150298] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [10932.150300] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [10932.150303] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [10932.150306] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10932.150308] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [10932.150315] Call Trace: [10932.150321] [10932.150324] ? show_regs.cold.9+0x22/0x2f [10932.150336] ? watchdog_timer_fn.cold.6+0x62/0x10a [10932.150343] ? __hrtimer_run_queues+0x1c2/0x4d0 [10932.150347] ? watchdog+0x40/0x40 [10932.151603] ? set_kthread_struct+0x70/0x70 [10932.152208] ? hrtimer_interrupt+0x128/0x2c0 [10932.153411] ret_from_fork+0x1f/0x30 [10932.154129] ? smp_apic_timer_interrupt+0xa6/0x270 [10932.257931] ? apic_timer_interrupt+0xf/0x20 [10932.261201] [10932.263176] ? native_safe_halt+0x2e/0x50 [10932.264149] kvm_wait+0xcd/0xe0 [10932.264930] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10932.266226] do_raw_spin_lock+0x177/0x190 [10932.279426] _raw_spin_lock+0x21/0x30 [10932.282541] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10932.287597] ? do_raw_spin_unlock+0x75/0x190 [10932.292910] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10932.297124] ? do_wait_intr+0xf0/0xf0 [10932.300396] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10932.303938] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10932.307166] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10932.309559] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10932.312464] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10932.315046] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10932.317564] mdc_intent_lock+0x353/0x7e0 [mdc] [10932.320622] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10932.322971] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10932.331643] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10932.338456] lmv_intent_open+0x338/0x1150 [lmv] [10932.344929] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10932.348618] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10932.351034] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10932.353482] ll_intent_lock+0xc5/0xaa0 [lustre] [10932.355569] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10932.358123] ll_lookup_it+0xe0e/0x2270 [lustre] [10932.360415] ? do_raw_spin_unlock+0x75/0x190 [10932.364809] ? _raw_spin_unlock+0x12/0x30 [10932.367181] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10932.370090] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10932.375493] ll_atomic_open+0x36a/0xd70 [lustre] [10932.379624] lookup_open+0x356/0xb40 [10932.382607] do_last+0x313/0xfc0 [10932.383436] ? nd_jump_root+0xe5/0x160 [10932.384393] ? path_init+0x437/0x520 [10932.385309] path_openat+0xf7/0x500 [10932.390630] ? do_raw_spin_unlock+0x75/0x190 [10932.396371] do_filp_open+0x99/0x140 [10932.400034] ? getname_flags+0x6e/0x330 [10932.403222] ? __check_object_size+0xff/0x256 [10932.408566] ? do_raw_spin_unlock+0x75/0x190 [10932.411275] ? _raw_spin_unlock+0x12/0x30 [10932.412922] do_sys_openat2+0x2b4/0x410 [10932.414969] do_sys_open+0x73/0xa0 [10932.417315] __x64_sys_openat+0x24/0x30 [10932.420355] do_syscall_64+0xc1/0x440 [10932.422487] entry_SYSCALL_64_after_hwframe+0x49/0xae [10932.425597] RIP: 0033:0x7ff54af45332 [10932.429949] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10932.441640] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10932.445705] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [10932.450307] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [10932.453603] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [10932.457210] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [10932.461187] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [10934.753083] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8975s! [10934.764853] Showing busy workqueues and worker pools: [10934.772900] workqueue events: flags=0x0 [10934.775687] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10934.783679] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [10934.792404] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [10934.798902] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [10934.805540] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10934.808951] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [10934.812337] workqueue events_long: flags=0x0 [10934.814302] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10934.820516] pending: lu_objects_flush [obdclass] [10934.829020] workqueue events_unbound: flags=0x2 [10934.833862] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [10934.836486] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10934.842464] workqueue events_power_efficient: flags=0x80 [10934.844830] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10934.851938] pending: neigh_periodic_work [10934.855807] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10934.858940] pending: do_cache_clean, check_lifetime [10934.861683] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10934.864662] pending: neigh_periodic_work [10934.866576] workqueue writeback: flags=0x4a [10934.868459] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10934.870513] pending: wb_workfn, wb_workfn [10934.872527] workqueue xprtiod: flags=0xa [10934.873723] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10934.875939] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [10934.878930] workqueue ipv6_addrconf: flags=0x40008 [10934.880759] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10934.883599] pending: addrconf_verify_work [10934.886512] workqueue knbd0-recv: flags=0x1a [10934.888896] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10934.894749] in-flight: 657:recv_work [10934.896816] workqueue ptlrpc_pinger: flags=0x2 [10934.899787] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10934.904058] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [10934.907636] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8976s workers=4 idle: 20831 12642 28859 [10934.913499] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10925s workers=2 idle: 659 [10936.060945] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [10936.064043] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10936.104248] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10936.110592] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10936.116772] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10936.124745] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10936.128957] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10936.154084] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10936.161879] RAX: 0000000000003611 RBX: ffff96848811cae0 RCX: 0000000000000001 [10936.175762] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [10936.187569] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [10936.192593] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [10936.200289] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [10936.207942] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [10936.213706] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10936.222023] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [10936.227476] Call Trace: [10936.229433] [10936.230731] ? show_regs.cold.9+0x22/0x2f [10936.239435] ? watchdog_timer_fn.cold.6+0x62/0x10a [10936.242919] ? __hrtimer_run_queues+0x1c2/0x4d0 [10936.249717] ? watchdog+0x40/0x40 [10936.250873] ? hrtimer_interrupt+0x128/0x2c0 [10936.255303] ? smp_apic_timer_interrupt+0xa6/0x270 [10936.258653] ? apic_timer_interrupt+0xf/0x20 [10936.262359] [10936.263507] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10936.271765] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10936.281719] do_raw_spin_lock+0x177/0x190 [10936.284884] _raw_spin_lock+0x21/0x30 [10936.288045] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10936.295460] process_one_work+0x2c8/0x700 [10936.297674] worker_thread+0x296/0x6e0 [10936.299841] ? rescuer_thread+0x570/0x570 [10936.310944] kthread+0x1d1/0x200 [10936.312920] ? set_kthread_struct+0x70/0x70 [10936.316415] ret_from_fork+0x1f/0x30 [10940.058030] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [10940.065471] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10940.106293] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10940.118957] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10940.127114] RIP: 0010:native_safe_halt+0x2e/0x50 [10940.131697] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10940.141250] RSP: 0018:ffffaf9fc1543ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10940.148320] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [10940.156819] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c20b4154 [10940.163029] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [10940.169245] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [10940.174871] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [10940.177015] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [10940.182279] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10940.193263] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [10940.202699] Call Trace: [10940.208320] [10940.212911] ? show_regs.cold.9+0x22/0x2f [10940.214325] ? watchdog_timer_fn.cold.6+0x62/0x10a [10940.216046] ? __hrtimer_run_queues+0x1c2/0x4d0 [10940.224805] ? watchdog+0x40/0x40 [10940.229582] ? hrtimer_interrupt+0x128/0x2c0 [10940.236593] ? smp_apic_timer_interrupt+0xa6/0x270 [10940.242600] ? apic_timer_interrupt+0xf/0x20 [10940.247102] [10940.248582] ? native_safe_halt+0x2e/0x50 [10940.249895] kvm_wait+0xcd/0xe0 [10940.251096] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10940.253090] do_raw_spin_lock+0x177/0x190 [10940.254822] _raw_spin_lock+0x21/0x30 [10940.256330] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10940.258800] ? do_raw_spin_unlock+0x75/0x190 [10940.260724] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10940.263418] ? do_wait_intr+0xf0/0xf0 [10940.265287] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10940.267592] mdc_close+0x53b/0xd70 [mdc] [10940.269502] lmv_close+0x18d/0x490 [lmv] [10940.271623] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [10940.274428] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10940.277177] ll_md_real_close+0xa5/0x2a0 [lustre] [10940.279705] ll_md_close+0x274/0x800 [lustre] [10940.282748] ll_file_release+0x128/0xc50 [lustre] [10940.285056] __fput+0x10d/0x440 [10940.286638] ____fput+0x12/0x20 [10940.287563] task_work_run+0xc6/0x110 [10940.289462] exit_to_usermode_loop+0x1dd/0x1f0 [10940.294268] do_syscall_64+0x3d6/0x440 [10940.296313] entry_SYSCALL_64_after_hwframe+0x49/0xae [10940.303355] RIP: 0033:0x7ff54af45cd8 [10940.305644] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10940.322821] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10940.332572] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [10940.337252] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [10940.342279] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [10940.346620] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [10940.349620] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10960.057825] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [10960.060510] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [10960.068325] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10960.068402] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10960.068410] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10960.068413] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10960.068427] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10960.068431] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10960.068436] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [10960.068438] RDX: 0000000000000155 RSI: 0000000000000003 RDI: ffff96848811cae0 [10960.068440] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [10960.068442] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [10960.068444] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10960.068448] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [10960.068450] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10960.068453] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [10960.068461] Call Trace: [10960.068467] [10960.068474] ? show_regs.cold.9+0x22/0x2f [10960.068496] ? watchdog_timer_fn.cold.6+0x62/0x10a [10960.068503] ? __hrtimer_run_queues+0x1c2/0x4d0 [10960.068516] ? watchdog+0x40/0x40 [10960.068522] ? hrtimer_interrupt+0x128/0x2c0 [10960.068526] ? smp_apic_timer_interrupt+0xa6/0x270 [10960.068534] ? apic_timer_interrupt+0xf/0x20 [10960.068601] [10960.068606] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10960.068611] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10960.068616] do_raw_spin_lock+0x177/0x190 [10960.068621] _raw_spin_lock+0x21/0x30 [10960.068627] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10960.073427] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10960.073527] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10960.073534] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10960.073538] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10960.073659] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10960.073664] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10960.073669] RAX: 00000000000057d4 RBX: ffff96848811cae0 RCX: 0000000000000001 [10960.073671] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [10960.073673] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [10960.073676] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [10960.073679] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [10960.073682] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [10960.073685] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10960.073688] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [10960.073697] Call Trace: [10960.073704] [10960.073710] ? show_regs.cold.9+0x22/0x2f [10960.073720] ? watchdog_timer_fn.cold.6+0x62/0x10a [10960.073735] ? __hrtimer_run_queues+0x1c2/0x4d0 [10960.073740] ? watchdog+0x40/0x40 [10960.073744] ? hrtimer_interrupt+0x128/0x2c0 [10960.073749] ? smp_apic_timer_interrupt+0xa6/0x270 [10960.073758] ? apic_timer_interrupt+0xf/0x20 [10960.073763] [10960.073766] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10960.073774] do_raw_spin_lock+0x177/0x190 [10960.073851] _raw_spin_lock+0x21/0x30 [10960.073858] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10960.074278] ? do_raw_spin_unlock+0x75/0x190 [10960.074287] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10960.074647] ? do_wait_intr+0xf0/0xf0 [10960.126834] obd_mod_free+0x46/0x140 [ptlrpc] [10960.132664] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10960.133090] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10960.133417] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10960.133795] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10960.134238] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10960.134297] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10960.134330] mdc_intent_lock+0x353/0x7e0 [mdc] [10960.134366] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10960.134467] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10960.134834] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10960.134865] lmv_intent_open+0x338/0x1150 [lmv] [10960.134906] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10960.135063] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10960.135222] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10960.135251] ll_intent_lock+0xc5/0xaa0 [lustre] [10960.135341] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10960.135552] ll_lookup_it+0xe0e/0x2270 [lustre] [10960.135700] ? do_raw_spin_unlock+0x75/0x190 [10960.135710] ? _raw_spin_unlock+0x12/0x30 [10960.135718] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10960.135745] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10960.135765] ll_atomic_open+0x36a/0xd70 [lustre] [10960.135853] lookup_open+0x356/0xb40 [10960.135862] do_last+0x313/0xfc0 [10960.135867] ? nd_jump_root+0xe5/0x160 [10960.135884] ? path_init+0x437/0x520 [10960.135889] path_openat+0xf7/0x500 [10960.135894] ? do_raw_spin_unlock+0x75/0x190 [10960.135899] do_filp_open+0x99/0x140 [10960.135906] ? getname_flags+0x6e/0x330 [10960.135910] ? __check_object_size+0xff/0x256 [10960.135917] ? do_raw_spin_unlock+0x75/0x190 [10960.135921] ? _raw_spin_unlock+0x12/0x30 [10960.135926] do_sys_openat2+0x2b4/0x410 [10960.135930] do_sys_open+0x73/0xa0 [10960.135944] __x64_sys_openat+0x24/0x30 [10960.135948] do_syscall_64+0xc1/0x440 [10960.135956] entry_SYSCALL_64_after_hwframe+0x49/0xae [10960.135962] RIP: 0033:0x7ff54af45332 [10960.135971] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10960.135975] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10960.135980] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [10960.135982] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [10960.135985] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [10960.135987] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [10960.135989] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [10960.426817] mdc_commit_open+0x91/0xb0 [mdc] [10960.429326] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10960.432625] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10960.435841] after_reply+0xe2c/0x1780 [ptlrpc] [10960.440029] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10960.442815] ptlrpcd+0xa82/0xe40 [ptlrpc] [10960.445812] ? do_wait_intr+0xf0/0xf0 [10960.448033] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10960.451664] kthread+0x1d1/0x200 [10960.453812] ? set_kthread_struct+0x70/0x70 [10960.456826] ret_from_fork+0x1f/0x30 [10964.061921] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [10964.067546] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10964.102887] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10964.110128] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10964.118825] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10964.125574] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10964.129026] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10964.144139] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10964.151609] RAX: 0000000000007785 RBX: ffff96848811cae0 RCX: 0000000000000001 [10964.154708] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [10964.157788] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [10964.161452] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [10964.162903] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [10964.165043] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [10964.168084] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10964.170516] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [10964.172382] Call Trace: [10964.173186] [10964.173970] ? show_regs.cold.9+0x22/0x2f [10964.175255] ? watchdog_timer_fn.cold.6+0x62/0x10a [10964.176593] ? __hrtimer_run_queues+0x1c2/0x4d0 [10964.179834] ? watchdog+0x40/0x40 [10964.180627] ? hrtimer_interrupt+0x128/0x2c0 [10964.184863] ? smp_apic_timer_interrupt+0xa6/0x270 [10964.187233] ? apic_timer_interrupt+0xf/0x20 [10964.190453] [10964.190907] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10964.193193] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10964.198634] do_raw_spin_lock+0x177/0x190 [10964.201722] _raw_spin_lock+0x21/0x30 [10964.204637] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10964.208544] process_one_work+0x2c8/0x700 [10964.209928] worker_thread+0x296/0x6e0 [10964.210902] ? rescuer_thread+0x570/0x570 [10964.214234] kthread+0x1d1/0x200 [10964.216149] ? set_kthread_struct+0x70/0x70 [10964.218887] ret_from_fork+0x1f/0x30 [10964.959463] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9006s! [10964.971253] Showing busy workqueues and worker pools: [10964.974417] workqueue events: flags=0x0 [10964.976268] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10964.979619] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [10964.983847] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [10964.987150] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [10964.992625] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10964.995805] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [10964.999156] workqueue events_long: flags=0x0 [10965.002757] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10965.006504] pending: lu_objects_flush [obdclass] [10965.009608] workqueue events_unbound: flags=0x2 [10965.014738] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [10965.017182] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10965.025815] workqueue events_power_efficient: flags=0x80 [10965.032471] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10965.035661] pending: neigh_periodic_work [10965.037547] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10965.040423] pending: do_cache_clean, check_lifetime [10965.045512] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10965.050316] pending: neigh_periodic_work [10965.052699] workqueue writeback: flags=0x4a [10965.054994] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10965.061572] pending: wb_workfn, wb_workfn [10965.065125] workqueue xprtiod: flags=0xa [10965.069336] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10965.077782] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [10965.084677] workqueue ipv6_addrconf: flags=0x40008 [10965.086715] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10965.090576] pending: addrconf_verify_work [10965.096433] workqueue knbd0-recv: flags=0x1a [10965.099561] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10965.103732] in-flight: 657:recv_work [10965.108842] workqueue ptlrpc_pinger: flags=0x2 [10965.113214] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10965.119741] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [10965.124959] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9006s workers=4 idle: 20831 12642 28859 [10965.130633] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10955s workers=2 idle: 659 [10968.058367] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [10968.064515] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10968.117175] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10968.130823] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10968.141795] RIP: 0010:native_safe_halt+0x2e/0x50 [10968.149772] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10968.168718] RSP: 0018:ffffaf9fc1543ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10968.175919] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [10968.182446] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c20b4154 [10968.188439] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [10968.193185] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [10968.195505] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [10968.203058] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [10968.208471] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10968.216704] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [10968.220473] Call Trace: [10968.222011] [10968.223261] ? show_regs.cold.9+0x22/0x2f [10968.226846] ? watchdog_timer_fn.cold.6+0x62/0x10a [10968.229889] ? __hrtimer_run_queues+0x1c2/0x4d0 [10968.236082] ? watchdog+0x40/0x40 [10968.238252] ? hrtimer_interrupt+0x128/0x2c0 [10968.241526] ? smp_apic_timer_interrupt+0xa6/0x270 [10968.250280] ? apic_timer_interrupt+0xf/0x20 [10968.255509] [10968.262914] ? native_safe_halt+0x2e/0x50 [10968.268330] kvm_wait+0xcd/0xe0 [10968.276745] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10968.283092] do_raw_spin_lock+0x177/0x190 [10968.288421] _raw_spin_lock+0x21/0x30 [10968.294647] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10968.299739] ? do_raw_spin_unlock+0x75/0x190 [10968.307254] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10968.314842] ? do_wait_intr+0xf0/0xf0 [10968.321460] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10968.330317] mdc_close+0x53b/0xd70 [mdc] [10968.336411] lmv_close+0x18d/0x490 [lmv] [10968.340422] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [10968.348337] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10968.356306] ll_md_real_close+0xa5/0x2a0 [lustre] [10968.362476] ll_md_close+0x274/0x800 [lustre] [10968.372350] ll_file_release+0x128/0xc50 [lustre] [10968.373926] __fput+0x10d/0x440 [10968.375944] ____fput+0x12/0x20 [10968.377101] task_work_run+0xc6/0x110 [10968.386178] exit_to_usermode_loop+0x1dd/0x1f0 [10968.396486] do_syscall_64+0x3d6/0x440 [10968.402385] entry_SYSCALL_64_after_hwframe+0x49/0xae [10968.414416] RIP: 0033:0x7ff54af45cd8 [10968.416443] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10968.440625] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10968.454391] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [10968.461397] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [10968.471779] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [10968.484101] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [10968.497308] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10988.061363] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [10988.065813] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10988.098049] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10988.105177] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10988.113109] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10988.120031] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10988.140522] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10988.149275] RAX: 0000000000004e07 RBX: ffff96848811cae0 RCX: 0000000000000001 [10988.157269] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [10988.161353] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [10988.165121] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [10988.172368] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [10988.178463] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [10988.182216] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10988.185494] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [10988.189353] Call Trace: [10988.192893] [10988.195801] ? show_regs.cold.9+0x22/0x2f [10988.201078] ? watchdog_timer_fn.cold.6+0x62/0x10a [10988.209331] ? __hrtimer_run_queues+0x1c2/0x4d0 [10988.212453] ? watchdog+0x40/0x40 [10988.214312] ? hrtimer_interrupt+0x128/0x2c0 [10988.217363] ? smp_apic_timer_interrupt+0xa6/0x270 [10988.225043] ? apic_timer_interrupt+0xf/0x20 [10988.228910] [10988.231130] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10988.234199] do_raw_spin_lock+0x177/0x190 [10988.236502] _raw_spin_lock+0x21/0x30 [10988.240238] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10988.251929] ? do_raw_spin_unlock+0x75/0x190 [10988.263645] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10988.274858] ? do_wait_intr+0xf0/0xf0 [10988.278898] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10988.286141] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10988.294105] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10988.301248] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10988.308950] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10988.313379] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10988.317300] mdc_intent_lock+0x353/0x7e0 [mdc] [10988.320204] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10988.325067] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10988.329808] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10988.333006] lmv_intent_open+0x338/0x1150 [lmv] [10988.335961] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10988.339316] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10988.344161] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10988.346182] ll_intent_lock+0xc5/0xaa0 [lustre] [10988.349296] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10988.351807] ll_lookup_it+0xe0e/0x2270 [lustre] [10988.354084] ? do_raw_spin_unlock+0x75/0x190 [10988.357209] ? _raw_spin_unlock+0x12/0x30 [10988.359194] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10988.361988] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10988.365002] ll_atomic_open+0x36a/0xd70 [lustre] [10988.368030] lookup_open+0x356/0xb40 [10988.369373] do_last+0x313/0xfc0 [10988.371137] ? nd_jump_root+0xe5/0x160 [10988.373626] ? path_init+0x437/0x520 [10988.375099] path_openat+0xf7/0x500 [10988.377175] ? do_raw_spin_unlock+0x75/0x190 [10988.379215] do_filp_open+0x99/0x140 [10988.381103] ? getname_flags+0x6e/0x330 [10988.383068] ? __check_object_size+0xff/0x256 [10988.385108] ? do_raw_spin_unlock+0x75/0x190 [10988.387046] ? _raw_spin_unlock+0x12/0x30 [10988.388934] do_sys_openat2+0x2b4/0x410 [10988.391001] do_sys_open+0x73/0xa0 [10988.392973] __x64_sys_openat+0x24/0x30 [10988.394225] do_syscall_64+0xc1/0x440 [10988.395351] entry_SYSCALL_64_after_hwframe+0x49/0xae [10988.398323] RIP: 0033:0x7ff54af45332 [10988.400226] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10988.410595] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10988.413987] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [10988.417585] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [10988.420654] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [10988.424181] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [10988.427892] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [10992.057441] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [10992.059059] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [10992.068999] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) [10992.074009] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10992.074094] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10992.074103] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10992.074111] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10992.074401] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30f/0x6d0 [10992.074413] Code: c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 <0f> 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 05 18 d4 [10992.074418] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10992.074423] RAX: 000000000000742a RBX: ffff96848811cae0 RCX: 0000000000000001 [10992.074425] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [10992.074427] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [10992.074430] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [10992.074432] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [10992.074435] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [10992.074438] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10992.074440] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [10992.074449] Call Trace: [10992.074456] [10992.074461] ? show_regs.cold.9+0x22/0x2f [10992.074471] ? watchdog_timer_fn.cold.6+0x62/0x10a [10992.074477] ? __hrtimer_run_queues+0x1c2/0x4d0 [10992.074481] ? watchdog+0x40/0x40 [10992.074485] ? hrtimer_interrupt+0x128/0x2c0 [10992.074489] ? smp_apic_timer_interrupt+0xa6/0x270 [10992.074497] ? apic_timer_interrupt+0xf/0x20 [10992.074503] [10992.074505] ? __pv_queued_spin_lock_slowpath+0x30f/0x6d0 [10992.074510] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10992.074516] do_raw_spin_lock+0x177/0x190 [10992.074521] _raw_spin_lock+0x21/0x30 [10992.074525] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10992.074869] process_one_work+0x2c8/0x700 [10992.074878] worker_thread+0x296/0x6e0 [10992.074882] ? rescuer_thread+0x570/0x570 [10992.074886] kthread+0x1d1/0x200 [10992.074892] ? set_kthread_struct+0x70/0x70 [10992.074896] ret_from_fork+0x1f/0x30 [10992.258549] rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10992.282582] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10992.292460] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10992.301391] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10992.304303] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10992.325344] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10992.333440] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [10992.338069] RDX: 00000000000062c1 RSI: 0000000000000003 RDI: ffff96848811cae0 [10992.349410] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [10992.356922] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [10992.361854] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10992.367152] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [10992.381062] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10992.385552] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [10992.393528] Call Trace: [10992.397620] [10992.399291] ? show_regs.cold.9+0x22/0x2f [10992.403219] ? watchdog_timer_fn.cold.6+0x62/0x10a [10992.407075] ? __hrtimer_run_queues+0x1c2/0x4d0 [10992.410350] ? watchdog+0x40/0x40 [10992.412377] ? hrtimer_interrupt+0x128/0x2c0 [10992.417590] ? smp_apic_timer_interrupt+0xa6/0x270 [10992.420539] ? apic_timer_interrupt+0xf/0x20 [10992.423971] [10992.425350] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10992.431813] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10992.439568] do_raw_spin_lock+0x177/0x190 [10992.446494] _raw_spin_lock+0x21/0x30 [10992.450698] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10992.453036] obd_mod_free+0x46/0x140 [ptlrpc] [10992.458054] mdc_commit_open+0x91/0xb0 [mdc] [10992.464058] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10992.468534] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10992.472067] after_reply+0xe2c/0x1780 [ptlrpc] [10992.475376] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10992.479895] ptlrpcd+0xa82/0xe40 [ptlrpc] [10992.482003] ? do_wait_intr+0xf0/0xf0 [10992.484554] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10992.490942] kthread+0x1d1/0x200 [10992.494444] ? set_kthread_struct+0x70/0x70 [10992.499948] ret_from_fork+0x1f/0x30 [10995.175461] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9036s! [10995.187907] Showing busy workqueues and worker pools: [10995.192896] workqueue events: flags=0x0 [10995.203486] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10995.209852] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [10995.219604] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [10995.223462] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [10995.237124] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10995.247617] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [10995.258413] workqueue events_long: flags=0x0 [10995.264509] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10995.273957] pending: lu_objects_flush [obdclass] [10995.286262] workqueue events_unbound: flags=0x2 [10995.297273] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [10995.302469] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10995.324095] workqueue events_power_efficient: flags=0x80 [10995.334015] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10995.348189] pending: neigh_periodic_work [10995.349712] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10995.356156] pending: do_cache_clean, check_lifetime [10995.367027] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10995.375017] pending: neigh_periodic_work [10995.385418] workqueue writeback: flags=0x4a [10995.391159] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10995.405715] pending: wb_workfn, wb_workfn [10995.409943] workqueue xprtiod: flags=0xa [10995.416612] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10995.423129] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [10995.439333] workqueue ipv6_addrconf: flags=0x40008 [10995.443386] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10995.452288] pending: addrconf_verify_work [10995.458090] workqueue knbd0-recv: flags=0x1a [10995.464533] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10995.472917] in-flight: 657:recv_work [10995.474975] workqueue ptlrpc_pinger: flags=0x2 [10995.481843] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10995.488210] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [10995.496454] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9036s workers=4 idle: 20831 12642 28859 [10995.508747] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10985s workers=2 idle: 659 [10996.063297] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [10996.068089] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10996.128163] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10996.135954] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10996.152110] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10996.160827] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10996.188875] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10996.196215] RAX: 0000000000007c16 RBX: ffff96848811cae0 RCX: 0000000000000001 [10996.205083] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [10996.212539] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [10996.216457] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [10996.220595] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [10996.227465] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [10996.239996] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10996.247387] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [10996.256314] Call Trace: [10996.257343] [10996.260365] ? show_regs.cold.9+0x22/0x2f [10996.265749] ? watchdog_timer_fn.cold.6+0x62/0x10a [10996.271537] ? __hrtimer_run_queues+0x1c2/0x4d0 [10996.276089] ? watchdog+0x40/0x40 [10996.277326] ? hrtimer_interrupt+0x128/0x2c0 [10996.280610] ? smp_apic_timer_interrupt+0xa6/0x270 [10996.283538] ? apic_timer_interrupt+0xf/0x20 [10996.285423] [10996.286432] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10996.288896] do_raw_spin_lock+0x177/0x190 [10996.290462] _raw_spin_lock+0x21/0x30 [10996.293016] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10996.295653] ? do_raw_spin_unlock+0x75/0x190 [10996.297879] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10996.300393] ? do_wait_intr+0xf0/0xf0 [10996.302329] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10996.304571] mdc_close+0x53b/0xd70 [mdc] [10996.306189] lmv_close+0x18d/0x490 [lmv] [10996.308045] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [10996.310485] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10996.313120] ll_md_real_close+0xa5/0x2a0 [lustre] [10996.315395] ll_md_close+0x274/0x800 [lustre] [10996.317463] ll_file_release+0x128/0xc50 [lustre] [10996.321625] __fput+0x10d/0x440 [10996.322706] ____fput+0x12/0x20 [10996.325478] task_work_run+0xc6/0x110 [10996.328249] exit_to_usermode_loop+0x1dd/0x1f0 [10996.331570] do_syscall_64+0x3d6/0x440 [10996.340163] entry_SYSCALL_64_after_hwframe+0x49/0xae [10996.349230] RIP: 0033:0x7ff54af45cd8 [10996.355408] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10996.372547] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10996.379551] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [10996.390918] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [10996.399074] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [10996.407693] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [10996.412393] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11014.095016] rcu: INFO: rcu_sched self-detected stall on CPU [11014.097004] rcu: 0-...!: (4792801 ticks this GP) idle=98a/1/0x4000000000000002 softirq=125154/125154 fqs=959 [11014.100793] (t=9060255 jiffies g=253585 q=5801) [11014.102345] rcu: rcu_sched kthread starved for 9055215 jiffies! g253585 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [11014.106735] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [11014.121212] rcu: RCU grace-period kthread stack dump: [11014.129111] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [11014.147185] Call Trace: [11014.150714] __schedule+0x351/0xcb0 [11014.157278] schedule+0xc0/0x180 [11014.162893] schedule_timeout+0xb4/0x190 [11014.168063] ? do_raw_spin_unlock+0x75/0x190 [11014.176553] ? __next_timer_interrupt+0x160/0x160 [11014.183277] ? prepare_to_swait_event+0xf3/0x140 [11014.185197] rcu_gp_kthread+0x8dd/0x15b0 [11014.190166] ? force_qs_rnp+0x2c0/0x2c0 [11014.198350] kthread+0x1d1/0x200 [11014.204873] ? set_kthread_struct+0x70/0x70 [11014.213687] ret_from_fork+0x1f/0x30 [11014.218151] rcu: Stack dump where RCU GP kthread last ran: [11014.230745] Sending NMI from CPU 0 to CPUs 1: [11014.237156] NMI backtrace for cpu 1 skipped: idling at native_safe_halt+0x2e/0x50 [11014.242520] NMI backtrace for cpu 0 [11014.244390] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11014.264173] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11014.279279] Call Trace: [11014.285707] [11014.286382] ? dump_stack+0xbb/0x10e [11014.293952] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [11014.299078] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [11014.303256] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [11014.305848] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [11014.308964] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [11014.312155] ? rcu_dump_cpu_stacks+0x11b/0x171 [11014.314143] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [11014.316146] ? update_process_times+0x6d/0xc0 [11014.318245] ? tick_sched_handle.isra.18+0x27/0x90 [11014.320403] ? tick_sched_timer+0x53/0xb0 [11014.321968] ? __hrtimer_run_queues+0x1c2/0x4d0 [11014.328235] ? tick_sched_handle.isra.18+0x90/0x90 [11014.336244] ? hrtimer_interrupt+0x128/0x2c0 [11014.339973] ? smp_apic_timer_interrupt+0xa6/0x270 [11014.342947] ? apic_timer_interrupt+0xf/0x20 [11014.344307] [11014.345151] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11014.347690] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11014.349971] ? do_raw_spin_lock+0x177/0x190 [11014.351945] ? _raw_spin_lock+0x21/0x30 [11014.353958] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11014.357227] ? obd_mod_free+0x46/0x140 [ptlrpc] [11014.366978] ? mdc_commit_open+0x91/0xb0 [mdc] [11014.369550] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11014.375347] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11014.380977] ? after_reply+0xe2c/0x1780 [ptlrpc] [11014.387692] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11014.392592] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [11014.394922] ? do_wait_intr+0xf0/0xf0 [11014.396906] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11014.399148] ? kthread+0x1d1/0x200 [11014.400340] ? set_kthread_struct+0x70/0x70 [11014.403085] ? ret_from_fork+0x1f/0x30 [11016.062945] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [11016.072081] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11016.107937] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11016.114692] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11016.118330] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11016.122854] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11016.141249] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11016.147443] RAX: 0000000000004425 RBX: ffff96848811cae0 RCX: 0000000000000001 [11016.159942] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [11016.165244] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [11016.173085] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [11016.180904] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [11016.185755] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [11016.204114] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11016.207378] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [11016.211516] Call Trace: [11016.213142] [11016.216739] ? show_regs.cold.9+0x22/0x2f [11016.222487] ? watchdog_timer_fn.cold.6+0x62/0x10a [11016.230384] ? __hrtimer_run_queues+0x1c2/0x4d0 [11016.234263] ? watchdog+0x40/0x40 [11016.237361] ? hrtimer_interrupt+0x128/0x2c0 [11016.239821] ? smp_apic_timer_interrupt+0xa6/0x270 [11016.242676] ? apic_timer_interrupt+0xf/0x20 [11016.245334] [11016.246080] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11016.248362] do_raw_spin_lock+0x177/0x190 [11016.249300] _raw_spin_lock+0x21/0x30 [11016.250128] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11016.253286] ? do_raw_spin_unlock+0x75/0x190 [11016.255378] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11016.260451] ? do_wait_intr+0xf0/0xf0 [11016.262256] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11016.265239] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11016.271962] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11016.275455] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11016.280079] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11016.283528] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11016.286095] mdc_intent_lock+0x353/0x7e0 [mdc] [11016.288269] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11016.291573] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11016.298359] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11016.303429] lmv_intent_open+0x338/0x1150 [lmv] [11016.305056] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11016.307158] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11016.308948] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11016.311059] ll_intent_lock+0xc5/0xaa0 [lustre] [11016.313113] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11016.315727] ll_lookup_it+0xe0e/0x2270 [lustre] [11016.318123] ? do_raw_spin_unlock+0x75/0x190 [11016.320059] ? _raw_spin_unlock+0x12/0x30 [11016.322713] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11016.327076] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11016.331684] ll_atomic_open+0x36a/0xd70 [lustre] [11016.336217] lookup_open+0x356/0xb40 [11016.339344] do_last+0x313/0xfc0 [11016.341212] ? nd_jump_root+0xe5/0x160 [11016.343143] ? path_init+0x437/0x520 [11016.345006] path_openat+0xf7/0x500 [11016.347440] ? do_raw_spin_unlock+0x75/0x190 [11016.350172] do_filp_open+0x99/0x140 [11016.352763] ? getname_flags+0x6e/0x330 [11016.354286] ? __check_object_size+0xff/0x256 [11016.356171] ? do_raw_spin_unlock+0x75/0x190 [11016.357948] ? _raw_spin_unlock+0x12/0x30 [11016.359784] do_sys_openat2+0x2b4/0x410 [11016.361312] do_sys_open+0x73/0xa0 [11016.364575] __x64_sys_openat+0x24/0x30 [11016.367328] do_syscall_64+0xc1/0x440 [11016.368511] entry_SYSCALL_64_after_hwframe+0x49/0xae [11016.371327] RIP: 0033:0x7ff54af45332 [11016.373475] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11016.391989] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11016.395968] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [11016.403115] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [11016.406573] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [11016.409359] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [11016.412485] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [11020.063444] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [11020.068337] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11020.107726] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11020.118231] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11020.132107] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11020.140678] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11020.146952] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11020.163791] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11020.173627] RAX: 0000000000007c08 RBX: ffff96848811cae0 RCX: 0000000000000001 [11020.177757] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [11020.184986] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [11020.189378] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [11020.195339] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [11020.202568] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [11020.211061] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11020.216664] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [11020.223757] Call Trace: [11020.229804] [11020.232596] ? show_regs.cold.9+0x22/0x2f [11020.236894] ? watchdog_timer_fn.cold.6+0x62/0x10a [11020.245014] ? __hrtimer_run_queues+0x1c2/0x4d0 [11020.248652] ? watchdog+0x40/0x40 [11020.249790] ? hrtimer_interrupt+0x128/0x2c0 [11020.251834] ? smp_apic_timer_interrupt+0xa6/0x270 [11020.253896] ? apic_timer_interrupt+0xf/0x20 [11020.255882] [11020.256973] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11020.259769] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11020.262498] do_raw_spin_lock+0x177/0x190 [11020.274136] _raw_spin_lock+0x21/0x30 [11020.276619] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11020.280906] process_one_work+0x2c8/0x700 [11020.283626] worker_thread+0x296/0x6e0 [11020.286555] ? rescuer_thread+0x570/0x570 [11020.289545] kthread+0x1d1/0x200 [11020.291753] ? set_kthread_struct+0x70/0x70 [11020.294382] ret_from_fork+0x1f/0x30 [11025.888097] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9067s! [11025.894714] Showing busy workqueues and worker pools: [11025.899030] workqueue events: flags=0x0 [11025.902601] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11025.906830] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [11025.912666] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [11025.918753] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [11025.934279] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11025.938421] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [11025.942865] workqueue events_long: flags=0x0 [11025.946081] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11025.949512] pending: lu_objects_flush [obdclass] [11025.951440] workqueue events_unbound: flags=0x2 [11025.957453] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [11025.961017] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11025.970790] workqueue events_power_efficient: flags=0x80 [11025.975589] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11025.980824] pending: neigh_periodic_work [11025.985218] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11025.990385] pending: do_cache_clean, check_lifetime [11025.994128] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11026.002090] pending: neigh_periodic_work [11026.010159] workqueue writeback: flags=0x4a [11026.012851] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11026.018253] pending: wb_workfn, wb_workfn [11026.022629] workqueue xprtiod: flags=0xa [11026.030153] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11026.036389] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [11026.044756] workqueue ipv6_addrconf: flags=0x40008 [11026.053704] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11026.062135] pending: addrconf_verify_work [11026.065023] workqueue knbd0-recv: flags=0x1a [11026.072873] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11026.075042] in-flight: 657:recv_work [11026.089188] workqueue ptlrpc_pinger: flags=0x2 [11026.092082] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11026.099822] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [11026.108703] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9067s workers=4 idle: 20831 12642 28859 [11026.117852] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11016s workers=2 idle: 659 [11040.058159] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [11040.062049] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11040.086854] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11040.096715] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11040.106124] RIP: 0010:native_safe_halt+0x2e/0x50 [11040.108659] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11040.118992] RSP: 0018:ffffaf9fc1543ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11040.122711] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [11040.126847] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c20b4154 [11040.131326] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [11040.136887] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [11040.143697] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [11040.154143] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [11040.158759] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11040.160779] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [11040.163193] Call Trace: [11040.164089] [11040.164795] ? show_regs.cold.9+0x22/0x2f [11040.166062] ? watchdog_timer_fn.cold.6+0x62/0x10a [11040.167166] ? __hrtimer_run_queues+0x1c2/0x4d0 [11040.168153] ? watchdog+0x40/0x40 [11040.168938] ? hrtimer_interrupt+0x128/0x2c0 [11040.171315] ? smp_apic_timer_interrupt+0xa6/0x270 [11040.173735] ? apic_timer_interrupt+0xf/0x20 [11040.175798] [11040.177144] ? native_safe_halt+0x2e/0x50 [11040.179610] kvm_wait+0xcd/0xe0 [11040.181569] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11040.184353] do_raw_spin_lock+0x177/0x190 [11040.187894] _raw_spin_lock+0x21/0x30 [11040.191757] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11040.193690] ? do_raw_spin_unlock+0x75/0x190 [11040.195867] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11040.198710] ? do_wait_intr+0xf0/0xf0 [11040.201403] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11040.204912] mdc_close+0x53b/0xd70 [mdc] [11040.207036] lmv_close+0x18d/0x490 [lmv] [11040.209800] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [11040.212733] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11040.217774] ll_md_real_close+0xa5/0x2a0 [lustre] [11040.221127] ll_md_close+0x274/0x800 [lustre] [11040.227060] ll_file_release+0x128/0xc50 [lustre] [11040.232125] __fput+0x10d/0x440 [11040.233837] ____fput+0x12/0x20 [11040.240799] task_work_run+0xc6/0x110 [11040.244910] exit_to_usermode_loop+0x1dd/0x1f0 [11040.249812] do_syscall_64+0x3d6/0x440 [11040.254605] entry_SYSCALL_64_after_hwframe+0x49/0xae [11040.260678] RIP: 0033:0x7ff54af45cd8 [11040.264478] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11040.281942] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11040.285699] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [11040.289877] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [11040.297782] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [11040.301698] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [11040.317032] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11044.060720] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [11044.065141] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11044.087574] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11044.100667] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11044.104689] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11044.106222] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11044.112081] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11044.119709] RAX: 0000000000007f87 RBX: ffff96848811cae0 RCX: 0000000000000001 [11044.125682] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [11044.130678] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [11044.136398] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [11044.141435] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [11044.152957] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [11044.161566] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11044.164816] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [11044.172344] Call Trace: [11044.173198] [11044.173937] ? show_regs.cold.9+0x22/0x2f [11044.187472] ? watchdog_timer_fn.cold.6+0x62/0x10a [11044.193814] ? __hrtimer_run_queues+0x1c2/0x4d0 [11044.200771] ? watchdog+0x40/0x40 [11044.203682] ? hrtimer_interrupt+0x128/0x2c0 [11044.207839] ? smp_apic_timer_interrupt+0xa6/0x270 [11044.213714] ? apic_timer_interrupt+0xf/0x20 [11044.215232] [11044.215964] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11044.223860] do_raw_spin_lock+0x177/0x190 [11044.226314] _raw_spin_lock+0x21/0x30 [11044.229485] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11044.235912] ? do_raw_spin_unlock+0x75/0x190 [11044.239441] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11044.243671] ? do_wait_intr+0xf0/0xf0 [11044.245736] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11044.254462] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11044.266366] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11044.275567] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11044.282769] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11044.286862] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11044.290932] mdc_intent_lock+0x353/0x7e0 [mdc] [11044.299435] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11044.305729] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11044.312413] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11044.314288] lmv_intent_open+0x338/0x1150 [lmv] [11044.324905] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11044.331928] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11044.341777] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11044.347789] ll_intent_lock+0xc5/0xaa0 [lustre] [11044.358413] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11044.360605] ll_lookup_it+0xe0e/0x2270 [lustre] [11044.368529] ? do_raw_spin_unlock+0x75/0x190 [11044.374727] ? _raw_spin_unlock+0x12/0x30 [11044.380548] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11044.386822] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11044.401255] ll_atomic_open+0x36a/0xd70 [lustre] [11044.408825] lookup_open+0x356/0xb40 [11044.412860] do_last+0x313/0xfc0 [11044.414070] ? nd_jump_root+0xe5/0x160 [11044.420477] ? path_init+0x437/0x520 [11044.422449] path_openat+0xf7/0x500 [11044.427779] ? do_raw_spin_unlock+0x75/0x190 [11044.429668] do_filp_open+0x99/0x140 [11044.432545] ? getname_flags+0x6e/0x330 [11044.434852] ? __check_object_size+0xff/0x256 [11044.439678] ? do_raw_spin_unlock+0x75/0x190 [11044.441943] ? _raw_spin_unlock+0x12/0x30 [11044.444352] do_sys_openat2+0x2b4/0x410 [11044.447572] do_sys_open+0x73/0xa0 [11044.449650] __x64_sys_openat+0x24/0x30 [11044.453390] do_syscall_64+0xc1/0x440 [11044.454654] entry_SYSCALL_64_after_hwframe+0x49/0xae [11044.458944] RIP: 0033:0x7ff54af45332 [11044.461088] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11044.478810] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11044.490579] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [11044.498603] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [11044.506676] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [11044.514189] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [11044.519357] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [11048.059027] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [11048.063346] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11048.098316] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11048.111322] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11048.125356] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11048.136324] RIP: 0010:native_safe_halt+0x2e/0x50 [11048.140666] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11048.162497] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11048.169288] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [11048.175862] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [11048.183625] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [11048.187769] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [11048.193244] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [11048.196533] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [11048.200514] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11048.207621] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [11048.213904] Call Trace: [11048.216167] [11048.218159] ? show_regs.cold.9+0x22/0x2f [11048.221500] ? watchdog_timer_fn.cold.6+0x62/0x10a [11048.224520] ? __hrtimer_run_queues+0x1c2/0x4d0 [11048.227194] ? watchdog+0x40/0x40 [11048.229351] ? hrtimer_interrupt+0x128/0x2c0 [11048.231367] ? smp_apic_timer_interrupt+0xa6/0x270 [11048.233034] ? apic_timer_interrupt+0xf/0x20 [11048.236626] [11048.237702] ? native_safe_halt+0x2e/0x50 [11048.242783] kvm_wait+0xcd/0xe0 [11048.244692] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11048.250366] do_raw_spin_lock+0x177/0x190 [11048.254472] _raw_spin_lock+0x21/0x30 [11048.261481] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11048.264792] process_one_work+0x2c8/0x700 [11048.272370] worker_thread+0x296/0x6e0 [11048.275333] ? rescuer_thread+0x570/0x570 [11048.281158] kthread+0x1d1/0x200 [11048.282207] ? set_kthread_struct+0x70/0x70 [11048.283671] ret_from_fork+0x1f/0x30 [11052.055703] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [11052.062203] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11052.106319] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11052.115654] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11052.122177] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11052.126340] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11052.153413] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11052.165828] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [11052.169730] RDX: 0000000000003a43 RSI: 0000000000000003 RDI: ffff96848811cae0 [11052.175898] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [11052.180698] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [11052.187536] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11052.195354] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [11052.201791] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11052.203870] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [11052.207309] Call Trace: [11052.208269] [11052.211371] ? show_regs.cold.9+0x22/0x2f [11052.214465] ? watchdog_timer_fn.cold.6+0x62/0x10a [11052.218636] ? __hrtimer_run_queues+0x1c2/0x4d0 [11052.220360] ? watchdog+0x40/0x40 [11052.226258] ? hrtimer_interrupt+0x128/0x2c0 [11052.237480] ? smp_apic_timer_interrupt+0xa6/0x270 [11052.239955] ? apic_timer_interrupt+0xf/0x20 [11052.241571] [11052.245234] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11052.249859] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11052.254958] do_raw_spin_lock+0x177/0x190 [11052.257057] _raw_spin_lock+0x21/0x30 [11052.261654] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11052.266603] obd_mod_free+0x46/0x140 [ptlrpc] [11052.274952] mdc_commit_open+0x91/0xb0 [mdc] [11052.277626] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11052.281775] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11052.293035] after_reply+0xe2c/0x1780 [ptlrpc] [11052.295862] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11052.297916] ptlrpcd+0xa82/0xe40 [ptlrpc] [11052.301157] ? do_wait_intr+0xf0/0xf0 [11052.302854] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11052.305427] kthread+0x1d1/0x200 [11052.307574] ? set_kthread_struct+0x70/0x70 [11052.310901] ret_from_fork+0x1f/0x30 [11056.607501] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9097s! [11056.612095] Showing busy workqueues and worker pools: [11056.617970] workqueue events: flags=0x0 [11056.619776] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11056.622509] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [11056.626466] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [11056.630617] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [11056.637019] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11056.642863] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [11056.647516] workqueue events_long: flags=0x0 [11056.649708] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11056.657981] pending: lu_objects_flush [obdclass] [11056.660474] workqueue events_unbound: flags=0x2 [11056.661874] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [11056.665339] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11056.668760] workqueue events_power_efficient: flags=0x80 [11056.671238] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11056.673660] pending: neigh_periodic_work [11056.675046] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11056.676795] pending: do_cache_clean, check_lifetime [11056.682907] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11056.688796] pending: neigh_periodic_work [11056.690764] workqueue writeback: flags=0x4a [11056.695658] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11056.702644] pending: wb_workfn, wb_workfn [11056.707140] workqueue xprtiod: flags=0xa [11056.709736] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11056.712363] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [11056.718497] workqueue ipv6_addrconf: flags=0x40008 [11056.722827] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11056.734889] pending: addrconf_verify_work [11056.736674] workqueue knbd0-recv: flags=0x1a [11056.740578] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11056.746966] in-flight: 657:recv_work [11056.750553] workqueue ptlrpc_pinger: flags=0x2 [11056.757121] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11056.768436] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [11056.772037] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9097s workers=4 idle: 20831 12642 28859 [11056.777941] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11046s workers=2 idle: 659 [11068.059949] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [11068.069764] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11068.129549] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11068.136574] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11068.149873] RIP: 0010:native_safe_halt+0x2e/0x50 [11068.157332] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11068.174756] RSP: 0018:ffffaf9fc1543ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11068.183964] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [11068.192155] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c20b4154 [11068.205441] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [11068.217750] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [11068.229823] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [11068.235666] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [11068.238983] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11068.241477] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [11068.244983] Call Trace: [11068.245886] [11068.246744] ? show_regs.cold.9+0x22/0x2f [11068.251618] ? watchdog_timer_fn.cold.6+0x62/0x10a [11068.255449] ? __hrtimer_run_queues+0x1c2/0x4d0 [11068.259403] ? watchdog+0x40/0x40 [11068.263480] ? hrtimer_interrupt+0x128/0x2c0 [11068.269646] ? smp_apic_timer_interrupt+0xa6/0x270 [11068.276587] ? apic_timer_interrupt+0xf/0x20 [11068.277943] [11068.287759] ? native_safe_halt+0x2e/0x50 [11068.292761] kvm_wait+0xcd/0xe0 [11068.295852] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11068.299267] do_raw_spin_lock+0x177/0x190 [11068.300157] _raw_spin_lock+0x21/0x30 [11068.300966] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11068.306707] ? do_raw_spin_unlock+0x75/0x190 [11068.314524] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11068.317787] ? do_wait_intr+0xf0/0xf0 [11068.325641] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11068.330431] mdc_close+0x53b/0xd70 [mdc] [11068.337498] lmv_close+0x18d/0x490 [lmv] [11068.338960] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [11068.344555] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11068.346619] ll_md_real_close+0xa5/0x2a0 [lustre] [11068.353720] ll_md_close+0x274/0x800 [lustre] [11068.364739] ll_file_release+0x128/0xc50 [lustre] [11068.376222] __fput+0x10d/0x440 [11068.381843] ____fput+0x12/0x20 [11068.391304] task_work_run+0xc6/0x110 [11068.399965] exit_to_usermode_loop+0x1dd/0x1f0 [11068.407880] do_syscall_64+0x3d6/0x440 [11068.414830] entry_SYSCALL_64_after_hwframe+0x49/0xae [11068.422887] RIP: 0033:0x7ff54af45cd8 [11068.423961] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11068.441796] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11068.451844] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [11068.460597] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [11068.468251] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [11068.475501] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [11068.489688] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11072.060019] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [11072.063861] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11072.084683] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11072.101338] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11072.116800] RIP: 0010:native_safe_halt+0x2e/0x50 [11072.127570] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11072.147106] RSP: 0000:ffffaf9fc15234f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11072.154636] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [11072.160416] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [11072.166116] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [11072.177276] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [11072.181727] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [11072.187471] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [11072.196105] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11072.201156] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [11072.210005] Call Trace: [11072.211194] [11072.212149] ? show_regs.cold.9+0x22/0x2f [11072.215044] ? watchdog_timer_fn.cold.6+0x62/0x10a [11072.217786] ? __hrtimer_run_queues+0x1c2/0x4d0 [11072.220184] ? watchdog+0x40/0x40 [11072.221718] ? hrtimer_interrupt+0x128/0x2c0 [11072.223777] ? smp_apic_timer_interrupt+0xa6/0x270 [11072.225630] ? apic_timer_interrupt+0xf/0x20 [11072.227592] [11072.228430] ? native_safe_halt+0x2e/0x50 [11072.230186] kvm_wait+0xcd/0xe0 [11072.231450] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11072.233949] do_raw_spin_lock+0x177/0x190 [11072.238713] _raw_spin_lock+0x21/0x30 [11072.245563] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11072.251160] ? do_raw_spin_unlock+0x75/0x190 [11072.252795] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11072.259123] ? do_wait_intr+0xf0/0xf0 [11072.260294] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11072.265843] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11072.267576] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11072.276824] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11072.283813] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11072.289647] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11072.298367] mdc_intent_lock+0x353/0x7e0 [mdc] [11072.305229] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11072.312474] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11072.314667] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11072.321230] lmv_intent_open+0x338/0x1150 [lmv] [11072.334719] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11072.336675] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11072.344277] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11072.350321] ll_intent_lock+0xc5/0xaa0 [lustre] [11072.352105] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11072.363568] ll_lookup_it+0xe0e/0x2270 [lustre] [11072.370005] ? do_raw_spin_unlock+0x75/0x190 [11072.380630] ? _raw_spin_unlock+0x12/0x30 [11072.382722] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11072.394011] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11072.400384] ll_atomic_open+0x36a/0xd70 [lustre] [11072.415920] lookup_open+0x356/0xb40 [11072.420509] do_last+0x313/0xfc0 [11072.421738] ? nd_jump_root+0xe5/0x160 [11072.438124] ? path_init+0x437/0x520 [11072.439377] path_openat+0xf7/0x500 [11072.449532] ? do_raw_spin_unlock+0x75/0x190 [11072.454357] do_filp_open+0x99/0x140 [11072.461251] ? getname_flags+0x6e/0x330 [11072.466450] ? __check_object_size+0xff/0x256 [11072.472249] ? do_raw_spin_unlock+0x75/0x190 [11072.474509] ? _raw_spin_unlock+0x12/0x30 [11072.479006] do_sys_openat2+0x2b4/0x410 [11072.482198] do_sys_open+0x73/0xa0 [11072.486743] __x64_sys_openat+0x24/0x30 [11072.488397] do_syscall_64+0xc1/0x440 [11072.492562] entry_SYSCALL_64_after_hwframe+0x49/0xae [11072.496206] RIP: 0033:0x7ff54af45332 [11072.499728] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11072.515683] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11072.528209] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [11072.533689] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [11072.541302] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [11072.549140] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [11072.553336] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [11076.061156] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [11076.069762] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11076.124795] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11076.132671] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11076.136695] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11076.144934] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11076.147646] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11076.157201] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11076.161197] RAX: 0000000000003e85 RBX: ffff96848811cae0 RCX: 0000000000000001 [11076.164914] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [11076.168045] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [11076.171029] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [11076.174678] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [11076.179825] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [11076.185941] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11076.193969] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [11076.202326] Call Trace: [11076.204839] [11076.205461] ? show_regs.cold.9+0x22/0x2f [11076.209761] ? watchdog_timer_fn.cold.6+0x62/0x10a [11076.214247] ? __hrtimer_run_queues+0x1c2/0x4d0 [11076.217463] ? watchdog+0x40/0x40 [11076.224216] ? hrtimer_interrupt+0x128/0x2c0 [11076.228099] ? smp_apic_timer_interrupt+0xa6/0x270 [11076.232152] ? apic_timer_interrupt+0xf/0x20 [11076.234025] [11076.234653] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11076.239361] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11076.252414] do_raw_spin_lock+0x177/0x190 [11076.253566] _raw_spin_lock+0x21/0x30 [11076.254714] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11076.264642] process_one_work+0x2c8/0x700 [11076.276742] worker_thread+0x296/0x6e0 [11076.280051] ? rescuer_thread+0x570/0x570 [11076.282584] kthread+0x1d1/0x200 [11076.284014] ? set_kthread_struct+0x70/0x70 [11076.290299] ret_from_fork+0x1f/0x30 [11084.058605] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [11084.072280] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11084.148984] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11084.164532] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11084.180625] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11084.185568] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11084.209634] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11084.219947] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [11084.227919] RDX: 0000000000003fce RSI: 0000000000000003 RDI: ffff96848811cae0 [11084.236873] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [11084.244228] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [11084.257277] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11084.267184] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [11084.279399] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11084.286863] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [11084.293397] Call Trace: [11084.294429] [11084.296407] ? show_regs.cold.9+0x22/0x2f [11084.303565] ? watchdog_timer_fn.cold.6+0x62/0x10a [11084.308616] ? __hrtimer_run_queues+0x1c2/0x4d0 [11084.314155] ? watchdog+0x40/0x40 [11084.316643] ? hrtimer_interrupt+0x128/0x2c0 [11084.320305] ? smp_apic_timer_interrupt+0xa6/0x270 [11084.324154] ? apic_timer_interrupt+0xf/0x20 [11084.327709] [11084.328607] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11084.332402] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11084.338134] do_raw_spin_lock+0x177/0x190 [11084.341184] _raw_spin_lock+0x21/0x30 [11084.344441] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11084.350150] obd_mod_free+0x46/0x140 [ptlrpc] [11084.352929] mdc_commit_open+0x91/0xb0 [mdc] [11084.354872] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11084.357048] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11084.359644] after_reply+0xe2c/0x1780 [ptlrpc] [11084.363319] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11084.368866] ptlrpcd+0xa82/0xe40 [ptlrpc] [11084.375359] ? do_wait_intr+0xf0/0xf0 [11084.381160] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11084.390287] kthread+0x1d1/0x200 [11084.393253] ? set_kthread_struct+0x70/0x70 [11084.401910] ret_from_fork+0x1f/0x30 [11086.815196] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9127s! [11086.823679] Showing busy workqueues and worker pools: [11086.835474] workqueue events: flags=0x0 [11086.837780] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11086.853354] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [11086.873112] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [11086.891818] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [11086.921416] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11086.936133] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [11086.946238] workqueue events_long: flags=0x0 [11086.951746] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11086.961424] pending: lu_objects_flush [obdclass] [11086.966415] workqueue events_unbound: flags=0x2 [11086.969635] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [11086.975103] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11086.989332] workqueue events_power_efficient: flags=0x80 [11086.999229] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11087.002466] pending: neigh_periodic_work [11087.004712] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11087.011170] pending: do_cache_clean, check_lifetime [11087.014737] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11087.018548] pending: neigh_periodic_work [11087.024307] workqueue writeback: flags=0x4a [11087.027671] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11087.043182] pending: wb_workfn, wb_workfn [11087.052799] workqueue xprtiod: flags=0xa [11087.061384] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11087.071580] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [11087.082481] workqueue ipv6_addrconf: flags=0x40008 [11087.083603] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11087.091534] pending: addrconf_verify_work [11087.095431] workqueue knbd0-recv: flags=0x1a [11087.098786] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11087.102792] in-flight: 657:recv_work [11087.110143] workqueue ptlrpc_pinger: flags=0x2 [11087.120592] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11087.123564] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [11087.126973] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9128s workers=4 idle: 20831 12642 28859 [11087.133983] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11077s workers=2 idle: 659 [11096.063372] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [11096.086844] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11096.263901] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11096.293583] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11096.314490] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11096.327067] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11096.359436] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11096.375137] RAX: 0000000000007c53 RBX: ffff96848811cae0 RCX: 0000000000000001 [11096.388597] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [11096.398584] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [11096.408497] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [11096.424751] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [11096.435794] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [11096.444529] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11096.459163] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [11096.467610] Call Trace: [11096.471550] [11096.473111] ? show_regs.cold.9+0x22/0x2f [11096.484143] ? watchdog_timer_fn.cold.6+0x62/0x10a [11096.496455] ? __hrtimer_run_queues+0x1c2/0x4d0 [11096.500621] ? watchdog+0x40/0x40 [11096.502885] ? hrtimer_interrupt+0x128/0x2c0 [11096.504723] ? smp_apic_timer_interrupt+0xa6/0x270 [11096.512824] ? apic_timer_interrupt+0xf/0x20 [11096.515680] [11096.519119] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11096.523701] do_raw_spin_lock+0x177/0x190 [11096.530411] _raw_spin_lock+0x21/0x30 [11096.534523] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11096.537805] ? do_raw_spin_unlock+0x75/0x190 [11096.539491] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11096.549483] ? do_wait_intr+0xf0/0xf0 [11096.555379] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11096.564070] mdc_close+0x53b/0xd70 [mdc] [11096.568134] lmv_close+0x18d/0x490 [lmv] [11096.574544] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [11096.577432] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11096.593396] ll_md_real_close+0xa5/0x2a0 [lustre] [11096.598683] ll_md_close+0x274/0x800 [lustre] [11096.613680] ll_file_release+0x128/0xc50 [lustre] [11096.615447] __fput+0x10d/0x440 [11096.616617] ____fput+0x12/0x20 [11096.628570] task_work_run+0xc6/0x110 [11096.633768] exit_to_usermode_loop+0x1dd/0x1f0 [11096.641384] do_syscall_64+0x3d6/0x440 [11096.646726] entry_SYSCALL_64_after_hwframe+0x49/0xae [11096.654485] RIP: 0033:0x7ff54af45cd8 [11096.658627] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11096.673556] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11096.681625] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [11096.686686] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [11096.695345] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [11096.700492] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [11096.713817] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11100.064246] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [11100.067416] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11100.121929] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11100.135090] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11100.145569] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11100.152204] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11100.165635] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11100.172540] RAX: 0000000000006195 RBX: ffff96848811cae0 RCX: 0000000000000001 [11100.177559] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [11100.185108] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [11100.194489] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [11100.198813] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [11100.202384] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [11100.208930] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11100.217102] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [11100.220910] Call Trace: [11100.221934] [11100.223114] ? show_regs.cold.9+0x22/0x2f [11100.225103] ? watchdog_timer_fn.cold.6+0x62/0x10a [11100.232074] ? __hrtimer_run_queues+0x1c2/0x4d0 [11100.233641] ? watchdog+0x40/0x40 [11100.236469] ? hrtimer_interrupt+0x128/0x2c0 [11100.246054] ? smp_apic_timer_interrupt+0xa6/0x270 [11100.252091] ? apic_timer_interrupt+0xf/0x20 [11100.254891] [11100.255556] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11100.262363] do_raw_spin_lock+0x177/0x190 [11100.266648] _raw_spin_lock+0x21/0x30 [11100.269129] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11100.273036] ? do_raw_spin_unlock+0x75/0x190 [11100.277536] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11100.280421] ? do_wait_intr+0xf0/0xf0 [11100.282142] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11100.287021] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11100.290009] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11100.295330] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11100.300270] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11100.301566] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11100.303221] mdc_intent_lock+0x353/0x7e0 [mdc] [11100.304691] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11100.318488] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11100.324992] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11100.331194] lmv_intent_open+0x338/0x1150 [lmv] [11100.334378] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11100.336223] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11100.339623] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11100.341074] ll_intent_lock+0xc5/0xaa0 [lustre] [11100.347741] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11100.349647] ll_lookup_it+0xe0e/0x2270 [lustre] [11100.357608] ? do_raw_spin_unlock+0x75/0x190 [11100.365279] ? _raw_spin_unlock+0x12/0x30 [11100.367121] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11100.370898] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11100.372501] ll_atomic_open+0x36a/0xd70 [lustre] [11100.375491] lookup_open+0x356/0xb40 [11100.377092] do_last+0x313/0xfc0 [11100.380053] ? nd_jump_root+0xe5/0x160 [11100.384365] ? path_init+0x437/0x520 [11100.386279] path_openat+0xf7/0x500 [11100.389104] ? do_raw_spin_unlock+0x75/0x190 [11100.391135] do_filp_open+0x99/0x140 [11100.392531] ? getname_flags+0x6e/0x330 [11100.393793] ? __check_object_size+0xff/0x256 [11100.396730] ? do_raw_spin_unlock+0x75/0x190 [11100.400376] ? _raw_spin_unlock+0x12/0x30 [11100.404354] do_sys_openat2+0x2b4/0x410 [11100.407339] do_sys_open+0x73/0xa0 [11100.409421] __x64_sys_openat+0x24/0x30 [11100.413025] do_syscall_64+0xc1/0x440 [11100.417052] entry_SYSCALL_64_after_hwframe+0x49/0xae [11100.421426] RIP: 0033:0x7ff54af45332 [11100.426207] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11100.433575] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11100.442116] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [11100.445377] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [11100.450013] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [11100.458345] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [11100.460487] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [11104.060078] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [11104.064472] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11104.105631] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11104.117950] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11104.130024] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11104.139488] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11104.144924] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11104.161475] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11104.171146] RAX: 0000000000006675 RBX: ffff96848811cae0 RCX: 0000000000000001 [11104.183452] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [11104.191393] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [11104.201135] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [11104.206996] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [11104.215347] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [11104.227886] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11104.232523] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [11104.247935] Call Trace: [11104.252188] [11104.252909] ? show_regs.cold.9+0x22/0x2f [11104.254242] ? watchdog_timer_fn.cold.6+0x62/0x10a [11104.260921] ? __hrtimer_run_queues+0x1c2/0x4d0 [11104.269190] ? watchdog+0x40/0x40 [11104.270205] ? hrtimer_interrupt+0x128/0x2c0 [11104.276013] ? smp_apic_timer_interrupt+0xa6/0x270 [11104.281031] ? apic_timer_interrupt+0xf/0x20 [11104.285034] [11104.287873] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11104.295152] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11104.296947] do_raw_spin_lock+0x177/0x190 [11104.305962] _raw_spin_lock+0x21/0x30 [11104.308202] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11104.316510] process_one_work+0x2c8/0x700 [11104.320383] worker_thread+0x296/0x6e0 [11104.323629] ? rescuer_thread+0x570/0x570 [11104.325539] kthread+0x1d1/0x200 [11104.326994] ? set_kthread_struct+0x70/0x70 [11104.328900] ret_from_fork+0x1f/0x30 [11112.055023] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [11112.060278] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11112.107757] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11112.127367] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11112.139962] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11112.141871] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11112.171475] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11112.175893] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [11112.183280] RDX: 0000000000003c4e RSI: 0000000000000003 RDI: ffff96848811cae0 [11112.197299] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [11112.203275] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [11112.206556] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11112.216369] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [11112.224643] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11112.230313] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [11112.238342] Call Trace: [11112.241310] [11112.243835] ? show_regs.cold.9+0x22/0x2f [11112.248898] ? watchdog_timer_fn.cold.6+0x62/0x10a [11112.251167] ? __hrtimer_run_queues+0x1c2/0x4d0 [11112.263846] ? watchdog+0x40/0x40 [11112.267366] ? hrtimer_interrupt+0x128/0x2c0 [11112.277230] ? smp_apic_timer_interrupt+0xa6/0x270 [11112.287133] ? apic_timer_interrupt+0xf/0x20 [11112.296124] [11112.297526] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11112.306809] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11112.313237] do_raw_spin_lock+0x177/0x190 [11112.316036] _raw_spin_lock+0x21/0x30 [11112.319964] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11112.324417] obd_mod_free+0x46/0x140 [ptlrpc] [11112.329366] mdc_commit_open+0x91/0xb0 [mdc] [11112.331052] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11112.335183] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11112.338756] after_reply+0xe2c/0x1780 [ptlrpc] [11112.347367] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11112.352912] ptlrpcd+0xa82/0xe40 [ptlrpc] [11112.355391] ? do_wait_intr+0xf0/0xf0 [11112.357871] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11112.364207] kthread+0x1d1/0x200 [11112.366079] ? set_kthread_struct+0x70/0x70 [11112.368609] ret_from_fork+0x1f/0x30 [11117.536216] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9158s! [11117.538906] Showing busy workqueues and worker pools: [11117.541576] workqueue events: flags=0x0 [11117.543303] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11117.547219] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [11117.556455] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [11117.561176] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [11117.568623] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11117.572292] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [11117.576443] workqueue events_long: flags=0x0 [11117.584744] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11117.591770] pending: lu_objects_flush [obdclass] [11117.594398] workqueue events_unbound: flags=0x2 [11117.597555] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [11117.600745] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11117.607015] workqueue events_power_efficient: flags=0x80 [11117.609935] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11117.613179] pending: neigh_periodic_work [11117.615709] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11117.619118] pending: do_cache_clean, check_lifetime [11117.622337] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11117.625960] pending: neigh_periodic_work [11117.628540] workqueue writeback: flags=0x4a [11117.630458] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11117.634046] pending: wb_workfn, wb_workfn [11117.652294] workqueue xprtiod: flags=0xa [11117.664381] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11117.667473] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [11117.693378] workqueue ipv6_addrconf: flags=0x40008 [11117.695182] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11117.700701] pending: addrconf_verify_work [11117.703381] workqueue knbd0-recv: flags=0x1a [11117.706775] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11117.710188] in-flight: 657:recv_work [11117.714329] workqueue ptlrpc_pinger: flags=0x2 [11117.720088] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11117.723080] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [11117.727891] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9158s workers=4 idle: 20831 12642 28859 [11117.732563] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11107s workers=2 idle: 659 [11124.062298] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [11124.072198] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11124.114195] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11124.118786] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11124.122573] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11124.124879] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11124.132821] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11124.135842] RAX: 0000000000007a1c RBX: ffff96848811cae0 RCX: 0000000000000001 [11124.138666] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [11124.141980] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [11124.144733] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [11124.147693] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [11124.150191] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [11124.154574] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11124.157482] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [11124.160783] Call Trace: [11124.161755] [11124.162568] ? show_regs.cold.9+0x22/0x2f [11124.165063] ? watchdog_timer_fn.cold.6+0x62/0x10a [11124.167562] ? __hrtimer_run_queues+0x1c2/0x4d0 [11124.169333] ? watchdog+0x40/0x40 [11124.170790] ? hrtimer_interrupt+0x128/0x2c0 [11124.172576] ? smp_apic_timer_interrupt+0xa6/0x270 [11124.174338] ? apic_timer_interrupt+0xf/0x20 [11124.176090] [11124.177153] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11124.180164] do_raw_spin_lock+0x177/0x190 [11124.181995] _raw_spin_lock+0x21/0x30 [11124.183705] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11124.191316] ? do_raw_spin_unlock+0x75/0x190 [11124.192731] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11124.199934] ? do_wait_intr+0xf0/0xf0 [11124.203820] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11124.208888] mdc_close+0x53b/0xd70 [mdc] [11124.210684] lmv_close+0x18d/0x490 [lmv] [11124.213154] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [11124.218714] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11124.223622] ll_md_real_close+0xa5/0x2a0 [lustre] [11124.229661] ll_md_close+0x274/0x800 [lustre] [11124.232792] ll_file_release+0x128/0xc50 [lustre] [11124.238084] __fput+0x10d/0x440 [11124.240618] ____fput+0x12/0x20 [11124.241781] task_work_run+0xc6/0x110 [11124.243691] exit_to_usermode_loop+0x1dd/0x1f0 [11124.245856] do_syscall_64+0x3d6/0x440 [11124.248728] entry_SYSCALL_64_after_hwframe+0x49/0xae [11124.251499] RIP: 0033:0x7ff54af45cd8 [11124.252927] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11124.261077] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11124.264648] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [11124.267976] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [11124.272458] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [11124.276152] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [11124.283777] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11128.060673] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [11128.071965] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11128.133318] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11128.153863] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11128.164099] RIP: 0010:native_safe_halt+0x2e/0x50 [11128.167389] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11128.198066] RSP: 0000:ffffaf9fc15234f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11128.202012] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [11128.221278] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c21b4154 [11128.235777] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [11128.247979] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [11128.262345] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [11128.273618] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [11128.283055] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11128.289851] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [11128.302816] Call Trace: [11128.303395] [11128.306819] ? show_regs.cold.9+0x22/0x2f [11128.312836] ? watchdog_timer_fn.cold.6+0x62/0x10a [11128.314441] ? __hrtimer_run_queues+0x1c2/0x4d0 [11128.325891] ? watchdog+0x40/0x40 [11128.332118] ? hrtimer_interrupt+0x128/0x2c0 [11128.338412] ? smp_apic_timer_interrupt+0xa6/0x270 [11128.342944] ? apic_timer_interrupt+0xf/0x20 [11128.347138] [11128.349607] ? native_safe_halt+0x2e/0x50 [11128.352565] kvm_wait+0xcd/0xe0 [11128.353898] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11128.357231] do_raw_spin_lock+0x177/0x190 [11128.359662] _raw_spin_lock+0x21/0x30 [11128.361902] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11128.365094] ? do_raw_spin_unlock+0x75/0x190 [11128.367735] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11128.370638] ? do_wait_intr+0xf0/0xf0 [11128.372692] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11128.380098] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11128.384751] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11128.387782] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11128.390410] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11128.393200] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11128.396108] mdc_intent_lock+0x353/0x7e0 [mdc] [11128.399135] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11128.404589] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11128.411706] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11128.415911] lmv_intent_open+0x338/0x1150 [lmv] [11128.418394] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11128.422643] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11128.426018] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11128.430854] ll_intent_lock+0xc5/0xaa0 [lustre] [11128.436609] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11128.440499] ll_lookup_it+0xe0e/0x2270 [lustre] [11128.445850] ? do_raw_spin_unlock+0x75/0x190 [11128.448000] ? _raw_spin_unlock+0x12/0x30 [11128.451932] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11128.456627] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11128.461199] ll_atomic_open+0x36a/0xd70 [lustre] [11128.465372] lookup_open+0x356/0xb40 [11128.467744] do_last+0x313/0xfc0 [11128.470592] ? nd_jump_root+0xe5/0x160 [11128.473920] ? path_init+0x437/0x520 [11128.476409] path_openat+0xf7/0x500 [11128.477997] ? do_raw_spin_unlock+0x75/0x190 [11128.480939] do_filp_open+0x99/0x140 [11128.485707] ? getname_flags+0x6e/0x330 [11128.491282] ? __check_object_size+0xff/0x256 [11128.493880] ? do_raw_spin_unlock+0x75/0x190 [11128.497385] ? _raw_spin_unlock+0x12/0x30 [11128.503208] do_sys_openat2+0x2b4/0x410 [11128.506220] do_sys_open+0x73/0xa0 [11128.508229] __x64_sys_openat+0x24/0x30 [11128.510973] do_syscall_64+0xc1/0x440 [11128.512816] entry_SYSCALL_64_after_hwframe+0x49/0xae [11128.515623] RIP: 0033:0x7ff54af45332 [11128.518862] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11128.541637] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11128.546608] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [11128.551225] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [11128.559137] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [11128.564165] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [11128.568938] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [11132.065601] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [11132.078910] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11132.145282] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11132.166302] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11132.175370] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11132.181869] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11132.189823] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11132.224932] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11132.233491] RAX: 0000000000007f86 RBX: ffff96848811cae0 RCX: 0000000000000001 [11132.237223] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [11132.241741] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [11132.250885] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [11132.267994] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [11132.291074] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [11132.314575] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11132.333002] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [11132.360392] Call Trace: [11132.365838] [11132.370193] ? show_regs.cold.9+0x22/0x2f [11132.383007] ? watchdog_timer_fn.cold.6+0x62/0x10a [11132.384702] ? __hrtimer_run_queues+0x1c2/0x4d0 [11132.396120] ? watchdog+0x40/0x40 [11132.400572] ? hrtimer_interrupt+0x128/0x2c0 [11132.407603] ? smp_apic_timer_interrupt+0xa6/0x270 [11132.417913] ? apic_timer_interrupt+0xf/0x20 [11132.430890] [11132.431726] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11132.440912] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11132.448398] do_raw_spin_lock+0x177/0x190 [11132.451748] _raw_spin_lock+0x21/0x30 [11132.460305] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11132.468577] process_one_work+0x2c8/0x700 [11132.471585] worker_thread+0x296/0x6e0 [11132.493595] ? rescuer_thread+0x570/0x570 [11132.495134] kthread+0x1d1/0x200 [11132.496262] ? set_kthread_struct+0x70/0x70 [11132.510743] ret_from_fork+0x1f/0x30 [11144.056463] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [11144.067548] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11144.086610] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11144.090534] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11144.093172] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11144.094977] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11144.100411] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11144.102371] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [11144.104584] RDX: 0000000000007f83 RSI: 0000000000000003 RDI: ffff96848811cae0 [11144.107471] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [11144.110359] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [11144.114510] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11144.117542] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [11144.120565] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11144.121905] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [11144.124122] Call Trace: [11144.124911] [11144.125551] ? show_regs.cold.9+0x22/0x2f [11144.126860] ? watchdog_timer_fn.cold.6+0x62/0x10a [11144.128074] ? __hrtimer_run_queues+0x1c2/0x4d0 [11144.129370] ? watchdog+0x40/0x40 [11144.130296] ? hrtimer_interrupt+0x128/0x2c0 [11144.131519] ? smp_apic_timer_interrupt+0xa6/0x270 [11144.132871] ? apic_timer_interrupt+0xf/0x20 [11144.134095] [11144.134860] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11144.136499] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11144.138056] do_raw_spin_lock+0x177/0x190 [11144.139132] _raw_spin_lock+0x21/0x30 [11144.140256] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11144.141883] obd_mod_free+0x46/0x140 [ptlrpc] [11144.143545] mdc_commit_open+0x91/0xb0 [mdc] [11144.144888] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11144.146713] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11144.148684] after_reply+0xe2c/0x1780 [ptlrpc] [11144.158070] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11144.160658] ptlrpcd+0xa82/0xe40 [ptlrpc] [11144.162694] ? do_wait_intr+0xf0/0xf0 [11144.165918] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11144.168569] kthread+0x1d1/0x200 [11144.170930] ? set_kthread_struct+0x70/0x70 [11144.172537] ret_from_fork+0x1f/0x30 [11147.743573] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9188s! [11147.748842] Showing busy workqueues and worker pools: [11147.752852] workqueue events: flags=0x0 [11147.756862] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11147.760258] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [11147.765666] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [11147.772842] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [11147.779549] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11147.783002] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [11147.786944] workqueue events_long: flags=0x0 [11147.790546] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11147.795753] pending: lu_objects_flush [obdclass] [11147.799015] workqueue events_unbound: flags=0x2 [11147.801378] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [11147.806846] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11147.813575] workqueue events_power_efficient: flags=0x80 [11147.816315] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11147.819723] pending: neigh_periodic_work [11147.823226] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11147.827991] pending: do_cache_clean, check_lifetime [11147.831702] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11147.836560] pending: neigh_periodic_work [11147.842879] workqueue writeback: flags=0x4a [11147.846362] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11147.850832] pending: wb_workfn, wb_workfn [11147.853051] workqueue xprtiod: flags=0xa [11147.855964] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11147.864502] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [11147.871039] workqueue ipv6_addrconf: flags=0x40008 [11147.882079] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11147.890711] pending: addrconf_verify_work [11147.900054] workqueue knbd0-recv: flags=0x1a [11147.902163] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11147.909785] in-flight: 657:recv_work [11147.913087] workqueue ptlrpc_pinger: flags=0x2 [11147.917022] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11147.923588] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [11147.936508] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9189s workers=4 idle: 20831 12642 28859 [11147.942561] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11138s workers=2 idle: 659 [11152.058100] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [11152.063681] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11152.095609] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11152.102539] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11152.106564] RIP: 0010:native_safe_halt+0x2e/0x50 [11152.112517] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11152.124355] RSP: 0018:ffffaf9fc1543ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11152.126734] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [11152.132546] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c20b4154 [11152.136453] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [11152.140681] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [11152.144376] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [11152.147482] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [11152.150259] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11152.152574] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [11152.155046] Call Trace: [11152.155693] [11152.156252] ? show_regs.cold.9+0x22/0x2f [11152.157618] ? watchdog_timer_fn.cold.6+0x62/0x10a [11152.159453] ? __hrtimer_run_queues+0x1c2/0x4d0 [11152.160492] ? watchdog+0x40/0x40 [11152.161511] ? hrtimer_interrupt+0x128/0x2c0 [11152.164725] ? smp_apic_timer_interrupt+0xa6/0x270 [11152.166459] ? apic_timer_interrupt+0xf/0x20 [11152.168470] [11152.169231] ? native_safe_halt+0x2e/0x50 [11152.170391] kvm_wait+0xcd/0xe0 [11152.171752] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11152.174427] do_raw_spin_lock+0x177/0x190 [11152.175755] _raw_spin_lock+0x21/0x30 [11152.177356] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11152.180576] ? do_raw_spin_unlock+0x75/0x190 [11152.182499] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11152.184969] ? do_wait_intr+0xf0/0xf0 [11152.186494] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11152.189486] mdc_close+0x53b/0xd70 [mdc] [11152.190955] lmv_close+0x18d/0x490 [lmv] [11152.192327] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [11152.194422] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11152.196530] ll_md_real_close+0xa5/0x2a0 [lustre] [11152.198580] ll_md_close+0x274/0x800 [lustre] [11152.200866] ll_file_release+0x128/0xc50 [lustre] [11152.203545] __fput+0x10d/0x440 [11152.204683] ____fput+0x12/0x20 [11152.206225] task_work_run+0xc6/0x110 [11152.209238] exit_to_usermode_loop+0x1dd/0x1f0 [11152.211003] do_syscall_64+0x3d6/0x440 [11152.217203] entry_SYSCALL_64_after_hwframe+0x49/0xae [11152.219662] RIP: 0033:0x7ff54af45cd8 [11152.221537] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11152.229681] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11152.234986] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [11152.243384] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [11152.245907] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [11152.248520] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [11152.251532] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11156.060024] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [11156.072760] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11156.136986] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11156.151645] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11156.163652] RIP: 0010:native_safe_halt+0x2e/0x50 [11156.169328] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11156.194324] RSP: 0000:ffffaf9fc15234f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11156.203380] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [11156.214234] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [11156.223598] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [11156.234480] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [11156.247068] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [11156.256594] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [11156.267590] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11156.274474] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [11156.288236] Call Trace: [11156.291981] [11156.296348] ? show_regs.cold.9+0x22/0x2f [11156.307572] ? watchdog_timer_fn.cold.6+0x62/0x10a [11156.316372] ? __hrtimer_run_queues+0x1c2/0x4d0 [11156.325731] ? watchdog+0x40/0x40 [11156.329359] ? hrtimer_interrupt+0x128/0x2c0 [11156.335657] ? smp_apic_timer_interrupt+0xa6/0x270 [11156.343278] ? apic_timer_interrupt+0xf/0x20 [11156.346091] [11156.355490] ? native_safe_halt+0x2e/0x50 [11156.359027] kvm_wait+0xcd/0xe0 [11156.365907] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11156.374329] do_raw_spin_lock+0x177/0x190 [11156.379761] _raw_spin_lock+0x21/0x30 [11156.385782] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11156.392162] ? do_raw_spin_unlock+0x75/0x190 [11156.393504] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11156.395975] ? do_wait_intr+0xf0/0xf0 [11156.397338] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11156.400084] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11156.401899] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11156.406226] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11156.414708] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11156.416788] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11156.427011] mdc_intent_lock+0x353/0x7e0 [mdc] [11156.436665] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11156.441304] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11156.446487] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11156.456369] lmv_intent_open+0x338/0x1150 [lmv] [11156.461459] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11156.465586] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11156.468733] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11156.472227] ll_intent_lock+0xc5/0xaa0 [lustre] [11156.478485] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11156.483562] ll_lookup_it+0xe0e/0x2270 [lustre] [11156.486355] ? do_raw_spin_unlock+0x75/0x190 [11156.489905] ? _raw_spin_unlock+0x12/0x30 [11156.491483] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11156.498892] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11156.503603] ll_atomic_open+0x36a/0xd70 [lustre] [11156.507404] lookup_open+0x356/0xb40 [11156.508709] do_last+0x313/0xfc0 [11156.513881] ? nd_jump_root+0xe5/0x160 [11156.516985] ? path_init+0x437/0x520 [11156.519785] path_openat+0xf7/0x500 [11156.523601] ? do_raw_spin_unlock+0x75/0x190 [11156.526111] do_filp_open+0x99/0x140 [11156.530277] ? getname_flags+0x6e/0x330 [11156.534969] ? __check_object_size+0xff/0x256 [11156.536627] ? do_raw_spin_unlock+0x75/0x190 [11156.539718] ? _raw_spin_unlock+0x12/0x30 [11156.541875] do_sys_openat2+0x2b4/0x410 [11156.544190] do_sys_open+0x73/0xa0 [11156.546379] __x64_sys_openat+0x24/0x30 [11156.548785] do_syscall_64+0xc1/0x440 [11156.550838] entry_SYSCALL_64_after_hwframe+0x49/0xae [11156.553781] RIP: 0033:0x7ff54af45332 [11156.557530] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11156.576921] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11156.585447] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [11156.588634] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [11156.593225] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [11156.604609] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [11156.609838] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [11160.061765] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [11160.068025] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11160.106840] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11160.115546] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11160.119869] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11160.122692] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11160.125205] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11160.133640] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11160.137192] RAX: 0000000000007b99 RBX: ffff96848811cae0 RCX: 0000000000000001 [11160.140625] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [11160.143627] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [11160.146212] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [11160.149574] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [11160.153404] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [11160.164389] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11160.166908] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [11160.173341] Call Trace: [11160.174381] [11160.175490] ? show_regs.cold.9+0x22/0x2f [11160.178051] ? watchdog_timer_fn.cold.6+0x62/0x10a [11160.181945] ? __hrtimer_run_queues+0x1c2/0x4d0 [11160.184506] ? watchdog+0x40/0x40 [11160.186725] ? hrtimer_interrupt+0x128/0x2c0 [11160.188844] ? smp_apic_timer_interrupt+0xa6/0x270 [11160.192217] ? apic_timer_interrupt+0xf/0x20 [11160.194055] [11160.195138] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11160.197533] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11160.199564] do_raw_spin_lock+0x177/0x190 [11160.201596] _raw_spin_lock+0x21/0x30 [11160.203569] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11160.206228] process_one_work+0x2c8/0x700 [11160.208312] worker_thread+0x296/0x6e0 [11160.211176] ? rescuer_thread+0x570/0x570 [11160.214112] kthread+0x1d1/0x200 [11160.215612] ? set_kthread_struct+0x70/0x70 [11160.217359] ret_from_fork+0x1f/0x30 [11172.055025] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [11172.057837] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11172.089254] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11172.097246] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11172.104376] RIP: 0010:native_safe_halt+0x2e/0x50 [11172.108088] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11172.123870] RSP: 0000:ffffaf9fc1b97bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11172.126892] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [11172.130235] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff96848811cae0 [11172.133769] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [11172.137011] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [11172.140135] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11172.145103] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [11172.151751] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11172.158917] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [11172.160798] Call Trace: [11172.161625] [11172.162994] ? show_regs.cold.9+0x22/0x2f [11172.170845] ? watchdog_timer_fn.cold.6+0x62/0x10a [11172.174448] ? __hrtimer_run_queues+0x1c2/0x4d0 [11172.179117] ? watchdog+0x40/0x40 [11172.182314] ? hrtimer_interrupt+0x128/0x2c0 [11172.184148] ? smp_apic_timer_interrupt+0xa6/0x270 [11172.187303] ? apic_timer_interrupt+0xf/0x20 [11172.194349] [11172.195150] ? native_safe_halt+0x2e/0x50 [11172.196538] kvm_wait+0xcd/0xe0 [11172.197634] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11172.213574] do_raw_spin_lock+0x177/0x190 [11172.217415] _raw_spin_lock+0x21/0x30 [11172.220995] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11172.224647] obd_mod_free+0x46/0x140 [ptlrpc] [11172.229850] mdc_commit_open+0x91/0xb0 [mdc] [11172.233585] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11172.236587] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11172.242185] after_reply+0xe2c/0x1780 [ptlrpc] [11172.244643] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11172.249142] ptlrpcd+0xa82/0xe40 [ptlrpc] [11172.254685] ? do_wait_intr+0xf0/0xf0 [11172.264446] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11172.271276] kthread+0x1d1/0x200 [11172.283222] ? set_kthread_struct+0x70/0x70 [11172.289201] ret_from_fork+0x1f/0x30 [11177.951126] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9219s! [11177.962912] Showing busy workqueues and worker pools: [11177.970741] workqueue events: flags=0x0 [11177.977977] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11177.991744] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [11178.004895] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [11178.013648] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [11178.024165] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11178.038581] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [11178.045515] workqueue events_long: flags=0x0 [11178.049355] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11178.066568] pending: lu_objects_flush [obdclass] [11178.080089] workqueue events_unbound: flags=0x2 [11178.096339] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [11178.108277] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11178.126501] workqueue events_power_efficient: flags=0x80 [11178.128463] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11178.151629] pending: neigh_periodic_work [11178.155313] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11178.163062] pending: do_cache_clean, check_lifetime [11178.169633] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11178.181254] pending: neigh_periodic_work [11178.187840] workqueue writeback: flags=0x4a [11178.192052] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11178.201162] pending: wb_workfn, wb_workfn [11178.202818] workqueue xprtiod: flags=0xa [11178.211071] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11178.224475] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [11178.233100] workqueue ipv6_addrconf: flags=0x40008 [11178.239021] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11178.244704] pending: addrconf_verify_work [11178.251378] workqueue knbd0-recv: flags=0x1a [11178.259531] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11178.267263] in-flight: 657:recv_work [11178.269648] workqueue ptlrpc_pinger: flags=0x2 [11178.271289] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11178.279410] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [11178.285718] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9219s workers=4 idle: 20831 12642 28859 [11178.292881] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11168s workers=2 idle: 659 [11180.058231] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [11180.061181] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11180.088174] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11180.096541] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11180.101437] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11180.106737] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11180.126132] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11180.131198] RAX: 0000000000006133 RBX: ffff96848811cae0 RCX: 0000000000000001 [11180.134701] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [11180.138057] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [11180.141066] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [11180.144053] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [11180.147459] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [11180.159193] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11180.161727] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [11180.167612] Call Trace: [11180.169787] [11180.175210] ? show_regs.cold.9+0x22/0x2f [11180.176629] ? watchdog_timer_fn.cold.6+0x62/0x10a [11180.178611] ? __hrtimer_run_queues+0x1c2/0x4d0 [11180.180311] ? watchdog+0x40/0x40 [11180.182582] ? hrtimer_interrupt+0x128/0x2c0 [11180.186209] ? smp_apic_timer_interrupt+0xa6/0x270 [11180.188598] ? apic_timer_interrupt+0xf/0x20 [11180.192789] [11180.193764] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11180.196455] do_raw_spin_lock+0x177/0x190 [11180.198568] _raw_spin_lock+0x21/0x30 [11180.200572] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11180.204353] ? do_raw_spin_unlock+0x75/0x190 [11180.206622] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11180.210041] ? do_wait_intr+0xf0/0xf0 [11180.212252] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11180.216274] mdc_close+0x53b/0xd70 [mdc] [11180.218156] lmv_close+0x18d/0x490 [lmv] [11180.219764] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [11180.233917] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11180.237965] ll_md_real_close+0xa5/0x2a0 [lustre] [11180.240575] ll_md_close+0x274/0x800 [lustre] [11180.243635] ll_file_release+0x128/0xc50 [lustre] [11180.246569] __fput+0x10d/0x440 [11180.248290] ____fput+0x12/0x20 [11180.250331] task_work_run+0xc6/0x110 [11180.252286] exit_to_usermode_loop+0x1dd/0x1f0 [11180.254625] do_syscall_64+0x3d6/0x440 [11180.260260] entry_SYSCALL_64_after_hwframe+0x49/0xae [11180.261578] RIP: 0033:0x7ff54af45cd8 [11180.262715] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11180.277358] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11180.282910] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [11180.286530] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [11180.294069] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [11180.297351] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [11180.300538] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11184.061650] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [11184.071706] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11184.139035] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11184.151549] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11184.154669] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11184.158490] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11184.170795] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11184.174722] RAX: 0000000000002e05 RBX: ffff96848811cae0 RCX: 0000000000000001 [11184.178254] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [11184.181679] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [11184.186538] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [11184.189181] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [11184.191458] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [11184.195403] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11184.201506] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [11184.204398] Call Trace: [11184.205646] [11184.206467] ? show_regs.cold.9+0x22/0x2f [11184.212332] ? watchdog_timer_fn.cold.6+0x62/0x10a [11184.216744] ? __hrtimer_run_queues+0x1c2/0x4d0 [11184.219377] ? watchdog+0x40/0x40 [11184.221422] ? hrtimer_interrupt+0x128/0x2c0 [11184.223902] ? smp_apic_timer_interrupt+0xa6/0x270 [11184.225693] ? apic_timer_interrupt+0xf/0x20 [11184.228618] [11184.229641] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11184.232165] do_raw_spin_lock+0x177/0x190 [11184.234310] _raw_spin_lock+0x21/0x30 [11184.236196] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11184.238687] ? do_raw_spin_unlock+0x75/0x190 [11184.240073] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11184.245399] ? do_wait_intr+0xf0/0xf0 [11184.249322] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11184.258039] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11184.262490] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11184.267348] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11184.272307] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11184.276385] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11184.278331] mdc_intent_lock+0x353/0x7e0 [mdc] [11184.279775] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11184.292013] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11184.298587] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11184.306622] lmv_intent_open+0x338/0x1150 [lmv] [11184.314422] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11184.317410] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11184.319605] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11184.321510] ll_intent_lock+0xc5/0xaa0 [lustre] [11184.323711] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11184.330434] ll_lookup_it+0xe0e/0x2270 [lustre] [11184.334205] ? do_raw_spin_unlock+0x75/0x190 [11184.341196] ? _raw_spin_unlock+0x12/0x30 [11184.342684] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11184.356977] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11184.358605] ll_atomic_open+0x36a/0xd70 [lustre] [11184.375876] lookup_open+0x356/0xb40 [11184.381033] do_last+0x313/0xfc0 [11184.383777] ? nd_jump_root+0xe5/0x160 [11184.384911] ? path_init+0x437/0x520 [11184.387196] path_openat+0xf7/0x500 [11184.389443] ? do_raw_spin_unlock+0x75/0x190 [11184.399554] do_filp_open+0x99/0x140 [11184.400782] ? getname_flags+0x6e/0x330 [11184.402010] ? __check_object_size+0xff/0x256 [11184.404829] ? do_raw_spin_unlock+0x75/0x190 [11184.406865] ? _raw_spin_unlock+0x12/0x30 [11184.420044] do_sys_openat2+0x2b4/0x410 [11184.423634] do_sys_open+0x73/0xa0 [11184.429305] __x64_sys_openat+0x24/0x30 [11184.434145] do_syscall_64+0xc1/0x440 [11184.441713] entry_SYSCALL_64_after_hwframe+0x49/0xae [11184.444838] RIP: 0033:0x7ff54af45332 [11184.447421] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11184.458635] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11184.471459] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [11184.473766] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [11184.476248] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [11184.478655] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [11184.481237] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [11188.059058] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [11188.063314] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11188.087660] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11188.101499] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11188.116622] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11188.122551] RIP: 0010:native_safe_halt+0x2e/0x50 [11188.127558] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11188.138834] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11188.146978] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [11188.157064] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c2134154 [11188.166791] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [11188.174699] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [11188.178626] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [11188.182779] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [11188.189284] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11188.192805] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [11188.197279] Call Trace: [11188.198670] [11188.199979] ? show_regs.cold.9+0x22/0x2f [11188.202167] ? watchdog_timer_fn.cold.6+0x62/0x10a [11188.204981] ? __hrtimer_run_queues+0x1c2/0x4d0 [11188.207677] ? watchdog+0x40/0x40 [11188.210088] ? hrtimer_interrupt+0x128/0x2c0 [11188.212129] ? smp_apic_timer_interrupt+0xa6/0x270 [11188.214564] ? apic_timer_interrupt+0xf/0x20 [11188.216093] [11188.220101] ? native_safe_halt+0x2e/0x50 [11188.222544] kvm_wait+0xcd/0xe0 [11188.227636] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11188.233026] do_raw_spin_lock+0x177/0x190 [11188.239790] _raw_spin_lock+0x21/0x30 [11188.243814] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11188.250648] process_one_work+0x2c8/0x700 [11188.253787] worker_thread+0x296/0x6e0 [11188.259369] ? rescuer_thread+0x570/0x570 [11188.263806] kthread+0x1d1/0x200 [11188.268200] ? set_kthread_struct+0x70/0x70 [11188.270603] ret_from_fork+0x1f/0x30 [11194.099336] rcu: INFO: rcu_sched self-detected stall on CPU [11194.101078] rcu: 0-...!: (4884730 ticks this GP) idle=98a/1/0x4000000000000002 softirq=125154/125154 fqs=959 [11194.110523] (t=9240259 jiffies g=253585 q=5859) [11194.114430] rcu: rcu_sched kthread starved for 9235219 jiffies! g253585 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [11194.120380] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [11194.125107] rcu: RCU grace-period kthread stack dump: [11194.128757] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [11194.134553] Call Trace: [11194.135592] __schedule+0x351/0xcb0 [11194.139787] schedule+0xc0/0x180 [11194.144567] schedule_timeout+0xb4/0x190 [11194.149407] ? do_raw_spin_unlock+0x75/0x190 [11194.151576] ? __next_timer_interrupt+0x160/0x160 [11194.155432] ? prepare_to_swait_event+0xf3/0x140 [11194.158546] rcu_gp_kthread+0x8dd/0x15b0 [11194.162102] ? force_qs_rnp+0x2c0/0x2c0 [11194.176416] kthread+0x1d1/0x200 [11194.177719] ? set_kthread_struct+0x70/0x70 [11194.189480] ret_from_fork+0x1f/0x30 [11194.195372] rcu: Stack dump where RCU GP kthread last ran: [11194.204786] Sending NMI from CPU 0 to CPUs 2: [11194.207041] NMI backtrace for cpu 2 skipped: idling at native_safe_halt+0x2e/0x50 [11194.207757] NMI backtrace for cpu 0 [11194.220640] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11194.230236] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11194.237282] Call Trace: [11194.243311] [11194.246736] ? dump_stack+0xbb/0x10e [11194.249813] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [11194.253291] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [11194.257542] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [11194.265423] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [11194.272085] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [11194.276161] ? rcu_dump_cpu_stacks+0x11b/0x171 [11194.280598] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [11194.283601] ? update_process_times+0x6d/0xc0 [11194.292045] ? tick_sched_handle.isra.18+0x27/0x90 [11194.293819] ? tick_sched_timer+0x53/0xb0 [11194.300884] ? __hrtimer_run_queues+0x1c2/0x4d0 [11194.302550] ? tick_sched_handle.isra.18+0x90/0x90 [11194.312433] ? hrtimer_interrupt+0x128/0x2c0 [11194.319420] ? smp_apic_timer_interrupt+0xa6/0x270 [11194.325284] ? apic_timer_interrupt+0xf/0x20 [11194.328268] [11194.331238] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11194.334880] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11194.338384] ? do_raw_spin_lock+0x177/0x190 [11194.342078] ? _raw_spin_lock+0x21/0x30 [11194.346958] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11194.352539] ? obd_mod_free+0x46/0x140 [ptlrpc] [11194.363479] ? mdc_commit_open+0x91/0xb0 [mdc] [11194.369931] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11194.375279] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11194.382492] ? after_reply+0xe2c/0x1780 [ptlrpc] [11194.385540] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11194.391095] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [11194.393398] ? do_wait_intr+0xf0/0xf0 [11194.403092] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11194.409148] ? kthread+0x1d1/0x200 [11194.411033] ? set_kthread_struct+0x70/0x70 [11194.412981] ? ret_from_fork+0x1f/0x30 [11208.058011] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [11208.062814] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11208.113248] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11208.130593] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11208.139924] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11208.146019] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11208.167226] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11208.175403] RAX: 00000000000002a5 RBX: ffff96848811cae0 RCX: 0000000000000001 [11208.185135] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [11208.200005] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [11208.205114] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [11208.213814] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [11208.221612] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [11208.230145] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11208.237418] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [11208.241239] Call Trace: [11208.242138] [11208.242754] ? show_regs.cold.9+0x22/0x2f [11208.243998] ? watchdog_timer_fn.cold.6+0x62/0x10a [11208.245524] ? __hrtimer_run_queues+0x1c2/0x4d0 [11208.246961] ? watchdog+0x40/0x40 [11208.248129] ? hrtimer_interrupt+0x128/0x2c0 [11208.249666] ? smp_apic_timer_interrupt+0xa6/0x270 [11208.251344] ? apic_timer_interrupt+0xf/0x20 [11208.253057] [11208.253822] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11208.255791] do_raw_spin_lock+0x177/0x190 [11208.261662] _raw_spin_lock+0x21/0x30 [11208.265373] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11208.270724] ? do_raw_spin_unlock+0x75/0x190 [11208.276242] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11208.282876] ? do_wait_intr+0xf0/0xf0 [11208.285282] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11208.288994] mdc_close+0x53b/0xd70 [mdc] [11208.290558] lmv_close+0x18d/0x490 [lmv] [11208.296963] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [11208.302315] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11208.307182] ll_md_real_close+0xa5/0x2a0 [lustre] [11208.312318] ll_md_close+0x274/0x800 [lustre] [11208.314655] ll_file_release+0x128/0xc50 [lustre] [11208.317466] __fput+0x10d/0x440 [11208.319046] ____fput+0x12/0x20 [11208.320971] task_work_run+0xc6/0x110 [11208.328049] exit_to_usermode_loop+0x1dd/0x1f0 [11208.332054] do_syscall_64+0x3d6/0x440 [11208.335492] entry_SYSCALL_64_after_hwframe+0x49/0xae [11208.339134] RIP: 0033:0x7ff54af45cd8 [11208.341487] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11208.354910] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11208.358438] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [11208.367123] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [11208.372144] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [11208.382914] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [11208.392336] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11208.671079] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9249s! [11208.674441] Showing busy workqueues and worker pools: [11208.676970] workqueue events: flags=0x0 [11208.679630] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11208.684337] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [11208.689475] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [11208.691702] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [11208.697583] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11208.700856] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [11208.704252] workqueue events_long: flags=0x0 [11208.706577] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11208.710295] pending: lu_objects_flush [obdclass] [11208.712335] workqueue events_unbound: flags=0x2 [11208.715198] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [11208.719395] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11208.724835] workqueue events_power_efficient: flags=0x80 [11208.727846] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11208.731329] pending: neigh_periodic_work [11208.733437] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11208.736730] pending: do_cache_clean, check_lifetime [11208.738645] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11208.741344] pending: neigh_periodic_work [11208.743494] workqueue writeback: flags=0x4a [11208.745610] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11208.748599] pending: wb_workfn, wb_workfn [11208.751251] workqueue xprtiod: flags=0xa [11208.753192] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11208.756294] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [11208.760229] workqueue ipv6_addrconf: flags=0x40008 [11208.762461] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11208.766157] pending: addrconf_verify_work [11208.768953] workqueue knbd0-recv: flags=0x1a [11208.770834] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11208.774455] in-flight: 657:recv_work [11208.776410] workqueue ptlrpc_pinger: flags=0x2 [11208.778659] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11208.781635] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [11208.784684] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9249s workers=4 idle: 20831 12642 28859 [11208.788818] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11199s workers=2 idle: 659 [11212.060102] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [11212.067117] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11212.092044] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11212.101304] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11212.105722] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11212.109130] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11212.119837] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11212.123434] RAX: 0000000000007127 RBX: ffff96848811cae0 RCX: 0000000000000001 [11212.127543] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [11212.131422] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [11212.134650] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [11212.137730] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [11212.141265] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [11212.145602] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11212.148536] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [11212.153751] Call Trace: [11212.155133] [11212.156208] ? show_regs.cold.9+0x22/0x2f [11212.158933] ? watchdog_timer_fn.cold.6+0x62/0x10a [11212.160432] ? __hrtimer_run_queues+0x1c2/0x4d0 [11212.162086] ? watchdog+0x40/0x40 [11212.163471] ? hrtimer_interrupt+0x128/0x2c0 [11212.165392] ? smp_apic_timer_interrupt+0xa6/0x270 [11212.169271] ? apic_timer_interrupt+0xf/0x20 [11212.172742] [11212.174714] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11212.178777] do_raw_spin_lock+0x177/0x190 [11212.180100] _raw_spin_lock+0x21/0x30 [11212.182856] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11212.189608] ? do_raw_spin_unlock+0x75/0x190 [11212.192166] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11212.198077] ? do_wait_intr+0xf0/0xf0 [11212.200311] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11212.203484] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11212.208868] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11212.215790] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11212.219786] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11212.223123] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11212.225597] mdc_intent_lock+0x353/0x7e0 [mdc] [11212.227744] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11212.230344] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11212.233629] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11212.236264] lmv_intent_open+0x338/0x1150 [lmv] [11212.240465] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11212.243747] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11212.247773] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11212.250594] ll_intent_lock+0xc5/0xaa0 [lustre] [11212.252806] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11212.254567] ll_lookup_it+0xe0e/0x2270 [lustre] [11212.258767] ? do_raw_spin_unlock+0x75/0x190 [11212.261072] ? _raw_spin_unlock+0x12/0x30 [11212.263613] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11212.265844] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11212.270292] ll_atomic_open+0x36a/0xd70 [lustre] [11212.273596] lookup_open+0x356/0xb40 [11212.277013] do_last+0x313/0xfc0 [11212.277836] ? nd_jump_root+0xe5/0x160 [11212.278557] ? path_init+0x437/0x520 [11212.279480] path_openat+0xf7/0x500 [11212.280887] ? do_raw_spin_unlock+0x75/0x190 [11212.282646] do_filp_open+0x99/0x140 [11212.284030] ? getname_flags+0x6e/0x330 [11212.285571] ? __check_object_size+0xff/0x256 [11212.287300] ? do_raw_spin_unlock+0x75/0x190 [11212.288950] ? _raw_spin_unlock+0x12/0x30 [11212.290427] do_sys_openat2+0x2b4/0x410 [11212.292036] do_sys_open+0x73/0xa0 [11212.293815] __x64_sys_openat+0x24/0x30 [11212.295424] do_syscall_64+0xc1/0x440 [11212.297555] entry_SYSCALL_64_after_hwframe+0x49/0xae [11212.303759] RIP: 0033:0x7ff54af45332 [11212.307075] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11212.315675] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11212.318406] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [11212.321164] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [11212.325549] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [11212.329457] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [11212.332985] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [11220.060965] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [11220.072399] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11220.139934] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11220.167906] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11220.185125] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11220.195077] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11220.203888] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11220.229359] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11220.243288] RAX: 0000000000002508 RBX: ffff96848811cae0 RCX: 0000000000000001 [11220.257292] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [11220.274700] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [11220.287688] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [11220.309902] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [11220.327087] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [11220.339231] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11220.347800] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [11220.359427] Call Trace: [11220.361377] [11220.364294] ? show_regs.cold.9+0x22/0x2f [11220.370471] ? watchdog_timer_fn.cold.6+0x62/0x10a [11220.376047] ? __hrtimer_run_queues+0x1c2/0x4d0 [11220.382833] ? watchdog+0x40/0x40 [11220.390804] ? hrtimer_interrupt+0x128/0x2c0 [11220.397111] ? smp_apic_timer_interrupt+0xa6/0x270 [11220.402966] ? apic_timer_interrupt+0xf/0x20 [11220.408126] [11220.410730] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11220.417358] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11220.434954] do_raw_spin_lock+0x177/0x190 [11220.447686] _raw_spin_lock+0x21/0x30 [11220.452760] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11220.461112] process_one_work+0x2c8/0x700 [11220.465568] worker_thread+0x296/0x6e0 [11220.470375] ? rescuer_thread+0x570/0x570 [11220.475862] kthread+0x1d1/0x200 [11220.479623] ? set_kthread_struct+0x70/0x70 [11220.485390] ret_from_fork+0x1f/0x30 [11236.055091] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [11236.058918] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11236.059323] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [11236.083226] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11236.083243] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11236.083247] RIP: 0010:native_safe_halt+0x2e/0x50 [11236.083263] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11236.083268] RSP: 0000:ffffaf9fc1b97bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11236.083274] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [11236.083277] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff96848811cae0 [11236.083280] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [11236.083283] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [11236.083285] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11236.083289] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [11236.083292] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11236.083303] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [11236.083311] Call Trace: [11236.083317] [11236.083324] ? show_regs.cold.9+0x22/0x2f [11236.083337] ? watchdog_timer_fn.cold.6+0x62/0x10a [11236.083344] ? __hrtimer_run_queues+0x1c2/0x4d0 [11236.083349] ? watchdog+0x40/0x40 [11236.083353] ? hrtimer_interrupt+0x128/0x2c0 [11236.083358] ? smp_apic_timer_interrupt+0xa6/0x270 [11236.083365] ? apic_timer_interrupt+0xf/0x20 [11236.083369] [11236.083372] ? native_safe_halt+0x2e/0x50 [11236.083376] kvm_wait+0xcd/0xe0 [11236.083386] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11236.083394] do_raw_spin_lock+0x177/0x190 [11236.083398] _raw_spin_lock+0x21/0x30 [11236.083405] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11236.084705] obd_mod_free+0x46/0x140 [ptlrpc] [11236.085023] mdc_commit_open+0x91/0xb0 [mdc] [11236.085070] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11236.085468] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11236.098312] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11236.098398] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11236.098407] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11236.098410] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11236.098443] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11236.098448] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11236.098453] RAX: 0000000000007948 RBX: ffff96848811cae0 RCX: 0000000000000001 [11236.098455] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [11236.098458] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [11236.098460] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [11236.098464] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [11236.098467] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [11236.098470] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11236.098473] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [11236.098481] Call Trace: [11236.098487] [11236.098494] ? show_regs.cold.9+0x22/0x2f [11236.098511] ? watchdog_timer_fn.cold.6+0x62/0x10a [11236.098518] ? __hrtimer_run_queues+0x1c2/0x4d0 [11236.098523] ? watchdog+0x40/0x40 [11236.098532] ? hrtimer_interrupt+0x128/0x2c0 [11236.098536] ? smp_apic_timer_interrupt+0xa6/0x270 [11236.098549] ? apic_timer_interrupt+0xf/0x20 [11236.098553] [11236.098556] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11236.098562] do_raw_spin_lock+0x177/0x190 [11236.098566] _raw_spin_lock+0x21/0x30 [11236.098572] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11236.098907] ? do_raw_spin_unlock+0x75/0x190 [11236.098913] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11236.099117] ? do_wait_intr+0xf0/0xf0 [11236.099136] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11236.099396] mdc_close+0x53b/0xd70 [mdc] [11236.099463] lmv_close+0x18d/0x490 [lmv] [11236.099507] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [11236.099794] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11236.099958] ll_md_real_close+0xa5/0x2a0 [lustre] [11236.100044] ll_md_close+0x274/0x800 [lustre] [11236.100141] ll_file_release+0x128/0xc50 [lustre] [11236.100236] __fput+0x10d/0x440 [11236.100249] ____fput+0x12/0x20 [11236.100254] task_work_run+0xc6/0x110 [11236.100262] exit_to_usermode_loop+0x1dd/0x1f0 [11236.100270] do_syscall_64+0x3d6/0x440 [11236.100275] entry_SYSCALL_64_after_hwframe+0x49/0xae [11236.100283] RIP: 0033:0x7ff54af45cd8 [11236.100291] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11236.100296] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11236.100301] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [11236.100303] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [11236.100306] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [11236.100309] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [11236.100311] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11236.619936] after_reply+0xe2c/0x1780 [ptlrpc] [11236.622517] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11236.624881] ptlrpcd+0xa82/0xe40 [ptlrpc] [11236.627134] ? do_wait_intr+0xf0/0xf0 [11236.628621] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11236.630730] kthread+0x1d1/0x200 [11236.632156] ? set_kthread_struct+0x70/0x70 [11236.633856] ret_from_fork+0x1f/0x30 [11238.879736] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9279s! [11238.890217] Showing busy workqueues and worker pools: [11238.896635] workqueue events: flags=0x0 [11238.902743] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11238.911107] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [11238.926156] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [11238.933186] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [11238.953672] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11238.961579] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [11238.968340] workqueue events_long: flags=0x0 [11238.970077] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11238.974125] pending: lu_objects_flush [obdclass] [11238.977165] workqueue events_unbound: flags=0x2 [11238.979299] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [11238.982759] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11238.988346] workqueue events_power_efficient: flags=0x80 [11238.991093] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11238.994770] pending: neigh_periodic_work [11238.997052] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11239.002508] pending: do_cache_clean, check_lifetime [11239.010077] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11239.015879] pending: neigh_periodic_work [11239.020593] workqueue writeback: flags=0x4a [11239.024947] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11239.029824] pending: wb_workfn, wb_workfn [11239.031825] workqueue xprtiod: flags=0xa [11239.033603] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11239.036698] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [11239.040313] workqueue ipv6_addrconf: flags=0x40008 [11239.044775] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11239.049909] pending: addrconf_verify_work [11239.053330] workqueue knbd0-recv: flags=0x1a [11239.055808] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11239.061848] in-flight: 657:recv_work [11239.063332] workqueue ptlrpc_pinger: flags=0x2 [11239.069209] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11239.077716] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [11239.087064] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9280s workers=4 idle: 20831 12642 28859 [11239.100136] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11229s workers=2 idle: 659 [11240.060345] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [11240.067442] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11240.129003] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11240.144803] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11240.151933] RIP: 0010:native_safe_halt+0x2e/0x50 [11240.157203] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11240.174119] RSP: 0000:ffffaf9fc15234f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11240.181412] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [11240.185816] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c21b4154 [11240.199757] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [11240.201860] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [11240.211702] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [11240.218823] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [11240.230436] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11240.237803] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [11240.246947] Call Trace: [11240.248043] [11240.248839] ? show_regs.cold.9+0x22/0x2f [11240.254122] ? watchdog_timer_fn.cold.6+0x62/0x10a [11240.257683] ? __hrtimer_run_queues+0x1c2/0x4d0 [11240.261633] ? watchdog+0x40/0x40 [11240.267710] ? hrtimer_interrupt+0x128/0x2c0 [11240.273990] ? smp_apic_timer_interrupt+0xa6/0x270 [11240.283957] ? apic_timer_interrupt+0xf/0x20 [11240.289417] [11240.292220] ? native_safe_halt+0x2e/0x50 [11240.297473] kvm_wait+0xcd/0xe0 [11240.298726] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11240.302926] do_raw_spin_lock+0x177/0x190 [11240.306474] _raw_spin_lock+0x21/0x30 [11240.311804] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11240.319869] ? do_raw_spin_unlock+0x75/0x190 [11240.326957] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11240.332520] ? do_wait_intr+0xf0/0xf0 [11240.334839] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11240.337315] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11240.340023] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11240.343647] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11240.346573] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11240.348922] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11240.351118] mdc_intent_lock+0x353/0x7e0 [mdc] [11240.353491] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11240.355133] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11240.357855] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11240.360531] lmv_intent_open+0x338/0x1150 [lmv] [11240.363196] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11240.366604] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11240.371141] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11240.373882] ll_intent_lock+0xc5/0xaa0 [lustre] [11240.377899] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11240.383263] ll_lookup_it+0xe0e/0x2270 [lustre] [11240.388216] ? do_raw_spin_unlock+0x75/0x190 [11240.390841] ? _raw_spin_unlock+0x12/0x30 [11240.392803] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11240.397076] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11240.401972] ll_atomic_open+0x36a/0xd70 [lustre] [11240.404683] lookup_open+0x356/0xb40 [11240.406190] do_last+0x313/0xfc0 [11240.410090] ? nd_jump_root+0xe5/0x160 [11240.411865] ? path_init+0x437/0x520 [11240.414426] path_openat+0xf7/0x500 [11240.416094] ? do_raw_spin_unlock+0x75/0x190 [11240.419032] do_filp_open+0x99/0x140 [11240.420804] ? getname_flags+0x6e/0x330 [11240.422768] ? __check_object_size+0xff/0x256 [11240.424800] ? do_raw_spin_unlock+0x75/0x190 [11240.426689] ? _raw_spin_unlock+0x12/0x30 [11240.429560] do_sys_openat2+0x2b4/0x410 [11240.430948] do_sys_open+0x73/0xa0 [11240.432204] __x64_sys_openat+0x24/0x30 [11240.437122] do_syscall_64+0xc1/0x440 [11240.439738] entry_SYSCALL_64_after_hwframe+0x49/0xae [11240.442045] RIP: 0033:0x7ff54af45332 [11240.444118] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11240.452553] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11240.460086] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [11240.463238] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [11240.466316] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [11240.469879] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [11240.472247] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [11248.059023] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [11248.060820] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11248.074296] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11248.078226] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11248.081397] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11248.083507] RIP: 0010:native_safe_halt+0x2e/0x50 [11248.085353] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11248.091077] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11248.093576] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [11248.096087] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c2134154 [11248.098481] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [11248.100899] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [11248.103311] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [11248.105308] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [11248.107353] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11248.108854] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [11248.110765] Call Trace: [11248.111518] [11248.112242] ? show_regs.cold.9+0x22/0x2f [11248.113364] ? watchdog_timer_fn.cold.6+0x62/0x10a [11248.114621] ? __hrtimer_run_queues+0x1c2/0x4d0 [11248.115867] ? watchdog+0x40/0x40 [11248.116698] ? hrtimer_interrupt+0x128/0x2c0 [11248.118043] ? smp_apic_timer_interrupt+0xa6/0x270 [11248.119472] ? apic_timer_interrupt+0xf/0x20 [11248.120812] [11248.121574] ? native_safe_halt+0x2e/0x50 [11248.122859] kvm_wait+0xcd/0xe0 [11248.124007] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11248.125885] do_raw_spin_lock+0x177/0x190 [11248.127267] _raw_spin_lock+0x21/0x30 [11248.128557] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11248.130616] process_one_work+0x2c8/0x700 [11248.131896] worker_thread+0x296/0x6e0 [11248.133016] ? rescuer_thread+0x570/0x570 [11248.134373] kthread+0x1d1/0x200 [11248.135501] ? set_kthread_struct+0x70/0x70 [11248.136882] ret_from_fork+0x1f/0x30 [11264.056882] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [11264.058854] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [11264.073175] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11264.073374] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11264.073381] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11264.073384] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11264.073409] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11264.073413] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11264.073418] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [11264.073420] RDX: 0000000000000a5e RSI: 0000000000000003 RDI: ffff96848811cae0 [11264.073423] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [11264.073425] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [11264.073846] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11264.073850] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [11264.073852] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11264.073855] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [11264.073862] Call Trace: [11264.073882] [11264.073889] ? show_regs.cold.9+0x22/0x2f [11264.073900] ? watchdog_timer_fn.cold.6+0x62/0x10a [11264.073907] ? __hrtimer_run_queues+0x1c2/0x4d0 [11264.073911] ? watchdog+0x40/0x40 [11264.073915] ? hrtimer_interrupt+0x128/0x2c0 [11264.073920] ? smp_apic_timer_interrupt+0xa6/0x270 [11264.073940] ? apic_timer_interrupt+0xf/0x20 [11264.073945] [11264.073947] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11264.073952] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11264.073957] do_raw_spin_lock+0x177/0x190 [11264.073961] _raw_spin_lock+0x21/0x30 [11264.073967] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11264.076657] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11264.076736] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11264.076744] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11264.076746] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11264.076759] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11264.076764] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11264.076768] RAX: 0000000000002ad4 RBX: ffff96848811cae0 RCX: 0000000000000001 [11264.076771] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [11264.076773] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [11264.076776] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [11264.076789] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [11264.076793] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [11264.076796] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11264.076798] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [11264.076805] Call Trace: [11264.076811] [11264.076816] ? show_regs.cold.9+0x22/0x2f [11264.076825] ? watchdog_timer_fn.cold.6+0x62/0x10a [11264.076831] ? __hrtimer_run_queues+0x1c2/0x4d0 [11264.076836] ? watchdog+0x40/0x40 [11264.076840] ? hrtimer_interrupt+0x128/0x2c0 [11264.076844] ? smp_apic_timer_interrupt+0xa6/0x270 [11264.076853] ? apic_timer_interrupt+0xf/0x20 [11264.076857] [11264.076860] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11264.076867] do_raw_spin_lock+0x177/0x190 [11264.076872] _raw_spin_lock+0x21/0x30 [11264.076877] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11264.079255] ? do_raw_spin_unlock+0x75/0x190 [11264.079266] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11264.079520] ? do_wait_intr+0xf0/0xf0 [11264.079527] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11264.079759] mdc_close+0x53b/0xd70 [mdc] [11264.080861] lmv_close+0x18d/0x490 [lmv] [11264.080893] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [11264.081686] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11264.081819] ll_md_real_close+0xa5/0x2a0 [lustre] [11264.081915] ll_md_close+0x274/0x800 [lustre] [11264.082326] ll_file_release+0x128/0xc50 [lustre] [11264.082422] __fput+0x10d/0x440 [11264.082432] ____fput+0x12/0x20 [11264.082436] task_work_run+0xc6/0x110 [11264.082444] exit_to_usermode_loop+0x1dd/0x1f0 [11264.082451] do_syscall_64+0x3d6/0x440 [11264.082456] entry_SYSCALL_64_after_hwframe+0x49/0xae [11264.082464] RIP: 0033:0x7ff54af45cd8 [11264.082512] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11264.082516] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11264.082522] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [11264.082525] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [11264.082527] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [11264.082530] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [11264.082532] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11264.374753] obd_mod_free+0x46/0x140 [ptlrpc] [11264.376871] mdc_commit_open+0x91/0xb0 [mdc] [11264.378409] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11264.380755] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11264.386520] after_reply+0xe2c/0x1780 [ptlrpc] [11264.388538] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11264.390477] ptlrpcd+0xa82/0xe40 [ptlrpc] [11264.392142] ? do_wait_intr+0xf0/0xf0 [11264.393356] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11264.395332] kthread+0x1d1/0x200 [11264.396351] ? set_kthread_struct+0x70/0x70 [11264.398912] ret_from_fork+0x1f/0x30 [11268.060377] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [11268.074098] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11268.120877] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11268.134603] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11268.140165] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11268.146807] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11268.159881] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11268.169829] RAX: 000000000000420c RBX: ffff96848811cae0 RCX: 0000000000000001 [11268.174053] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [11268.176896] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [11268.180332] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [11268.184667] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [11268.188845] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [11268.193854] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11268.198114] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [11268.202152] Call Trace: [11268.204167] [11268.205150] ? show_regs.cold.9+0x22/0x2f [11268.206783] ? watchdog_timer_fn.cold.6+0x62/0x10a [11268.214160] ? __hrtimer_run_queues+0x1c2/0x4d0 [11268.216497] ? watchdog+0x40/0x40 [11268.221328] ? hrtimer_interrupt+0x128/0x2c0 [11268.225091] ? smp_apic_timer_interrupt+0xa6/0x270 [11268.227516] ? apic_timer_interrupt+0xf/0x20 [11268.229562] [11268.230704] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11268.234577] do_raw_spin_lock+0x177/0x190 [11268.236657] _raw_spin_lock+0x21/0x30 [11268.238492] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11268.241117] ? do_raw_spin_unlock+0x75/0x190 [11268.246334] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11268.248996] ? do_wait_intr+0xf0/0xf0 [11268.251615] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11268.254082] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11268.257731] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11268.264442] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11268.267030] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11268.273605] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11268.277233] mdc_intent_lock+0x353/0x7e0 [mdc] [11268.281726] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11268.290544] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11268.298197] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11268.301456] lmv_intent_open+0x338/0x1150 [lmv] [11268.303799] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11268.306921] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11268.316274] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11268.319760] ll_intent_lock+0xc5/0xaa0 [lustre] [11268.324833] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11268.328468] ll_lookup_it+0xe0e/0x2270 [lustre] [11268.331900] ? do_raw_spin_unlock+0x75/0x190 [11268.337808] ? _raw_spin_unlock+0x12/0x30 [11268.340105] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11268.342186] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11268.343814] ll_atomic_open+0x36a/0xd70 [lustre] [11268.350665] lookup_open+0x356/0xb40 [11268.359432] do_last+0x313/0xfc0 [11268.361628] ? nd_jump_root+0xe5/0x160 [11268.364692] ? path_init+0x437/0x520 [11268.367848] path_openat+0xf7/0x500 [11268.369828] ? do_raw_spin_unlock+0x75/0x190 [11268.371812] do_filp_open+0x99/0x140 [11268.374330] ? getname_flags+0x6e/0x330 [11268.378528] ? __check_object_size+0xff/0x256 [11268.379728] ? do_raw_spin_unlock+0x75/0x190 [11268.380914] ? _raw_spin_unlock+0x12/0x30 [11268.391156] do_sys_openat2+0x2b4/0x410 [11268.394057] do_sys_open+0x73/0xa0 [11268.407840] __x64_sys_openat+0x24/0x30 [11268.409289] do_syscall_64+0xc1/0x440 [11268.424365] entry_SYSCALL_64_after_hwframe+0x49/0xae [11268.426199] RIP: 0033:0x7ff54af45332 [11268.444269] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11268.477917] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11268.480446] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [11268.500273] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [11268.511215] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [11268.526008] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [11268.539239] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [11269.602695] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9310s! [11269.621455] Showing busy workqueues and worker pools: [11269.632033] workqueue events: flags=0x0 [11269.633897] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11269.642138] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [11269.653589] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [11269.663448] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [11269.687412] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11269.699948] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [11269.722662] workqueue events_long: flags=0x0 [11269.731304] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11269.741543] pending: lu_objects_flush [obdclass] [11269.747550] workqueue events_unbound: flags=0x2 [11269.755092] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [11269.762705] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11269.767914] workqueue events_power_efficient: flags=0x80 [11269.776846] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11269.787781] pending: neigh_periodic_work [11269.793620] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11269.802622] pending: do_cache_clean, check_lifetime [11269.808627] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11269.817489] pending: neigh_periodic_work [11269.819623] workqueue writeback: flags=0x4a [11269.829646] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11269.835748] pending: wb_workfn, wb_workfn [11269.838704] workqueue xprtiod: flags=0xa [11269.841915] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11269.849720] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [11269.869972] workqueue ipv6_addrconf: flags=0x40008 [11269.873777] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11269.882035] pending: addrconf_verify_work [11269.896543] workqueue knbd0-recv: flags=0x1a [11269.903607] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11269.915594] in-flight: 657:recv_work [11269.918801] workqueue ptlrpc_pinger: flags=0x2 [11269.926842] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11269.929027] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [11269.945151] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9311s workers=4 idle: 20831 12642 28859 [11269.951543] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11260s workers=2 idle: 659 [11276.062769] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [11276.067801] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11276.093058] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11276.099653] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11276.104115] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11276.107717] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11276.110211] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11276.119185] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11276.123203] RAX: 0000000000006e0c RBX: ffff96848811cae0 RCX: 0000000000000001 [11276.126996] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [11276.132106] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [11276.136029] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [11276.139795] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [11276.143711] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [11276.148626] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11276.151714] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [11276.154771] Call Trace: [11276.155921] [11276.156954] ? show_regs.cold.9+0x22/0x2f [11276.158469] ? watchdog_timer_fn.cold.6+0x62/0x10a [11276.160902] ? __hrtimer_run_queues+0x1c2/0x4d0 [11276.163704] ? watchdog+0x40/0x40 [11276.164875] ? hrtimer_interrupt+0x128/0x2c0 [11276.167057] ? smp_apic_timer_interrupt+0xa6/0x270 [11276.170070] ? apic_timer_interrupt+0xf/0x20 [11276.173319] [11276.174925] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11276.179695] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11276.183674] do_raw_spin_lock+0x177/0x190 [11276.186509] _raw_spin_lock+0x21/0x30 [11276.188956] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11276.194845] process_one_work+0x2c8/0x700 [11276.196802] worker_thread+0x296/0x6e0 [11276.208515] ? rescuer_thread+0x570/0x570 [11276.214175] kthread+0x1d1/0x200 [11276.217140] ? set_kthread_struct+0x70/0x70 [11276.223541] ret_from_fork+0x1f/0x30 [11292.058972] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [11292.062315] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11292.091676] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11292.099227] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11292.104462] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11292.106659] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11292.116442] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11292.122643] RAX: 0000000000006814 RBX: ffff96848811cae0 RCX: 0000000000000001 [11292.127658] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [11292.133994] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [11292.140269] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [11292.147019] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [11292.150616] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [11292.154704] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11292.157814] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [11292.161605] Call Trace: [11292.163877] [11292.166364] ? show_regs.cold.9+0x22/0x2f [11292.171015] ? watchdog_timer_fn.cold.6+0x62/0x10a [11292.175232] ? __hrtimer_run_queues+0x1c2/0x4d0 [11292.177471] ? watchdog+0x40/0x40 [11292.180816] ? hrtimer_interrupt+0x128/0x2c0 [11292.183593] ? smp_apic_timer_interrupt+0xa6/0x270 [11292.185922] ? apic_timer_interrupt+0xf/0x20 [11292.189423] [11292.192583] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11292.198999] do_raw_spin_lock+0x177/0x190 [11292.203995] _raw_spin_lock+0x21/0x30 [11292.205073] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11292.206605] ? do_raw_spin_unlock+0x75/0x190 [11292.211440] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11292.214566] ? do_wait_intr+0xf0/0xf0 [11292.216514] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11292.219299] mdc_close+0x53b/0xd70 [mdc] [11292.223693] lmv_close+0x18d/0x490 [lmv] [11292.225721] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [11292.228871] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11292.231401] ll_md_real_close+0xa5/0x2a0 [lustre] [11292.233378] ll_md_close+0x274/0x800 [lustre] [11292.235028] ll_file_release+0x128/0xc50 [lustre] [11292.237455] __fput+0x10d/0x440 [11292.241390] ____fput+0x12/0x20 [11292.242708] task_work_run+0xc6/0x110 [11292.245347] exit_to_usermode_loop+0x1dd/0x1f0 [11292.247303] do_syscall_64+0x3d6/0x440 [11292.249346] entry_SYSCALL_64_after_hwframe+0x49/0xae [11292.252529] RIP: 0033:0x7ff54af45cd8 [11292.254662] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11292.274342] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11292.278933] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [11292.281359] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [11292.285563] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [11292.293389] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [11292.297982] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11296.055144] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [11296.058437] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11296.061001] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [11296.088033] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11296.088050] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11296.088053] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11296.088068] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11296.088073] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11296.088079] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [11296.088081] RDX: 00000000000050fa RSI: 0000000000000003 RDI: ffff96848811cae0 [11296.088083] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [11296.088085] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [11296.088088] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11296.088091] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [11296.088093] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11296.088095] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [11296.088103] Call Trace: [11296.088110] [11296.088116] ? show_regs.cold.9+0x22/0x2f [11296.088127] ? watchdog_timer_fn.cold.6+0x62/0x10a [11296.088133] ? __hrtimer_run_queues+0x1c2/0x4d0 [11296.088138] ? watchdog+0x40/0x40 [11296.088142] ? hrtimer_interrupt+0x128/0x2c0 [11296.088146] ? smp_apic_timer_interrupt+0xa6/0x270 [11296.088155] ? apic_timer_interrupt+0xf/0x20 [11296.088159] [11296.088162] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11296.088167] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11296.088172] do_raw_spin_lock+0x177/0x190 [11296.088177] _raw_spin_lock+0x21/0x30 [11296.088184] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11296.088632] obd_mod_free+0x46/0x140 [ptlrpc] [11296.088919] mdc_commit_open+0x91/0xb0 [mdc] [11296.088955] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11296.089153] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11296.089364] after_reply+0xe2c/0x1780 [ptlrpc] [11296.089822] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11296.090042] ptlrpcd+0xa82/0xe40 [ptlrpc] [11296.090249] ? do_wait_intr+0xf0/0xf0 [11296.090261] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11296.090791] kthread+0x1d1/0x200 [11296.090798] ? set_kthread_struct+0x70/0x70 [11296.090802] ret_from_fork+0x1f/0x30 [11296.244427] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11296.267285] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11296.277752] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11296.283714] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11296.287409] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11296.305943] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11296.308901] RAX: 0000000000007f09 RBX: ffff96848811cae0 RCX: 0000000000000001 [11296.312062] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [11296.314663] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [11296.318018] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [11296.321249] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [11296.323926] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [11296.326979] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11296.329494] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [11296.332239] Call Trace: [11296.333519] [11296.334202] ? show_regs.cold.9+0x22/0x2f [11296.335876] ? watchdog_timer_fn.cold.6+0x62/0x10a [11296.337999] ? __hrtimer_run_queues+0x1c2/0x4d0 [11296.339841] ? watchdog+0x40/0x40 [11296.341315] ? hrtimer_interrupt+0x128/0x2c0 [11296.343210] ? smp_apic_timer_interrupt+0xa6/0x270 [11296.351046] ? apic_timer_interrupt+0xf/0x20 [11296.352312] [11296.357130] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11296.361478] do_raw_spin_lock+0x177/0x190 [11296.365280] _raw_spin_lock+0x21/0x30 [11296.366613] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11296.369577] ? do_raw_spin_unlock+0x75/0x190 [11296.370537] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11296.374400] ? do_wait_intr+0xf0/0xf0 [11296.375722] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11296.378708] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11296.381162] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11296.386327] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11296.389201] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11296.391664] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11296.393723] mdc_intent_lock+0x353/0x7e0 [mdc] [11296.395587] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11296.398317] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11296.401590] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11296.403978] lmv_intent_open+0x338/0x1150 [lmv] [11296.406615] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11296.409414] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11296.411741] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11296.413498] ll_intent_lock+0xc5/0xaa0 [lustre] [11296.415917] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11296.420871] ll_lookup_it+0xe0e/0x2270 [lustre] [11296.422039] ? do_raw_spin_unlock+0x75/0x190 [11296.424483] ? _raw_spin_unlock+0x12/0x30 [11296.425928] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11296.427416] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11296.429980] ll_atomic_open+0x36a/0xd70 [lustre] [11296.434232] lookup_open+0x356/0xb40 [11296.436205] do_last+0x313/0xfc0 [11296.437311] ? nd_jump_root+0xe5/0x160 [11296.439241] ? path_init+0x437/0x520 [11296.440766] path_openat+0xf7/0x500 [11296.442243] ? do_raw_spin_unlock+0x75/0x190 [11296.447069] do_filp_open+0x99/0x140 [11296.452185] ? getname_flags+0x6e/0x330 [11296.453958] ? __check_object_size+0xff/0x256 [11296.456261] ? do_raw_spin_unlock+0x75/0x190 [11296.461010] ? _raw_spin_unlock+0x12/0x30 [11296.462929] do_sys_openat2+0x2b4/0x410 [11296.464317] do_sys_open+0x73/0xa0 [11296.465342] __x64_sys_openat+0x24/0x30 [11296.467250] do_syscall_64+0xc1/0x440 [11296.468638] entry_SYSCALL_64_after_hwframe+0x49/0xae [11296.471479] RIP: 0033:0x7ff54af45332 [11296.473098] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11296.481320] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11296.484580] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [11296.487561] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [11296.494448] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [11296.500017] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [11296.503989] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [11300.319695] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9341s! [11300.326415] Showing busy workqueues and worker pools: [11300.332286] workqueue events: flags=0x0 [11300.334980] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11300.341005] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [11300.349457] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [11300.357217] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [11300.372110] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11300.382086] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [11300.387504] workqueue events_long: flags=0x0 [11300.391181] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11300.400597] pending: lu_objects_flush [obdclass] [11300.408503] workqueue events_unbound: flags=0x2 [11300.417494] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [11300.425428] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11300.437741] workqueue events_power_efficient: flags=0x80 [11300.441577] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11300.447401] pending: neigh_periodic_work [11300.450308] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11300.454980] pending: do_cache_clean, check_lifetime [11300.462369] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11300.470301] pending: neigh_periodic_work [11300.473367] workqueue writeback: flags=0x4a [11300.476521] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11300.480479] pending: wb_workfn, wb_workfn [11300.483347] workqueue xprtiod: flags=0xa [11300.487556] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11300.494005] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [11300.500595] workqueue ipv6_addrconf: flags=0x40008 [11300.503351] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11300.507725] pending: addrconf_verify_work [11300.509903] workqueue knbd0-recv: flags=0x1a [11300.512620] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11300.517962] in-flight: 657:recv_work [11300.525442] workqueue ptlrpc_pinger: flags=0x2 [11300.531402] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11300.536161] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [11300.540711] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9341s workers=4 idle: 20831 12642 28859 [11300.544106] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11290s workers=2 idle: 659 [11304.059585] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [11304.068793] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11304.105389] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11304.116370] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11304.122437] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11304.125914] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11304.131513] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11304.146914] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11304.155958] RAX: 000000000000428f RBX: ffff96848811cae0 RCX: 0000000000000001 [11304.162761] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [11304.167531] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [11304.173985] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [11304.179371] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [11304.182388] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [11304.190505] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11304.193651] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [11304.201457] Call Trace: [11304.206537] [11304.207326] ? show_regs.cold.9+0x22/0x2f [11304.209015] ? watchdog_timer_fn.cold.6+0x62/0x10a [11304.213412] ? __hrtimer_run_queues+0x1c2/0x4d0 [11304.215049] ? watchdog+0x40/0x40 [11304.216696] ? hrtimer_interrupt+0x128/0x2c0 [11304.218688] ? smp_apic_timer_interrupt+0xa6/0x270 [11304.221569] ? apic_timer_interrupt+0xf/0x20 [11304.222960] [11304.223596] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11304.234021] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11304.238355] do_raw_spin_lock+0x177/0x190 [11304.245233] _raw_spin_lock+0x21/0x30 [11304.249421] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11304.252505] process_one_work+0x2c8/0x700 [11304.254289] worker_thread+0x296/0x6e0 [11304.256075] ? rescuer_thread+0x570/0x570 [11304.258324] kthread+0x1d1/0x200 [11304.260209] ? set_kthread_struct+0x70/0x70 [11304.263295] ret_from_fork+0x1f/0x30 [11320.058031] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [11320.061845] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11320.129317] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11320.141308] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11320.147826] RIP: 0010:native_safe_halt+0x2e/0x50 [11320.153785] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11320.170419] RSP: 0018:ffffaf9fc1543ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11320.177745] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [11320.183988] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c20b4154 [11320.191199] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [11320.196019] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [11320.203843] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [11320.208831] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [11320.216387] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11320.227219] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [11320.233533] Call Trace: [11320.234363] [11320.236029] ? show_regs.cold.9+0x22/0x2f [11320.242886] ? watchdog_timer_fn.cold.6+0x62/0x10a [11320.247116] ? __hrtimer_run_queues+0x1c2/0x4d0 [11320.251913] ? watchdog+0x40/0x40 [11320.256035] ? hrtimer_interrupt+0x128/0x2c0 [11320.259973] ? smp_apic_timer_interrupt+0xa6/0x270 [11320.264321] ? apic_timer_interrupt+0xf/0x20 [11320.268870] [11320.269880] ? native_safe_halt+0x2e/0x50 [11320.274893] kvm_wait+0xcd/0xe0 [11320.276168] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11320.281789] do_raw_spin_lock+0x177/0x190 [11320.283671] _raw_spin_lock+0x21/0x30 [11320.290341] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11320.296323] ? do_raw_spin_unlock+0x75/0x190 [11320.298988] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11320.304119] ? do_wait_intr+0xf0/0xf0 [11320.307200] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11320.312160] mdc_close+0x53b/0xd70 [mdc] [11320.318222] lmv_close+0x18d/0x490 [lmv] [11320.325889] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [11320.328940] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11320.334156] ll_md_real_close+0xa5/0x2a0 [lustre] [11320.339784] ll_md_close+0x274/0x800 [lustre] [11320.344770] ll_file_release+0x128/0xc50 [lustre] [11320.347608] __fput+0x10d/0x440 [11320.349111] ____fput+0x12/0x20 [11320.355229] task_work_run+0xc6/0x110 [11320.357776] exit_to_usermode_loop+0x1dd/0x1f0 [11320.360958] do_syscall_64+0x3d6/0x440 [11320.365912] entry_SYSCALL_64_after_hwframe+0x49/0xae [11320.371726] RIP: 0033:0x7ff54af45cd8 [11320.374850] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11320.389653] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11320.397335] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [11320.403950] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [11320.410268] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [11320.417021] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [11320.419445] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11324.062685] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [11324.074908] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11324.125206] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11324.131647] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11324.139882] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11324.148252] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11324.159636] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11324.170282] RAX: 0000000000004687 RBX: ffff96848811cae0 RCX: 0000000000000001 [11324.183016] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [11324.189955] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [11324.196070] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [11324.199603] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [11324.208303] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [11324.216446] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11324.223877] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [11324.230969] Call Trace: [11324.235615] [11324.237889] ? show_regs.cold.9+0x22/0x2f [11324.242288] ? watchdog_timer_fn.cold.6+0x62/0x10a [11324.248398] ? __hrtimer_run_queues+0x1c2/0x4d0 [11324.255388] ? watchdog+0x40/0x40 [11324.261207] ? hrtimer_interrupt+0x128/0x2c0 [11324.271368] ? smp_apic_timer_interrupt+0xa6/0x270 [11324.280066] ? apic_timer_interrupt+0xf/0x20 [11324.284393] [11324.285176] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11324.289164] do_raw_spin_lock+0x177/0x190 [11324.291768] _raw_spin_lock+0x21/0x30 [11324.294311] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11324.305130] ? do_raw_spin_unlock+0x75/0x190 [11324.310710] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11324.315733] ? do_wait_intr+0xf0/0xf0 [11324.319206] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11324.323264] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11324.326839] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11324.330929] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11324.335016] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11324.340118] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11324.348215] mdc_intent_lock+0x353/0x7e0 [mdc] [11324.353782] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11324.359860] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11324.362123] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11324.368201] lmv_intent_open+0x338/0x1150 [lmv] [11324.373345] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11324.380233] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11324.383789] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11324.386316] ll_intent_lock+0xc5/0xaa0 [lustre] [11324.390071] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11324.397051] ll_lookup_it+0xe0e/0x2270 [lustre] [11324.399372] ? do_raw_spin_unlock+0x75/0x190 [11324.402367] ? _raw_spin_unlock+0x12/0x30 [11324.405116] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11324.408969] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11324.417085] ll_atomic_open+0x36a/0xd70 [lustre] [11324.420297] lookup_open+0x356/0xb40 [11324.423617] do_last+0x313/0xfc0 [11324.425036] ? nd_jump_root+0xe5/0x160 [11324.427144] ? path_init+0x437/0x520 [11324.428711] path_openat+0xf7/0x500 [11324.430051] ? do_raw_spin_unlock+0x75/0x190 [11324.431845] do_filp_open+0x99/0x140 [11324.433292] ? getname_flags+0x6e/0x330 [11324.435003] ? __check_object_size+0xff/0x256 [11324.436806] ? do_raw_spin_unlock+0x75/0x190 [11324.438513] ? _raw_spin_unlock+0x12/0x30 [11324.440091] do_sys_openat2+0x2b4/0x410 [11324.444029] do_sys_open+0x73/0xa0 [11324.445353] __x64_sys_openat+0x24/0x30 [11324.450468] do_syscall_64+0xc1/0x440 [11324.452788] entry_SYSCALL_64_after_hwframe+0x49/0xae [11324.456213] RIP: 0033:0x7ff54af45332 [11324.457859] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11324.471735] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11324.479239] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [11324.488372] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [11324.494222] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [11324.500341] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [11324.507491] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [11328.055465] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [11328.064984] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11328.094204] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11328.102827] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11328.106953] RIP: 0010:native_safe_halt+0x2e/0x50 [11328.109122] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11328.124916] RSP: 0000:ffffaf9fc1b97bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11328.129206] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [11328.132190] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff96848811cae0 [11328.137819] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [11328.142996] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [11328.148780] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11328.152004] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [11328.156157] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11328.161127] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [11328.163722] Call Trace: [11328.165085] [11328.165949] ? show_regs.cold.9+0x22/0x2f [11328.170901] ? watchdog_timer_fn.cold.6+0x62/0x10a [11328.174123] ? __hrtimer_run_queues+0x1c2/0x4d0 [11328.183727] ? watchdog+0x40/0x40 [11328.184893] ? hrtimer_interrupt+0x128/0x2c0 [11328.186111] ? smp_apic_timer_interrupt+0xa6/0x270 [11328.189742] ? apic_timer_interrupt+0xf/0x20 [11328.191840] [11328.192983] ? native_safe_halt+0x2e/0x50 [11328.197675] kvm_wait+0xcd/0xe0 [11328.202923] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11328.212786] do_raw_spin_lock+0x177/0x190 [11328.214149] _raw_spin_lock+0x21/0x30 [11328.221279] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11328.226808] obd_mod_free+0x46/0x140 [ptlrpc] [11328.232157] mdc_commit_open+0x91/0xb0 [mdc] [11328.243623] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11328.252125] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11328.255988] after_reply+0xe2c/0x1780 [ptlrpc] [11328.258832] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11328.261699] ptlrpcd+0xa82/0xe40 [ptlrpc] [11328.264426] ? do_wait_intr+0xf0/0xf0 [11328.266056] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11328.268600] kthread+0x1d1/0x200 [11328.270016] ? set_kthread_struct+0x70/0x70 [11328.271883] ret_from_fork+0x1f/0x30 [11331.040849] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9372s! [11331.051835] Showing busy workqueues and worker pools: [11331.055873] workqueue events: flags=0x0 [11331.058638] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11331.064578] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [11331.077192] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [11331.088145] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [11331.097166] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11331.104734] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [11331.114478] workqueue events_long: flags=0x0 [11331.116713] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11331.121049] pending: lu_objects_flush [obdclass] [11331.125475] workqueue events_unbound: flags=0x2 [11331.129533] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [11331.133458] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11331.139829] workqueue events_power_efficient: flags=0x80 [11331.142850] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11331.152167] pending: neigh_periodic_work [11331.157969] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11331.160221] pending: do_cache_clean, check_lifetime [11331.169717] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11331.174931] pending: neigh_periodic_work [11331.182303] workqueue writeback: flags=0x4a [11331.185254] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11331.193995] pending: wb_workfn, wb_workfn [11331.199797] workqueue xprtiod: flags=0xa [11331.202264] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11331.208478] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [11331.211954] workqueue ipv6_addrconf: flags=0x40008 [11331.214076] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11331.217880] pending: addrconf_verify_work [11331.222897] workqueue knbd0-recv: flags=0x1a [11331.226161] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11331.232206] in-flight: 657:recv_work [11331.234925] workqueue ptlrpc_pinger: flags=0x2 [11331.239020] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11331.242015] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [11331.245844] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9372s workers=4 idle: 20831 12642 28859 [11331.252647] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11321s workers=2 idle: 659 [11332.060006] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [11332.067403] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11332.114829] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11332.121161] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11332.128893] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11332.133002] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11332.135805] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11332.145950] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11332.150919] RAX: 000000000000548a RBX: ffff96848811cae0 RCX: 0000000000000001 [11332.156167] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [11332.160985] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [11332.165024] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [11332.168230] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [11332.172055] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [11332.176195] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11332.178966] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [11332.184340] Call Trace: [11332.184941] [11332.185557] ? show_regs.cold.9+0x22/0x2f [11332.186539] ? watchdog_timer_fn.cold.6+0x62/0x10a [11332.189654] ? __hrtimer_run_queues+0x1c2/0x4d0 [11332.191164] ? watchdog+0x40/0x40 [11332.192756] ? hrtimer_interrupt+0x128/0x2c0 [11332.194665] ? smp_apic_timer_interrupt+0xa6/0x270 [11332.196185] ? apic_timer_interrupt+0xf/0x20 [11332.198822] [11332.199372] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11332.200944] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11332.205095] do_raw_spin_lock+0x177/0x190 [11332.206158] _raw_spin_lock+0x21/0x30 [11332.208017] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11332.213039] process_one_work+0x2c8/0x700 [11332.217954] worker_thread+0x296/0x6e0 [11332.219846] ? rescuer_thread+0x570/0x570 [11332.224082] kthread+0x1d1/0x200 [11332.228175] ? set_kthread_struct+0x70/0x70 [11332.234753] ret_from_fork+0x1f/0x30 [11348.060499] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [11348.073497] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11348.101723] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11348.110015] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11348.116100] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11348.118728] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11348.125148] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11348.145883] RAX: 0000000000004860 RBX: ffff96848811cae0 RCX: 0000000000000001 [11348.149682] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [11348.152173] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [11348.157776] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [11348.161815] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [11348.165055] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [11348.170527] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11348.172204] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [11348.176091] Call Trace: [11348.178061] [11348.178934] ? show_regs.cold.9+0x22/0x2f [11348.180911] ? watchdog_timer_fn.cold.6+0x62/0x10a [11348.182973] ? __hrtimer_run_queues+0x1c2/0x4d0 [11348.185042] ? watchdog+0x40/0x40 [11348.186313] ? hrtimer_interrupt+0x128/0x2c0 [11348.188774] ? smp_apic_timer_interrupt+0xa6/0x270 [11348.192529] ? apic_timer_interrupt+0xf/0x20 [11348.196282] [11348.198415] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11348.204496] do_raw_spin_lock+0x177/0x190 [11348.207099] _raw_spin_lock+0x21/0x30 [11348.209700] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11348.211985] ? do_raw_spin_unlock+0x75/0x190 [11348.213688] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11348.216095] ? do_wait_intr+0xf0/0xf0 [11348.217648] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11348.220094] mdc_close+0x53b/0xd70 [mdc] [11348.221611] lmv_close+0x18d/0x490 [lmv] [11348.223086] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [11348.225367] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11348.227706] ll_md_real_close+0xa5/0x2a0 [lustre] [11348.229479] ll_md_close+0x274/0x800 [lustre] [11348.231016] ll_file_release+0x128/0xc50 [lustre] [11348.233013] __fput+0x10d/0x440 [11348.234047] ____fput+0x12/0x20 [11348.235024] task_work_run+0xc6/0x110 [11348.236418] exit_to_usermode_loop+0x1dd/0x1f0 [11348.237851] do_syscall_64+0x3d6/0x440 [11348.238911] entry_SYSCALL_64_after_hwframe+0x49/0xae [11348.240509] RIP: 0033:0x7ff54af45cd8 [11348.241698] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11348.252341] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11348.266464] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [11348.268956] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [11348.274405] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [11348.282899] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [11348.287795] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11352.061617] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [11352.069902] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11352.104619] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11352.111575] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11352.115420] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11352.117728] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11352.130756] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11352.141740] RAX: 000000000000297e RBX: ffff96848811cae0 RCX: 0000000000000001 [11352.144186] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [11352.150014] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [11352.155732] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [11352.164472] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [11352.171434] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [11352.176544] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11352.181464] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [11352.184859] Call Trace: [11352.186015] [11352.187379] ? show_regs.cold.9+0x22/0x2f [11352.189914] ? watchdog_timer_fn.cold.6+0x62/0x10a [11352.193045] ? __hrtimer_run_queues+0x1c2/0x4d0 [11352.196593] ? watchdog+0x40/0x40 [11352.199069] ? hrtimer_interrupt+0x128/0x2c0 [11352.201539] ? smp_apic_timer_interrupt+0xa6/0x270 [11352.203990] ? apic_timer_interrupt+0xf/0x20 [11352.209947] [11352.210955] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11352.213617] do_raw_spin_lock+0x177/0x190 [11352.215902] _raw_spin_lock+0x21/0x30 [11352.218544] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11352.221655] ? do_raw_spin_unlock+0x75/0x190 [11352.229056] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11352.237124] ? do_wait_intr+0xf0/0xf0 [11352.241731] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11352.245555] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11352.247615] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11352.253896] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11352.258007] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11352.266992] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11352.273238] mdc_intent_lock+0x353/0x7e0 [mdc] [11352.280779] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11352.282821] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11352.290941] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11352.295081] lmv_intent_open+0x338/0x1150 [lmv] [11352.297153] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11352.299593] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11352.305791] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11352.307909] ll_intent_lock+0xc5/0xaa0 [lustre] [11352.311506] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11352.315890] ll_lookup_it+0xe0e/0x2270 [lustre] [11352.319674] ? do_raw_spin_unlock+0x75/0x190 [11352.321741] ? _raw_spin_unlock+0x12/0x30 [11352.324190] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11352.326706] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11352.328336] ll_atomic_open+0x36a/0xd70 [lustre] [11352.337676] lookup_open+0x356/0xb40 [11352.339473] do_last+0x313/0xfc0 [11352.340932] ? nd_jump_root+0xe5/0x160 [11352.342730] ? path_init+0x437/0x520 [11352.344752] path_openat+0xf7/0x500 [11352.349497] ? do_raw_spin_unlock+0x75/0x190 [11352.350963] do_filp_open+0x99/0x140 [11352.363403] ? getname_flags+0x6e/0x330 [11352.364577] ? __check_object_size+0xff/0x256 [11352.367832] ? do_raw_spin_unlock+0x75/0x190 [11352.372452] ? _raw_spin_unlock+0x12/0x30 [11352.375757] do_sys_openat2+0x2b4/0x410 [11352.376999] do_sys_open+0x73/0xa0 [11352.377896] __x64_sys_openat+0x24/0x30 [11352.378808] do_syscall_64+0xc1/0x440 [11352.380032] entry_SYSCALL_64_after_hwframe+0x49/0xae [11352.381601] RIP: 0033:0x7ff54af45332 [11352.382622] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11352.391070] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11352.392660] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [11352.394331] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [11352.395849] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [11352.397493] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [11352.399267] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [11356.064640] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [11356.075119] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11356.099965] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11356.105992] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11356.111289] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11356.114677] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11356.138908] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11356.144934] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [11356.147648] RDX: 00000000000065d9 RSI: 0000000000000003 RDI: ffff96848811cae0 [11356.156302] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [11356.167538] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [11356.172563] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11356.175340] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [11356.183133] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11356.185228] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [11356.199335] Call Trace: [11356.201004] [11356.202311] ? show_regs.cold.9+0x22/0x2f [11356.208660] ? watchdog_timer_fn.cold.6+0x62/0x10a [11356.218676] ? __hrtimer_run_queues+0x1c2/0x4d0 [11356.225051] ? watchdog+0x40/0x40 [11356.228046] ? hrtimer_interrupt+0x128/0x2c0 [11356.230840] ? smp_apic_timer_interrupt+0xa6/0x270 [11356.235338] ? apic_timer_interrupt+0xf/0x20 [11356.236948] [11356.241055] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11356.244232] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11356.247429] do_raw_spin_lock+0x177/0x190 [11356.250325] _raw_spin_lock+0x21/0x30 [11356.253307] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11356.257261] obd_mod_free+0x46/0x140 [ptlrpc] [11356.259221] mdc_commit_open+0x91/0xb0 [mdc] [11356.267074] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11356.272472] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11356.278622] after_reply+0xe2c/0x1780 [ptlrpc] [11356.285023] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11356.296009] ptlrpcd+0xa82/0xe40 [ptlrpc] [11356.300994] ? do_wait_intr+0xf0/0xf0 [11356.306370] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11356.312504] kthread+0x1d1/0x200 [11356.315258] ? set_kthread_struct+0x70/0x70 [11356.317088] ret_from_fork+0x1f/0x30 [11360.059086] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [11360.064596] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11360.095675] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11360.101774] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11360.106700] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11360.111115] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11360.114413] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11360.124633] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11360.128561] RAX: 0000000000000b9c RBX: ffff96848811cae0 RCX: 0000000000000001 [11360.132995] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [11360.136829] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [11360.140629] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [11360.144179] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [11360.148352] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [11360.151856] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11360.154869] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [11360.158918] Call Trace: [11360.160147] [11360.161258] ? show_regs.cold.9+0x22/0x2f [11360.163621] ? watchdog_timer_fn.cold.6+0x62/0x10a [11360.166084] ? __hrtimer_run_queues+0x1c2/0x4d0 [11360.168334] ? watchdog+0x40/0x40 [11360.170144] ? hrtimer_interrupt+0x128/0x2c0 [11360.172294] ? smp_apic_timer_interrupt+0xa6/0x270 [11360.174239] ? apic_timer_interrupt+0xf/0x20 [11360.176435] [11360.177585] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11360.179990] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11360.182673] do_raw_spin_lock+0x177/0x190 [11360.184757] _raw_spin_lock+0x21/0x30 [11360.186849] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11360.189965] process_one_work+0x2c8/0x700 [11360.191937] worker_thread+0x296/0x6e0 [11360.193790] ? rescuer_thread+0x570/0x570 [11360.195728] kthread+0x1d1/0x200 [11360.197491] ? set_kthread_struct+0x70/0x70 [11360.199583] ret_from_fork+0x1f/0x30 [11361.763765] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9402s! [11361.775958] Showing busy workqueues and worker pools: [11361.789487] workqueue events: flags=0x0 [11361.791605] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11361.798529] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [11361.807393] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [11361.815584] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [11361.827926] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11361.831207] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [11361.835695] workqueue events_long: flags=0x0 [11361.837201] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11361.844669] pending: lu_objects_flush [obdclass] [11361.848910] workqueue events_unbound: flags=0x2 [11361.851556] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [11361.859702] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11361.866501] workqueue events_power_efficient: flags=0x80 [11361.869019] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11361.874612] pending: neigh_periodic_work [11361.879656] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11361.883889] pending: do_cache_clean, check_lifetime [11361.886994] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11361.891208] pending: neigh_periodic_work [11361.894654] workqueue writeback: flags=0x4a [11361.896548] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11361.902524] pending: wb_workfn, wb_workfn [11361.905145] workqueue xprtiod: flags=0xa [11361.906756] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11361.912835] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [11361.919752] workqueue ipv6_addrconf: flags=0x40008 [11361.921978] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11361.927092] pending: addrconf_verify_work [11361.929478] workqueue knbd0-recv: flags=0x1a [11361.932580] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11361.937812] in-flight: 657:recv_work [11361.945559] workqueue ptlrpc_pinger: flags=0x2 [11361.953646] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11361.960648] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [11361.968676] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9403s workers=4 idle: 20831 12642 28859 [11361.973025] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11352s workers=2 idle: 659 [11374.106267] rcu: INFO: rcu_sched self-detected stall on CPU [11374.113260] rcu: 0-...!: (4981774 ticks this GP) idle=98a/1/0x4000000000000002 softirq=125154/125154 fqs=959 [11374.125222] (t=9420266 jiffies g=253585 q=5892) [11374.128881] rcu: rcu_sched kthread starved for 9415226 jiffies! g253585 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [11374.143372] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [11374.150253] rcu: RCU grace-period kthread stack dump: [11374.154267] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [11374.164280] Call Trace: [11374.167041] __schedule+0x351/0xcb0 [11374.169628] schedule+0xc0/0x180 [11374.175585] schedule_timeout+0xb4/0x190 [11374.176793] ? do_raw_spin_unlock+0x75/0x190 [11374.181082] ? __next_timer_interrupt+0x160/0x160 [11374.184341] ? prepare_to_swait_event+0xf3/0x140 [11374.188563] rcu_gp_kthread+0x8dd/0x15b0 [11374.190845] ? force_qs_rnp+0x2c0/0x2c0 [11374.195848] kthread+0x1d1/0x200 [11374.197697] ? set_kthread_struct+0x70/0x70 [11374.204779] ret_from_fork+0x1f/0x30 [11374.209184] rcu: Stack dump where RCU GP kthread last ran: [11374.215593] Sending NMI from CPU 0 to CPUs 2: [11374.229524] NMI backtrace for cpu 2 [11374.229536] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11374.229538] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11374.229540] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11374.229543] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11374.229545] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11374.229548] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000216 [11374.229551] RAX: 00000000000072a4 RBX: ffff96848811cae0 RCX: 0000000000000001 [11374.229552] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [11374.229554] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [11374.229555] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [11374.229557] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [11374.229558] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [11374.229560] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11374.229561] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [11374.229563] Call Trace: [11374.229563] [11374.229564] ? show_regs.cold.9+0x22/0x2f [11374.229565] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [11374.229566] ? nmi_cpu_backtrace_handler+0x15/0x40 [11374.229567] ? nmi_handle+0xbd/0x210 [11374.229568] ? default_do_nmi+0x4b/0x1b0 [11374.229569] ? do_nmi+0x280/0x380 [11374.229570] ? end_repeat_nmi+0x16/0x1f [11374.229572] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11374.229573] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11374.229575] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11374.229576] [11374.229577] do_raw_spin_lock+0x177/0x190 [11374.229578] _raw_spin_lock+0x21/0x30 [11374.229579] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11374.229581] process_one_work+0x2c8/0x700 [11374.229582] worker_thread+0x296/0x6e0 [11374.229583] ? rescuer_thread+0x570/0x570 [11374.229584] kthread+0x1d1/0x200 [11374.229585] ? set_kthread_struct+0x70/0x70 [11374.229586] ret_from_fork+0x1f/0x30 [11374.230028] NMI backtrace for cpu 0 [11374.455571] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11374.462405] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11374.467201] Call Trace: [11374.468341] [11374.470983] ? dump_stack+0xbb/0x10e [11374.472289] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [11374.477372] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [11374.483373] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [11374.486564] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [11374.491776] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [11374.497263] ? rcu_dump_cpu_stacks+0x11b/0x171 [11374.499404] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [11374.503479] ? update_process_times+0x6d/0xc0 [11374.511420] ? tick_sched_handle.isra.18+0x27/0x90 [11374.515009] ? tick_sched_timer+0x53/0xb0 [11374.518026] ? __hrtimer_run_queues+0x1c2/0x4d0 [11374.521597] ? tick_sched_handle.isra.18+0x90/0x90 [11374.524760] ? hrtimer_interrupt+0x128/0x2c0 [11374.529926] ? smp_apic_timer_interrupt+0xa6/0x270 [11374.531690] ? apic_timer_interrupt+0xf/0x20 [11374.538422] [11374.539990] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11374.543433] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11374.557492] ? do_raw_spin_lock+0x177/0x190 [11374.564410] ? _raw_spin_lock+0x21/0x30 [11374.565910] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11374.570171] ? obd_mod_free+0x46/0x140 [ptlrpc] [11374.573364] ? mdc_commit_open+0x91/0xb0 [mdc] [11374.577705] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11374.587541] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11374.594961] ? after_reply+0xe2c/0x1780 [ptlrpc] [11374.596951] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11374.603958] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [11374.607362] ? do_wait_intr+0xf0/0xf0 [11374.609433] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11374.616845] ? kthread+0x1d1/0x200 [11374.622075] ? set_kthread_struct+0x70/0x70 [11374.632717] ? ret_from_fork+0x1f/0x30 [11376.058604] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [11376.062847] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11376.100710] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11376.113717] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11376.123422] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11376.130752] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11376.151590] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [11376.160772] RAX: 000000000000611d RBX: ffff96848811cae0 RCX: 0000000000000001 [11376.163765] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [11376.171940] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [11376.176259] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [11376.182273] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [11376.189047] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [11376.195569] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11376.197439] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [11376.201469] Call Trace: [11376.202454] [11376.203299] ? show_regs.cold.9+0x22/0x2f [11376.205216] ? watchdog_timer_fn.cold.6+0x62/0x10a [11376.213783] ? __hrtimer_run_queues+0x1c2/0x4d0 [11376.217525] ? watchdog+0x40/0x40 [11376.219612] ? hrtimer_interrupt+0x128/0x2c0 [11376.224854] ? smp_apic_timer_interrupt+0xa6/0x270 [11376.232544] ? apic_timer_interrupt+0xf/0x20 [11376.240858] [11376.246434] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11376.251945] do_raw_spin_lock+0x177/0x190 [11376.253812] _raw_spin_lock+0x21/0x30 [11376.257642] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11376.260813] ? do_raw_spin_unlock+0x75/0x190 [11376.264399] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11376.267790] ? do_wait_intr+0xf0/0xf0 [11376.277553] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11376.283550] mdc_close+0x53b/0xd70 [mdc] [11376.287528] lmv_close+0x18d/0x490 [lmv] [11376.289602] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [11376.295288] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11376.298571] ll_md_real_close+0xa5/0x2a0 [lustre] [11376.301618] ll_md_close+0x274/0x800 [lustre] [11376.304782] ll_file_release+0x128/0xc50 [lustre] [11376.309159] __fput+0x10d/0x440 [11376.317565] ____fput+0x12/0x20 [11376.319462] task_work_run+0xc6/0x110 [11376.322886] exit_to_usermode_loop+0x1dd/0x1f0 [11376.325639] do_syscall_64+0x3d6/0x440 [11376.328130] entry_SYSCALL_64_after_hwframe+0x49/0xae [11376.330609] RIP: 0033:0x7ff54af45cd8 [11376.332189] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11376.340060] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11376.343630] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [11376.346630] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [11376.349652] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [11376.353607] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [11376.356657] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11380.060998] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [11380.067472] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11380.116072] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11380.123721] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11380.135967] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11380.144291] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11380.169398] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11380.176787] RAX: 0000000000007e1f RBX: ffff96848811cae0 RCX: 0000000000000001 [11380.184298] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [11380.187692] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [11380.194076] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [11380.202641] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [11380.209991] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [11380.217165] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11380.219599] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [11380.227310] Call Trace: [11380.227840] [11380.229518] ? show_regs.cold.9+0x22/0x2f [11380.232507] ? watchdog_timer_fn.cold.6+0x62/0x10a [11380.235610] ? __hrtimer_run_queues+0x1c2/0x4d0 [11380.237847] ? watchdog+0x40/0x40 [11380.239510] ? hrtimer_interrupt+0x128/0x2c0 [11380.243188] ? smp_apic_timer_interrupt+0xa6/0x270 [11380.246696] ? apic_timer_interrupt+0xf/0x20 [11380.250947] [11380.253223] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11380.254789] do_raw_spin_lock+0x177/0x190 [11380.259219] _raw_spin_lock+0x21/0x30 [11380.264546] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11380.269003] ? do_raw_spin_unlock+0x75/0x190 [11380.272368] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11380.275168] ? do_wait_intr+0xf0/0xf0 [11380.279920] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11380.281814] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11380.288212] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11380.293189] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11380.296844] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11380.307546] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11380.313449] mdc_intent_lock+0x353/0x7e0 [mdc] [11380.315994] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11380.321082] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11380.324573] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11380.329147] lmv_intent_open+0x338/0x1150 [lmv] [11380.333752] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11380.336290] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11380.337854] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11380.341182] ll_intent_lock+0xc5/0xaa0 [lustre] [11380.342953] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11380.350736] ll_lookup_it+0xe0e/0x2270 [lustre] [11380.354096] ? do_raw_spin_unlock+0x75/0x190 [11380.357771] ? _raw_spin_unlock+0x12/0x30 [11380.362412] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11380.367068] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11380.369570] ll_atomic_open+0x36a/0xd70 [lustre] [11380.376604] lookup_open+0x356/0xb40 [11380.378689] do_last+0x313/0xfc0 [11380.383200] ? nd_jump_root+0xe5/0x160 [11380.388512] ? path_init+0x437/0x520 [11380.396117] path_openat+0xf7/0x500 [11380.402012] ? do_raw_spin_unlock+0x75/0x190 [11380.404512] do_filp_open+0x99/0x140 [11380.406868] ? getname_flags+0x6e/0x330 [11380.408001] ? __check_object_size+0xff/0x256 [11380.410540] ? do_raw_spin_unlock+0x75/0x190 [11380.413090] ? _raw_spin_unlock+0x12/0x30 [11380.415366] do_sys_openat2+0x2b4/0x410 [11380.418649] do_sys_open+0x73/0xa0 [11380.426179] __x64_sys_openat+0x24/0x30 [11380.431008] do_syscall_64+0xc1/0x440 [11380.435563] entry_SYSCALL_64_after_hwframe+0x49/0xae [11380.440798] RIP: 0033:0x7ff54af45332 [11380.447150] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11380.471305] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11380.476770] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [11380.480360] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [11380.483791] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [11380.493771] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [11380.498816] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [11392.479033] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9433s! [11392.495530] Showing busy workqueues and worker pools: [11392.506192] workqueue events: flags=0x0 [11392.515573] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11392.532639] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [11392.555806] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [11392.567430] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [11392.592146] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11392.602440] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [11392.616066] workqueue events_long: flags=0x0 [11392.620974] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11392.631297] pending: lu_objects_flush [obdclass] [11392.635579] workqueue events_unbound: flags=0x2 [11392.640076] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [11392.646532] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11392.660735] workqueue events_power_efficient: flags=0x80 [11392.667114] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11392.676413] pending: neigh_periodic_work [11392.678608] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11392.690881] pending: do_cache_clean, check_lifetime [11392.702030] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11392.715567] pending: neigh_periodic_work [11392.722972] workqueue writeback: flags=0x4a [11392.728473] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11392.738547] pending: wb_workfn, wb_workfn [11392.743730] workqueue xprtiod: flags=0xa [11392.751252] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11392.758146] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [11392.764230] workqueue ipv6_addrconf: flags=0x40008 [11392.766675] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11392.779266] pending: addrconf_verify_work [11392.787174] workqueue knbd0-recv: flags=0x1a [11392.791290] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11392.804047] in-flight: 657:recv_work [11392.808574] workqueue ptlrpc_pinger: flags=0x2 [11392.816527] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11392.823560] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [11392.833691] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9433s workers=4 idle: 20831 12642 28859 [11392.842556] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11383s workers=2 idle: 659 [11400.059196] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [11400.063480] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11400.098546] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11400.104558] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11400.114886] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11400.122230] RIP: 0010:native_safe_halt+0x2e/0x50 [11400.124865] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11400.132476] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11400.134741] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [11400.138394] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c2134154 [11400.140619] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [11400.143572] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [11400.146974] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [11400.150519] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [11400.154515] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11400.157363] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [11400.160320] Call Trace: [11400.161396] [11400.162369] ? show_regs.cold.9+0x22/0x2f [11400.163957] ? watchdog_timer_fn.cold.6+0x62/0x10a [11400.166326] ? __hrtimer_run_queues+0x1c2/0x4d0 [11400.168698] ? watchdog+0x40/0x40 [11400.170370] ? hrtimer_interrupt+0x128/0x2c0 [11400.172447] ? smp_apic_timer_interrupt+0xa6/0x270 [11400.174640] ? apic_timer_interrupt+0xf/0x20 [11400.176705] [11400.177471] ? native_safe_halt+0x2e/0x50 [11400.178876] kvm_wait+0xcd/0xe0 [11400.180340] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11400.182761] do_raw_spin_lock+0x177/0x190 [11400.185227] _raw_spin_lock+0x21/0x30 [11400.187016] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11400.189506] process_one_work+0x2c8/0x700 [11400.191202] worker_thread+0x296/0x6e0 [11400.193527] ? rescuer_thread+0x570/0x570 [11400.196485] kthread+0x1d1/0x200 [11400.198647] ? set_kthread_struct+0x70/0x70 [11400.202259] ret_from_fork+0x1f/0x30 [11404.062691] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [11404.070215] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11404.115372] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11404.123081] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11404.131525] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11404.136544] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11404.162596] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11404.170428] RAX: 0000000000007b27 RBX: ffff96848811cae0 RCX: 0000000000000001 [11404.182644] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [11404.188407] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [11404.193340] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [11404.197580] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [11404.204425] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [11404.211998] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11404.214913] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [11404.223653] Call Trace: [11404.224484] [11404.225147] ? show_regs.cold.9+0x22/0x2f [11404.228810] ? watchdog_timer_fn.cold.6+0x62/0x10a [11404.233831] ? __hrtimer_run_queues+0x1c2/0x4d0 [11404.235367] ? watchdog+0x40/0x40 [11404.236598] ? hrtimer_interrupt+0x128/0x2c0 [11404.238712] ? smp_apic_timer_interrupt+0xa6/0x270 [11404.241612] ? apic_timer_interrupt+0xf/0x20 [11404.243240] [11404.247722] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11404.256555] do_raw_spin_lock+0x177/0x190 [11404.260875] _raw_spin_lock+0x21/0x30 [11404.263589] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11404.269272] ? do_raw_spin_unlock+0x75/0x190 [11404.276587] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11404.282225] ? do_wait_intr+0xf0/0xf0 [11404.284561] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11404.287540] mdc_close+0x53b/0xd70 [mdc] [11404.289482] lmv_close+0x18d/0x490 [lmv] [11404.291636] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [11404.294044] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11404.295872] ll_md_real_close+0xa5/0x2a0 [lustre] [11404.297390] ll_md_close+0x274/0x800 [lustre] [11404.298718] ll_file_release+0x128/0xc50 [lustre] [11404.300874] __fput+0x10d/0x440 [11404.302576] ____fput+0x12/0x20 [11404.303983] task_work_run+0xc6/0x110 [11404.310599] exit_to_usermode_loop+0x1dd/0x1f0 [11404.311912] do_syscall_64+0x3d6/0x440 [11404.313918] entry_SYSCALL_64_after_hwframe+0x49/0xae [11404.316163] RIP: 0033:0x7ff54af45cd8 [11404.317574] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11404.328575] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11404.332201] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [11404.340431] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [11404.344416] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [11404.348880] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [11404.352057] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11408.063214] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [11408.070257] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11408.123929] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11408.130799] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11408.138407] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11408.148819] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11408.163509] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11408.172735] RAX: 0000000000000e82 RBX: ffff96848811cae0 RCX: 0000000000000001 [11408.178268] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [11408.186262] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [11408.192222] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [11408.195858] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [11408.199480] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [11408.209746] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11408.214305] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [11408.225076] Call Trace: [11408.227469] [11408.234944] ? show_regs.cold.9+0x22/0x2f [11408.236402] ? watchdog_timer_fn.cold.6+0x62/0x10a [11408.238184] ? __hrtimer_run_queues+0x1c2/0x4d0 [11408.242490] ? watchdog+0x40/0x40 [11408.246983] ? hrtimer_interrupt+0x128/0x2c0 [11408.249460] ? smp_apic_timer_interrupt+0xa6/0x270 [11408.259916] ? apic_timer_interrupt+0xf/0x20 [11408.262418] [11408.263253] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11408.273824] do_raw_spin_lock+0x177/0x190 [11408.278083] _raw_spin_lock+0x21/0x30 [11408.280145] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11408.285298] ? do_raw_spin_unlock+0x75/0x190 [11408.286408] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11408.290037] ? do_wait_intr+0xf0/0xf0 [11408.292963] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11408.300494] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11408.306145] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11408.310302] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11408.315060] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11408.317904] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11408.324879] mdc_intent_lock+0x353/0x7e0 [mdc] [11408.328820] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11408.334193] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11408.339441] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11408.346484] lmv_intent_open+0x338/0x1150 [lmv] [11408.350153] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11408.353275] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11408.359139] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11408.360553] ll_intent_lock+0xc5/0xaa0 [lustre] [11408.362866] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11408.364664] ll_lookup_it+0xe0e/0x2270 [lustre] [11408.369875] ? do_raw_spin_unlock+0x75/0x190 [11408.373408] ? _raw_spin_unlock+0x12/0x30 [11408.375662] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11408.379309] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11408.382055] ll_atomic_open+0x36a/0xd70 [lustre] [11408.390957] lookup_open+0x356/0xb40 [11408.397245] do_last+0x313/0xfc0 [11408.400207] ? nd_jump_root+0xe5/0x160 [11408.403665] ? path_init+0x437/0x520 [11408.405298] path_openat+0xf7/0x500 [11408.407312] ? do_raw_spin_unlock+0x75/0x190 [11408.410393] do_filp_open+0x99/0x140 [11408.415860] ? getname_flags+0x6e/0x330 [11408.421559] ? __check_object_size+0xff/0x256 [11408.426372] ? do_raw_spin_unlock+0x75/0x190 [11408.429868] ? _raw_spin_unlock+0x12/0x30 [11408.433200] do_sys_openat2+0x2b4/0x410 [11408.435545] do_sys_open+0x73/0xa0 [11408.439230] __x64_sys_openat+0x24/0x30 [11408.441985] do_syscall_64+0xc1/0x440 [11408.447312] entry_SYSCALL_64_after_hwframe+0x49/0xae [11408.452287] RIP: 0033:0x7ff54af45332 [11408.455078] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11408.470489] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11408.474891] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [11408.482437] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [11408.485087] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [11408.497844] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [11408.503472] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [11420.055293] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [11420.060559] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11420.118506] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11420.126719] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11420.131615] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11420.137141] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11420.147582] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11420.151428] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [11420.156302] RDX: 00000000000002f7 RSI: 0000000000000003 RDI: ffff96848811cae0 [11420.162596] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [11420.168985] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [11420.177232] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11420.180276] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [11420.190120] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11420.193222] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [11420.197442] Call Trace: [11420.199011] [11420.200123] ? show_regs.cold.9+0x22/0x2f [11420.202689] ? watchdog_timer_fn.cold.6+0x62/0x10a [11420.206569] ? __hrtimer_run_queues+0x1c2/0x4d0 [11420.208927] ? watchdog+0x40/0x40 [11420.210986] ? hrtimer_interrupt+0x128/0x2c0 [11420.214043] ? smp_apic_timer_interrupt+0xa6/0x270 [11420.216982] ? apic_timer_interrupt+0xf/0x20 [11420.219160] [11420.220232] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11420.223658] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11420.226710] do_raw_spin_lock+0x177/0x190 [11420.229882] _raw_spin_lock+0x21/0x30 [11420.231334] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11420.238103] obd_mod_free+0x46/0x140 [ptlrpc] [11420.241769] mdc_commit_open+0x91/0xb0 [mdc] [11420.247634] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11420.252262] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11420.256762] after_reply+0xe2c/0x1780 [ptlrpc] [11420.260131] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11420.264159] ptlrpcd+0xa82/0xe40 [ptlrpc] [11420.267128] ? do_wait_intr+0xf0/0xf0 [11420.271039] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11420.274922] kthread+0x1d1/0x200 [11420.276237] ? set_kthread_struct+0x70/0x70 [11420.279852] ret_from_fork+0x1f/0x30 [11423.199068] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9464s! [11423.207695] Showing busy workqueues and worker pools: [11423.210883] workqueue events: flags=0x0 [11423.212149] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11423.225589] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [11423.232795] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [11423.237124] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [11423.246933] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11423.250735] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [11423.255092] workqueue events_long: flags=0x0 [11423.258522] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11423.264093] pending: lu_objects_flush [obdclass] [11423.269381] workqueue events_unbound: flags=0x2 [11423.271023] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [11423.273132] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11423.276765] workqueue events_power_efficient: flags=0x80 [11423.278429] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11423.280132] pending: neigh_periodic_work [11423.281364] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11423.283430] pending: do_cache_clean, check_lifetime [11423.284917] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11423.286773] pending: neigh_periodic_work [11423.287915] workqueue writeback: flags=0x4a [11423.289013] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11423.290709] pending: wb_workfn, wb_workfn [11423.291786] workqueue xprtiod: flags=0xa [11423.292723] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11423.294200] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [11423.296695] workqueue ipv6_addrconf: flags=0x40008 [11423.297808] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11423.299395] pending: addrconf_verify_work [11423.300536] workqueue knbd0-recv: flags=0x1a [11423.301561] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11423.303035] in-flight: 657:recv_work [11423.304022] workqueue ptlrpc_pinger: flags=0x2 [11423.305091] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11423.306610] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [11423.308140] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9464s workers=4 idle: 20831 12642 28859 [11423.310832] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11413s workers=2 idle: 659 [11428.059024] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [11428.062034] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11428.108251] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11428.120917] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11428.126164] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11428.133676] RIP: 0010:native_safe_halt+0x2e/0x50 [11428.136984] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11428.150305] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11428.156648] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [11428.159820] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c2134154 [11428.172938] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [11428.188049] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [11428.196688] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [11428.202886] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [11428.211070] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11428.216745] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [11428.225795] Call Trace: [11428.226981] [11428.228015] ? show_regs.cold.9+0x22/0x2f [11428.230292] ? watchdog_timer_fn.cold.6+0x62/0x10a [11428.237241] ? __hrtimer_run_queues+0x1c2/0x4d0 [11428.238909] ? watchdog+0x40/0x40 [11428.245950] ? hrtimer_interrupt+0x128/0x2c0 [11428.250835] ? smp_apic_timer_interrupt+0xa6/0x270 [11428.257307] ? apic_timer_interrupt+0xf/0x20 [11428.259906] [11428.260984] ? native_safe_halt+0x2e/0x50 [11428.262829] kvm_wait+0xcd/0xe0 [11428.263862] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11428.266220] do_raw_spin_lock+0x177/0x190 [11428.267980] _raw_spin_lock+0x21/0x30 [11428.269632] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11428.272185] process_one_work+0x2c8/0x700 [11428.275174] worker_thread+0x296/0x6e0 [11428.276597] ? rescuer_thread+0x570/0x570 [11428.278561] kthread+0x1d1/0x200 [11428.279959] ? set_kthread_struct+0x70/0x70 [11428.281749] ret_from_fork+0x1f/0x30 [11432.062302] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [11432.069239] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11432.112787] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11432.126090] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11432.138222] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11432.147010] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11432.176327] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11432.186250] RAX: 0000000000005112 RBX: ffff96848811cae0 RCX: 0000000000000001 [11432.197215] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [11432.207487] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [11432.228134] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [11432.237207] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [11432.243150] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [11432.254800] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11432.260170] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [11432.270260] Call Trace: [11432.272737] [11432.274235] ? show_regs.cold.9+0x22/0x2f [11432.275181] ? watchdog_timer_fn.cold.6+0x62/0x10a [11432.277739] ? __hrtimer_run_queues+0x1c2/0x4d0 [11432.281080] ? watchdog+0x40/0x40 [11432.283603] ? hrtimer_interrupt+0x128/0x2c0 [11432.284971] ? smp_apic_timer_interrupt+0xa6/0x270 [11432.290148] ? apic_timer_interrupt+0xf/0x20 [11432.298032] [11432.299066] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11432.301618] do_raw_spin_lock+0x177/0x190 [11432.307222] _raw_spin_lock+0x21/0x30 [11432.312021] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11432.320401] ? do_raw_spin_unlock+0x75/0x190 [11432.321598] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11432.328686] ? do_wait_intr+0xf0/0xf0 [11432.330188] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11432.334129] mdc_close+0x53b/0xd70 [mdc] [11432.338217] lmv_close+0x18d/0x490 [lmv] [11432.340642] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [11432.346228] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11432.351421] ll_md_real_close+0xa5/0x2a0 [lustre] [11432.355620] ll_md_close+0x274/0x800 [lustre] [11432.357113] ll_file_release+0x128/0xc50 [lustre] [11432.358299] __fput+0x10d/0x440 [11432.363148] ____fput+0x12/0x20 [11432.368689] task_work_run+0xc6/0x110 [11432.370017] exit_to_usermode_loop+0x1dd/0x1f0 [11432.373969] do_syscall_64+0x3d6/0x440 [11432.375142] entry_SYSCALL_64_after_hwframe+0x49/0xae [11432.377813] RIP: 0033:0x7ff54af45cd8 [11432.379294] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11432.400194] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11432.404729] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [11432.409836] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [11432.418094] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [11432.433326] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [11432.442780] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11436.071844] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [11436.078129] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11436.162202] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11436.175631] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11436.188724] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11436.198037] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11436.214924] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11436.235695] RAX: 0000000000004717 RBX: ffff96848811cae0 RCX: 0000000000000001 [11436.247931] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [11436.251639] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [11436.256447] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [11436.263923] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [11436.267875] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [11436.275405] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11436.281758] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [11436.286379] Call Trace: [11436.287729] [11436.290146] ? show_regs.cold.9+0x22/0x2f [11436.292098] ? watchdog_timer_fn.cold.6+0x62/0x10a [11436.301080] ? __hrtimer_run_queues+0x1c2/0x4d0 [11436.303578] ? watchdog+0x40/0x40 [11436.307877] ? hrtimer_interrupt+0x128/0x2c0 [11436.311782] ? smp_apic_timer_interrupt+0xa6/0x270 [11436.315295] ? apic_timer_interrupt+0xf/0x20 [11436.320723] [11436.321712] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11436.327646] do_raw_spin_lock+0x177/0x190 [11436.329062] _raw_spin_lock+0x21/0x30 [11436.337034] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11436.340536] ? do_raw_spin_unlock+0x75/0x190 [11436.345036] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11436.352648] ? do_wait_intr+0xf0/0xf0 [11436.354763] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11436.360368] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11436.368743] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11436.371765] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11436.374684] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11436.380705] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11436.382865] mdc_intent_lock+0x353/0x7e0 [mdc] [11436.394695] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11436.396572] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11436.406722] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11436.409699] lmv_intent_open+0x338/0x1150 [lmv] [11436.416195] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11436.419858] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11436.428060] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11436.432739] ll_intent_lock+0xc5/0xaa0 [lustre] [11436.437015] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11436.445698] ll_lookup_it+0xe0e/0x2270 [lustre] [11436.454645] ? do_raw_spin_unlock+0x75/0x190 [11436.457010] ? _raw_spin_unlock+0x12/0x30 [11436.461349] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11436.468144] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11436.476240] ll_atomic_open+0x36a/0xd70 [lustre] [11436.480224] lookup_open+0x356/0xb40 [11436.481535] do_last+0x313/0xfc0 [11436.483424] ? nd_jump_root+0xe5/0x160 [11436.489694] ? path_init+0x437/0x520 [11436.495795] path_openat+0xf7/0x500 [11436.500609] ? do_raw_spin_unlock+0x75/0x190 [11436.502752] do_filp_open+0x99/0x140 [11436.509933] ? getname_flags+0x6e/0x330 [11436.516714] ? __check_object_size+0xff/0x256 [11436.523578] ? do_raw_spin_unlock+0x75/0x190 [11436.525726] ? _raw_spin_unlock+0x12/0x30 [11436.532261] do_sys_openat2+0x2b4/0x410 [11436.534823] do_sys_open+0x73/0xa0 [11436.541285] __x64_sys_openat+0x24/0x30 [11436.543478] do_syscall_64+0xc1/0x440 [11436.550453] entry_SYSCALL_64_after_hwframe+0x49/0xae [11436.552059] RIP: 0033:0x7ff54af45332 [11436.561125] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11436.592099] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11436.600947] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [11436.610694] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [11436.618996] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [11436.630788] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [11436.633192] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [11448.056991] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [11448.067931] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11448.105708] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11448.111854] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11448.115635] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11448.118035] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11448.126696] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11448.130722] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [11448.134694] RDX: 0000000000006a71 RSI: 0000000000000003 RDI: ffff96848811cae0 [11448.139097] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [11448.149966] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [11448.156356] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11448.159899] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [11448.169139] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11448.171958] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [11448.175693] Call Trace: [11448.177707] [11448.178773] ? show_regs.cold.9+0x22/0x2f [11448.181179] ? watchdog_timer_fn.cold.6+0x62/0x10a [11448.187334] ? __hrtimer_run_queues+0x1c2/0x4d0 [11448.193487] ? watchdog+0x40/0x40 [11448.197471] ? hrtimer_interrupt+0x128/0x2c0 [11448.199770] ? smp_apic_timer_interrupt+0xa6/0x270 [11448.201729] ? apic_timer_interrupt+0xf/0x20 [11448.204253] [11448.205609] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11448.209626] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11448.212580] do_raw_spin_lock+0x177/0x190 [11448.214773] _raw_spin_lock+0x21/0x30 [11448.216675] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11448.219619] obd_mod_free+0x46/0x140 [ptlrpc] [11448.222482] mdc_commit_open+0x91/0xb0 [mdc] [11448.224655] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11448.227844] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11448.230896] after_reply+0xe2c/0x1780 [ptlrpc] [11448.239073] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11448.242503] ptlrpcd+0xa82/0xe40 [ptlrpc] [11448.245312] ? do_wait_intr+0xf0/0xf0 [11448.246977] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11448.249799] kthread+0x1d1/0x200 [11448.251564] ? set_kthread_struct+0x70/0x70 [11448.253563] ret_from_fork+0x1f/0x30 [11453.407996] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9494s! [11453.428225] Showing busy workqueues and worker pools: [11453.432627] workqueue events: flags=0x0 [11453.438668] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11453.443042] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [11453.448827] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [11453.452710] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [11453.458128] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11453.464799] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [11453.471984] workqueue events_long: flags=0x0 [11453.476494] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11453.486185] pending: lu_objects_flush [obdclass] [11453.489774] workqueue events_unbound: flags=0x2 [11453.490939] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [11453.494745] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11453.502872] workqueue events_power_efficient: flags=0x80 [11453.508645] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11453.513563] pending: neigh_periodic_work [11453.518536] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11453.522712] pending: do_cache_clean, check_lifetime [11453.526121] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11453.531138] pending: neigh_periodic_work [11453.533859] workqueue writeback: flags=0x4a [11453.538331] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11453.541642] pending: wb_workfn, wb_workfn [11453.544446] workqueue xprtiod: flags=0xa [11453.549716] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11453.558915] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [11453.574131] workqueue ipv6_addrconf: flags=0x40008 [11453.579571] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11453.584440] pending: addrconf_verify_work [11453.592014] workqueue knbd0-recv: flags=0x1a [11453.597645] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11453.604452] in-flight: 657:recv_work [11453.605965] workqueue ptlrpc_pinger: flags=0x2 [11453.609707] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11453.624906] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [11453.637951] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9494s workers=4 idle: 20831 12642 28859 [11453.647635] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11443s workers=2 idle: 659 [11456.061145] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [11456.075362] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11456.109621] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11456.124647] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11456.131664] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11456.135115] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11456.140765] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11456.159960] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11456.171746] RAX: 00000000000049b9 RBX: ffff96848811cae0 RCX: 0000000000000001 [11456.174463] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [11456.177730] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [11456.183240] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [11456.186117] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [11456.195070] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [11456.200536] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11456.202529] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [11456.212198] Call Trace: [11456.213562] [11456.214053] ? show_regs.cold.9+0x22/0x2f [11456.215823] ? watchdog_timer_fn.cold.6+0x62/0x10a [11456.217525] ? __hrtimer_run_queues+0x1c2/0x4d0 [11456.218715] ? watchdog+0x40/0x40 [11456.219689] ? hrtimer_interrupt+0x128/0x2c0 [11456.229423] ? smp_apic_timer_interrupt+0xa6/0x270 [11456.231975] ? apic_timer_interrupt+0xf/0x20 [11456.234061] [11456.234666] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11456.236366] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11456.244108] do_raw_spin_lock+0x177/0x190 [11456.245796] _raw_spin_lock+0x21/0x30 [11456.248139] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11456.249793] process_one_work+0x2c8/0x700 [11456.252436] worker_thread+0x296/0x6e0 [11456.254738] ? rescuer_thread+0x570/0x570 [11456.257074] kthread+0x1d1/0x200 [11456.261517] ? set_kthread_struct+0x70/0x70 [11456.266772] ret_from_fork+0x1f/0x30 [11460.060186] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [11460.068374] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11460.127646] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11460.140832] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11460.147587] RIP: 0010:native_safe_halt+0x2e/0x50 [11460.154963] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11460.175701] RSP: 0018:ffffaf9fc1543ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11460.177910] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [11460.186918] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c20b4154 [11460.198737] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [11460.208884] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [11460.218736] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [11460.229927] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [11460.247386] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11460.257697] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [11460.260221] Call Trace: [11460.265945] [11460.267802] ? show_regs.cold.9+0x22/0x2f [11460.272472] ? watchdog_timer_fn.cold.6+0x62/0x10a [11460.280359] ? __hrtimer_run_queues+0x1c2/0x4d0 [11460.284530] ? watchdog+0x40/0x40 [11460.285994] ? hrtimer_interrupt+0x128/0x2c0 [11460.293550] ? smp_apic_timer_interrupt+0xa6/0x270 [11460.306565] ? apic_timer_interrupt+0xf/0x20 [11460.309972] [11460.311870] ? native_safe_halt+0x2e/0x50 [11460.317987] kvm_wait+0xcd/0xe0 [11460.320577] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11460.323994] do_raw_spin_lock+0x177/0x190 [11460.325426] _raw_spin_lock+0x21/0x30 [11460.330827] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11460.335672] ? do_raw_spin_unlock+0x75/0x190 [11460.342821] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11460.347437] ? do_wait_intr+0xf0/0xf0 [11460.348919] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11460.356471] mdc_close+0x53b/0xd70 [mdc] [11460.358952] lmv_close+0x18d/0x490 [lmv] [11460.360777] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [11460.363402] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11460.367471] ll_md_real_close+0xa5/0x2a0 [lustre] [11460.372972] ll_md_close+0x274/0x800 [lustre] [11460.375657] ll_file_release+0x128/0xc50 [lustre] [11460.378416] __fput+0x10d/0x440 [11460.380980] ____fput+0x12/0x20 [11460.384543] task_work_run+0xc6/0x110 [11460.388387] exit_to_usermode_loop+0x1dd/0x1f0 [11460.390612] do_syscall_64+0x3d6/0x440 [11460.392183] entry_SYSCALL_64_after_hwframe+0x49/0xae [11460.394604] RIP: 0033:0x7ff54af45cd8 [11460.395619] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11460.402275] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11460.405630] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [11460.408721] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [11460.411622] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [11460.414703] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [11460.418612] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11464.063728] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [11464.069198] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11464.105039] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11464.116012] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11464.120565] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11464.125923] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11464.150026] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11464.160062] RAX: 0000000000002222 RBX: ffff96848811cae0 RCX: 0000000000000001 [11464.170810] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [11464.177987] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [11464.187374] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [11464.197595] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [11464.211599] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [11464.232548] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11464.242721] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [11464.252892] Call Trace: [11464.260849] [11464.261754] ? show_regs.cold.9+0x22/0x2f [11464.272467] ? watchdog_timer_fn.cold.6+0x62/0x10a [11464.276305] ? __hrtimer_run_queues+0x1c2/0x4d0 [11464.285274] ? watchdog+0x40/0x40 [11464.287323] ? hrtimer_interrupt+0x128/0x2c0 [11464.295123] ? smp_apic_timer_interrupt+0xa6/0x270 [11464.300713] ? apic_timer_interrupt+0xf/0x20 [11464.304972] [11464.306246] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11464.308925] do_raw_spin_lock+0x177/0x190 [11464.311300] _raw_spin_lock+0x21/0x30 [11464.324414] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11464.335691] ? do_raw_spin_unlock+0x75/0x190 [11464.337046] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11464.345131] ? do_wait_intr+0xf0/0xf0 [11464.354367] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11464.363213] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11464.366352] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11464.377646] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11464.379709] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11464.381188] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11464.405518] mdc_intent_lock+0x353/0x7e0 [mdc] [11464.407012] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11464.410009] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11464.418601] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11464.428306] lmv_intent_open+0x338/0x1150 [lmv] [11464.432884] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11464.441950] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11464.451799] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11464.454847] ll_intent_lock+0xc5/0xaa0 [lustre] [11464.459397] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11464.464908] ll_lookup_it+0xe0e/0x2270 [lustre] [11464.466465] ? do_raw_spin_unlock+0x75/0x190 [11464.472425] ? _raw_spin_unlock+0x12/0x30 [11464.476583] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11464.480678] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11464.488006] ll_atomic_open+0x36a/0xd70 [lustre] [11464.496299] lookup_open+0x356/0xb40 [11464.499328] do_last+0x313/0xfc0 [11464.500767] ? nd_jump_root+0xe5/0x160 [11464.502284] ? path_init+0x437/0x520 [11464.506534] path_openat+0xf7/0x500 [11464.507939] ? do_raw_spin_unlock+0x75/0x190 [11464.509415] do_filp_open+0x99/0x140 [11464.518516] ? getname_flags+0x6e/0x330 [11464.521514] ? __check_object_size+0xff/0x256 [11464.525608] ? do_raw_spin_unlock+0x75/0x190 [11464.528703] ? _raw_spin_unlock+0x12/0x30 [11464.533562] do_sys_openat2+0x2b4/0x410 [11464.540496] do_sys_open+0x73/0xa0 [11464.545380] __x64_sys_openat+0x24/0x30 [11464.547844] do_syscall_64+0xc1/0x440 [11464.548972] entry_SYSCALL_64_after_hwframe+0x49/0xae [11464.552627] RIP: 0033:0x7ff54af45332 [11464.557650] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11464.578028] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11464.589895] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [11464.597028] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [11464.609352] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [11464.611877] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [11464.614958] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [11480.057807] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [11480.078122] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11480.153968] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11480.169108] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11480.177156] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11480.182013] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11480.204149] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11480.228832] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [11480.245274] RDX: 00000000000073c0 RSI: 0000000000000003 RDI: ffff96848811cae0 [11480.265343] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [11480.278817] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [11480.299761] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11480.308941] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [11480.324058] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11480.336115] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [11480.354447] Call Trace: [11480.357227] [11480.360145] ? show_regs.cold.9+0x22/0x2f [11480.366554] ? watchdog_timer_fn.cold.6+0x62/0x10a [11480.370593] ? __hrtimer_run_queues+0x1c2/0x4d0 [11480.374177] ? watchdog+0x40/0x40 [11480.378236] ? hrtimer_interrupt+0x128/0x2c0 [11480.395947] ? smp_apic_timer_interrupt+0xa6/0x270 [11480.408530] ? apic_timer_interrupt+0xf/0x20 [11480.410413] [11480.411346] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11480.419501] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11480.423370] do_raw_spin_lock+0x177/0x190 [11480.435724] _raw_spin_lock+0x21/0x30 [11480.439410] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11480.451479] obd_mod_free+0x46/0x140 [ptlrpc] [11480.458518] mdc_commit_open+0x91/0xb0 [mdc] [11480.463153] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11480.466847] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11480.471642] after_reply+0xe2c/0x1780 [ptlrpc] [11480.475175] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11480.482401] ptlrpcd+0xa82/0xe40 [ptlrpc] [11480.487295] ? do_wait_intr+0xf0/0xf0 [11480.491126] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11480.512801] kthread+0x1d1/0x200 [11480.514247] ? set_kthread_struct+0x70/0x70 [11480.530103] ret_from_fork+0x1f/0x30 [11484.065909] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [11484.080706] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11484.128562] BUG: workqueue lockup - pool [11484.163685] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11484.163701] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11484.163708] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11484.164071] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11484.164082] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11484.164086] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11484.164090] RAX: 0000000000001688 RBX: ffff96848811cae0 RCX: 0000000000000001 [11484.164093] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [11484.164095] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [11484.164097] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [11484.164100] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [11484.164103] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [11484.164107] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11484.164109] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [11484.164117] Call Trace: [11484.164122] [11484.164128] ? show_regs.cold.9+0x22/0x2f [11484.164136] ? watchdog_timer_fn.cold.6+0x62/0x10a [11484.164143] ? __hrtimer_run_queues+0x1c2/0x4d0 [11484.164147] ? watchdog+0x40/0x40 [11484.164152] ? hrtimer_interrupt+0x128/0x2c0 [11484.164157] ? smp_apic_timer_interrupt+0xa6/0x270 [11484.164166] ? apic_timer_interrupt+0xf/0x20 [11484.164171] [11484.164174] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11484.164179] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11484.164184] do_raw_spin_lock+0x177/0x190 [11484.164189] _raw_spin_lock+0x21/0x30 [11484.164193] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11484.165168] process_one_work+0x2c8/0x700 [11484.165175] worker_thread+0x296/0x6e0 [11484.165180] ? rescuer_thread+0x570/0x570 [11484.165183] kthread+0x1d1/0x200 [11484.165187] ? set_kthread_struct+0x70/0x70 [11484.165191] ret_from_fork+0x1f/0x30 [11484.453231] cpus=0-3 flags=0x4 nice=0 stuck for 9525s! [11484.472250] Showing busy workqueues and worker pools: [11484.482605] workqueue events: flags=0x0 [11484.497251] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11484.519649] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [11484.527302] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [11484.534226] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [11484.546840] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11484.557153] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [11484.567449] workqueue events_long: flags=0x0 [11484.569065] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11484.575905] pending: lu_objects_flush [obdclass] [11484.578491] workqueue events_unbound: flags=0x2 [11484.583280] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [11484.590323] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11484.604724] workqueue events_power_efficient: flags=0x80 [11484.607695] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11484.615187] pending: neigh_periodic_work [11484.616796] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11484.621699] pending: do_cache_clean, check_lifetime [11484.633616] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11484.637139] pending: neigh_periodic_work [11484.645352] workqueue writeback: flags=0x4a [11484.648076] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11484.655814] pending: wb_workfn, wb_workfn [11484.659986] workqueue xprtiod: flags=0xa [11484.669379] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11484.681372] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [11484.694588] workqueue ipv6_addrconf: flags=0x40008 [11484.703787] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11484.716054] pending: addrconf_verify_work [11484.725591] workqueue knbd0-recv: flags=0x1a [11484.728598] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11484.731772] in-flight: 657:recv_work [11484.733456] workqueue ptlrpc_pinger: flags=0x2 [11484.736031] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11484.742660] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [11484.754388] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9525s workers=4 idle: 20831 12642 28859 [11484.767196] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11474s workers=2 idle: 659 [11488.062200] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [11488.077887] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11488.117530] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11488.123664] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11488.132087] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11488.142121] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11488.171076] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11488.179743] RAX: 0000000000002788 RBX: ffff96848811cae0 RCX: 0000000000000001 [11488.192113] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [11488.203818] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [11488.209598] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [11488.214512] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [11488.219452] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [11488.233431] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11488.242874] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [11488.248036] Call Trace: [11488.249849] [11488.254999] ? show_regs.cold.9+0x22/0x2f [11488.259639] ? watchdog_timer_fn.cold.6+0x62/0x10a [11488.262390] ? __hrtimer_run_queues+0x1c2/0x4d0 [11488.265645] ? watchdog+0x40/0x40 [11488.266883] ? hrtimer_interrupt+0x128/0x2c0 [11488.270479] ? smp_apic_timer_interrupt+0xa6/0x270 [11488.273530] ? apic_timer_interrupt+0xf/0x20 [11488.279932] [11488.281075] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11488.287455] do_raw_spin_lock+0x177/0x190 [11488.289958] _raw_spin_lock+0x21/0x30 [11488.294640] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11488.298629] ? do_raw_spin_unlock+0x75/0x190 [11488.302736] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11488.307698] ? do_wait_intr+0xf0/0xf0 [11488.310058] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11488.313926] mdc_close+0x53b/0xd70 [mdc] [11488.317111] lmv_close+0x18d/0x490 [lmv] [11488.324892] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [11488.330475] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11488.334125] ll_md_real_close+0xa5/0x2a0 [lustre] [11488.336875] ll_md_close+0x274/0x800 [lustre] [11488.339503] ll_file_release+0x128/0xc50 [lustre] [11488.341972] __fput+0x10d/0x440 [11488.343719] ____fput+0x12/0x20 [11488.345733] task_work_run+0xc6/0x110 [11488.347443] exit_to_usermode_loop+0x1dd/0x1f0 [11488.350822] do_syscall_64+0x3d6/0x440 [11488.353239] entry_SYSCALL_64_after_hwframe+0x49/0xae [11488.356785] RIP: 0033:0x7ff54af45cd8 [11488.358078] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11488.379805] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11488.382740] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [11488.390373] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [11488.396935] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [11488.406920] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [11488.423335] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11492.066876] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [11492.075370] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11492.137497] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11492.142573] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11492.146802] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11492.149631] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11492.159738] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11492.166079] RAX: 0000000000007b56 RBX: ffff96848811cae0 RCX: 0000000000000001 [11492.170285] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [11492.179646] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [11492.185472] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [11492.200539] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [11492.215040] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [11492.227117] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11492.238456] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [11492.250331] Call Trace: [11492.255654] [11492.260674] ? show_regs.cold.9+0x22/0x2f [11492.266117] ? watchdog_timer_fn.cold.6+0x62/0x10a [11492.271177] ? __hrtimer_run_queues+0x1c2/0x4d0 [11492.272720] ? watchdog+0x40/0x40 [11492.273901] ? hrtimer_interrupt+0x128/0x2c0 [11492.284368] ? smp_apic_timer_interrupt+0xa6/0x270 [11492.288621] ? apic_timer_interrupt+0xf/0x20 [11492.291715] [11492.293316] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11492.296749] do_raw_spin_lock+0x177/0x190 [11492.299962] _raw_spin_lock+0x21/0x30 [11492.301359] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11492.303884] ? do_raw_spin_unlock+0x75/0x190 [11492.310224] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11492.317438] ? do_wait_intr+0xf0/0xf0 [11492.320884] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11492.328904] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11492.330775] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11492.335412] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11492.348534] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11492.355210] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11492.360397] mdc_intent_lock+0x353/0x7e0 [mdc] [11492.363829] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11492.366330] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11492.374636] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11492.380542] lmv_intent_open+0x338/0x1150 [lmv] [11492.387403] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11492.393603] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11492.398926] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11492.404495] ll_intent_lock+0xc5/0xaa0 [lustre] [11492.408603] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11492.410491] ll_lookup_it+0xe0e/0x2270 [lustre] [11492.417445] ? do_raw_spin_unlock+0x75/0x190 [11492.419240] ? _raw_spin_unlock+0x12/0x30 [11492.425348] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11492.434504] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11492.437853] ll_atomic_open+0x36a/0xd70 [lustre] [11492.442281] lookup_open+0x356/0xb40 [11492.444132] do_last+0x313/0xfc0 [11492.449110] ? nd_jump_root+0xe5/0x160 [11492.450427] ? path_init+0x437/0x520 [11492.452143] path_openat+0xf7/0x500 [11492.453472] ? do_raw_spin_unlock+0x75/0x190 [11492.455671] do_filp_open+0x99/0x140 [11492.460693] ? getname_flags+0x6e/0x330 [11492.464228] ? __check_object_size+0xff/0x256 [11492.468676] ? do_raw_spin_unlock+0x75/0x190 [11492.476277] ? _raw_spin_unlock+0x12/0x30 [11492.481657] do_sys_openat2+0x2b4/0x410 [11492.486338] do_sys_open+0x73/0xa0 [11492.487601] __x64_sys_openat+0x24/0x30 [11492.492553] do_syscall_64+0xc1/0x440 [11492.497759] entry_SYSCALL_64_after_hwframe+0x49/0xae [11492.501687] RIP: 0033:0x7ff54af45332 [11492.506903] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11492.512415] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11492.514401] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [11492.516247] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [11492.517878] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [11492.519516] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [11492.520798] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [11512.055950] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [11512.059060] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [11512.062606] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) [11512.067408] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) [11512.079955] ptlrpc(O) [11512.080018] obdclass(O) [11512.091592] obdclass(O) ksocklnd(O) lnet(O) [11512.094259] ksocklnd(O) [11512.096721] libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11512.096843] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11512.096851] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11512.096854] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11512.096868] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11512.096872] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11512.096876] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [11512.096879] RDX: 0000000000001ab9 RSI: 0000000000000003 RDI: ffff96848811cae0 [11512.096881] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [11512.096884] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [11512.096886] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11512.096889] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [11512.096892] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11512.096894] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [11512.096902] Call Trace: [11512.096907] [11512.096912] ? show_regs.cold.9+0x22/0x2f [11512.096921] ? watchdog_timer_fn.cold.6+0x62/0x10a [11512.096927] ? __hrtimer_run_queues+0x1c2/0x4d0 [11512.096932] ? watchdog+0x40/0x40 [11512.096936] ? hrtimer_interrupt+0x128/0x2c0 [11512.096940] ? smp_apic_timer_interrupt+0xa6/0x270 [11512.096949] ? apic_timer_interrupt+0xf/0x20 [11512.096953] [11512.096956] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11512.096971] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11512.096976] do_raw_spin_lock+0x177/0x190 [11512.096981] _raw_spin_lock+0x21/0x30 [11512.096987] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11512.097275] obd_mod_free+0x46/0x140 [ptlrpc] [11512.097509] mdc_commit_open+0x91/0xb0 [mdc] [11512.097550] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11512.098184] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11512.098388] after_reply+0xe2c/0x1780 [ptlrpc] [11512.098891] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11512.099242] ptlrpcd+0xa82/0xe40 [ptlrpc] [11512.099675] ? do_wait_intr+0xf0/0xf0 [11512.099687] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11512.100066] kthread+0x1d1/0x200 [11512.100073] ? set_kthread_struct+0x70/0x70 [11512.100077] ret_from_fork+0x1f/0x30 [11512.240576] lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11512.256526] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11512.268727] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11512.274413] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11512.277330] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11512.279816] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11512.300282] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11512.308348] RAX: 0000000000000294 RBX: ffff96848811cae0 RCX: 0000000000000001 [11512.311247] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [11512.315029] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [11512.318619] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [11512.324273] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [11512.331536] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [11512.338720] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11512.341446] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [11512.348324] Call Trace: [11512.351958] [11512.352565] ? show_regs.cold.9+0x22/0x2f [11512.361941] ? watchdog_timer_fn.cold.6+0x62/0x10a [11512.364663] ? __hrtimer_run_queues+0x1c2/0x4d0 [11512.371773] ? watchdog+0x40/0x40 [11512.373309] ? hrtimer_interrupt+0x128/0x2c0 [11512.383247] ? smp_apic_timer_interrupt+0xa6/0x270 [11512.386972] ? apic_timer_interrupt+0xf/0x20 [11512.388391] [11512.399434] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11512.402002] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11512.403953] do_raw_spin_lock+0x177/0x190 [11512.405194] _raw_spin_lock+0x21/0x30 [11512.406272] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11512.408344] process_one_work+0x2c8/0x700 [11512.409693] worker_thread+0x296/0x6e0 [11512.411587] ? rescuer_thread+0x570/0x570 [11512.413310] kthread+0x1d1/0x200 [11512.414686] ? set_kthread_struct+0x70/0x70 [11512.416277] ret_from_fork+0x1f/0x30 [11514.848975] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9555s! [11514.855224] Showing busy workqueues and worker pools: [11514.865437] workqueue events: flags=0x0 [11514.866805] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11514.880940] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [11514.892236] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [11514.904404] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [11514.928839] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11514.933431] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [11514.939940] workqueue events_long: flags=0x0 [11514.943246] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11514.960877] pending: lu_objects_flush [obdclass] [11514.962798] workqueue events_unbound: flags=0x2 [11514.967118] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [11514.979625] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11514.996144] workqueue events_power_efficient: flags=0x80 [11515.005124] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11515.013699] pending: neigh_periodic_work [11515.017230] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11515.026893] pending: do_cache_clean, check_lifetime [11515.032179] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11515.039968] pending: neigh_periodic_work [11515.046183] workqueue writeback: flags=0x4a [11515.047947] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11515.057133] pending: wb_workfn, wb_workfn [11515.061421] workqueue xprtiod: flags=0xa [11515.064838] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11515.075962] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [11515.085485] workqueue ipv6_addrconf: flags=0x40008 [11515.091770] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11515.100409] pending: addrconf_verify_work [11515.103439] workqueue knbd0-recv: flags=0x1a [11515.108375] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11515.117197] in-flight: 657:recv_work [11515.119413] workqueue ptlrpc_pinger: flags=0x2 [11515.122793] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11515.129369] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [11515.139984] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9556s workers=4 idle: 20831 12642 28859 [11515.150141] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11505s workers=2 idle: 659 [11516.064820] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [11516.070857] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11516.095587] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11516.099977] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11516.104559] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11516.108365] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11516.122761] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11516.125550] RAX: 0000000000001393 RBX: ffff96848811cae0 RCX: 0000000000000001 [11516.128080] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [11516.130655] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [11516.133892] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [11516.144418] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [11516.155187] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [11516.165965] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11516.171895] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [11516.175444] Call Trace: [11516.175960] [11516.176331] ? show_regs.cold.9+0x22/0x2f [11516.184309] ? watchdog_timer_fn.cold.6+0x62/0x10a [11516.186262] ? __hrtimer_run_queues+0x1c2/0x4d0 [11516.188299] ? watchdog+0x40/0x40 [11516.196622] ? hrtimer_interrupt+0x128/0x2c0 [11516.199819] ? smp_apic_timer_interrupt+0xa6/0x270 [11516.207598] ? apic_timer_interrupt+0xf/0x20 [11516.208547] [11516.209071] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11516.216034] do_raw_spin_lock+0x177/0x190 [11516.217385] _raw_spin_lock+0x21/0x30 [11516.221352] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11516.224578] ? do_raw_spin_unlock+0x75/0x190 [11516.237772] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11516.240886] ? do_wait_intr+0xf0/0xf0 [11516.244267] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11516.246084] mdc_close+0x53b/0xd70 [mdc] [11516.257243] lmv_close+0x18d/0x490 [lmv] [11516.261477] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [11516.270647] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11516.273536] ll_md_real_close+0xa5/0x2a0 [lustre] [11516.274916] ll_md_close+0x274/0x800 [lustre] [11516.293245] ll_file_release+0x128/0xc50 [lustre] [11516.306550] __fput+0x10d/0x440 [11516.310862] ____fput+0x12/0x20 [11516.322474] task_work_run+0xc6/0x110 [11516.323610] exit_to_usermode_loop+0x1dd/0x1f0 [11516.324909] do_syscall_64+0x3d6/0x440 [11516.344801] entry_SYSCALL_64_after_hwframe+0x49/0xae [11516.352352] RIP: 0033:0x7ff54af45cd8 [11516.360620] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11516.396555] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11516.423019] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [11516.425453] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [11516.440176] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [11516.448318] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [11516.457396] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11520.060026] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [11520.065127] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11520.092481] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11520.100032] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11520.104493] RIP: 0010:native_safe_halt+0x2e/0x50 [11520.106796] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11520.119546] RSP: 0000:ffffaf9fc15234f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11520.124741] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [11520.128451] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c21b4154 [11520.132770] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [11520.137049] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [11520.141167] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [11520.145140] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [11520.149358] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11520.152070] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [11520.155440] Call Trace: [11520.156647] [11520.157627] ? show_regs.cold.9+0x22/0x2f [11520.159883] ? watchdog_timer_fn.cold.6+0x62/0x10a [11520.161725] ? __hrtimer_run_queues+0x1c2/0x4d0 [11520.164365] ? watchdog+0x40/0x40 [11520.165999] ? hrtimer_interrupt+0x128/0x2c0 [11520.167959] ? smp_apic_timer_interrupt+0xa6/0x270 [11520.170142] ? apic_timer_interrupt+0xf/0x20 [11520.172425] [11520.173458] ? native_safe_halt+0x2e/0x50 [11520.175456] kvm_wait+0xcd/0xe0 [11520.176967] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11520.179280] do_raw_spin_lock+0x177/0x190 [11520.181383] _raw_spin_lock+0x21/0x30 [11520.183035] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11520.186006] ? do_raw_spin_unlock+0x75/0x190 [11520.188802] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11520.192269] ? do_wait_intr+0xf0/0xf0 [11520.194484] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11520.198345] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11520.201406] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11520.206171] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11520.210406] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11520.215110] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11520.219057] mdc_intent_lock+0x353/0x7e0 [mdc] [11520.222112] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11520.226018] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11520.232979] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11520.236731] lmv_intent_open+0x338/0x1150 [lmv] [11520.239182] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11520.242115] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11520.245721] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11520.249080] ll_intent_lock+0xc5/0xaa0 [lustre] [11520.254156] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11520.257444] ll_lookup_it+0xe0e/0x2270 [lustre] [11520.264969] ? do_raw_spin_unlock+0x75/0x190 [11520.271841] ? _raw_spin_unlock+0x12/0x30 [11520.282080] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11520.292975] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11520.299272] ll_atomic_open+0x36a/0xd70 [lustre] [11520.302270] lookup_open+0x356/0xb40 [11520.304578] do_last+0x313/0xfc0 [11520.306427] ? nd_jump_root+0xe5/0x160 [11520.310273] ? path_init+0x437/0x520 [11520.312419] path_openat+0xf7/0x500 [11520.316030] ? do_raw_spin_unlock+0x75/0x190 [11520.318783] do_filp_open+0x99/0x140 [11520.322077] ? getname_flags+0x6e/0x330 [11520.325022] ? __check_object_size+0xff/0x256 [11520.327804] ? do_raw_spin_unlock+0x75/0x190 [11520.331084] ? _raw_spin_unlock+0x12/0x30 [11520.337939] do_sys_openat2+0x2b4/0x410 [11520.344788] do_sys_open+0x73/0xa0 [11520.355770] __x64_sys_openat+0x24/0x30 [11520.359382] do_syscall_64+0xc1/0x440 [11520.362106] entry_SYSCALL_64_after_hwframe+0x49/0xae [11520.365177] RIP: 0033:0x7ff54af45332 [11520.367828] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11520.378570] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11520.383060] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [11520.387095] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [11520.391436] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [11520.395701] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [11520.399805] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [11540.061120] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [11540.064756] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [11540.065227] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11540.065301] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11540.065308] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11540.065314] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11540.066010] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11540.066022] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11540.066026] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11540.066031] RAX: 0000000000006c87 RBX: ffff96848811cae0 RCX: 0000000000000001 [11540.066034] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [11540.066036] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [11540.066038] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [11540.066041] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [11540.066044] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [11540.066047] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11540.066049] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [11540.066058] Call Trace: [11540.066064] [11540.066070] ? show_regs.cold.9+0x22/0x2f [11540.066079] ? watchdog_timer_fn.cold.6+0x62/0x10a [11540.066085] ? __hrtimer_run_queues+0x1c2/0x4d0 [11540.066088] ? watchdog+0x40/0x40 [11540.066093] ? hrtimer_interrupt+0x128/0x2c0 [11540.066096] ? smp_apic_timer_interrupt+0xa6/0x270 [11540.066104] ? apic_timer_interrupt+0xf/0x20 [11540.066108] [11540.066111] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11540.066115] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11540.066120] do_raw_spin_lock+0x177/0x190 [11540.066124] _raw_spin_lock+0x21/0x30 [11540.066129] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11540.066847] process_one_work+0x2c8/0x700 [11540.066856] worker_thread+0x296/0x6e0 [11540.066860] ? rescuer_thread+0x570/0x570 [11540.066863] kthread+0x1d1/0x200 [11540.066868] ? set_kthread_struct+0x70/0x70 [11540.066871] ret_from_fork+0x1f/0x30 [11540.462464] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11540.521323] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11540.538471] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11540.550883] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11540.558292] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11540.583116] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11540.592744] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [11540.599045] RDX: 000000000000743e RSI: 0000000000000003 RDI: ffff96848811cae0 [11540.616069] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [11540.623825] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [11540.631388] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11540.648917] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [11540.657154] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11540.665292] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [11540.673279] Call Trace: [11540.676867] [11540.682797] ? show_regs.cold.9+0x22/0x2f [11540.688962] ? watchdog_timer_fn.cold.6+0x62/0x10a [11540.694256] ? __hrtimer_run_queues+0x1c2/0x4d0 [11540.705240] ? watchdog+0x40/0x40 [11540.707268] ? hrtimer_interrupt+0x128/0x2c0 [11540.711263] ? smp_apic_timer_interrupt+0xa6/0x270 [11540.717995] ? apic_timer_interrupt+0xf/0x20 [11540.720501] [11540.721553] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11540.736884] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11540.743996] do_raw_spin_lock+0x177/0x190 [11540.747297] _raw_spin_lock+0x21/0x30 [11540.750013] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11540.755337] obd_mod_free+0x46/0x140 [ptlrpc] [11540.763247] mdc_commit_open+0x91/0xb0 [mdc] [11540.768949] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11540.775020] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11540.786006] after_reply+0xe2c/0x1780 [ptlrpc] [11540.796439] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11540.803187] ptlrpcd+0xa82/0xe40 [ptlrpc] [11540.809965] ? do_wait_intr+0xf0/0xf0 [11540.811422] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11540.835333] kthread+0x1d1/0x200 [11540.839341] ? set_kthread_struct+0x70/0x70 [11540.841054] ret_from_fork+0x1f/0x30 [11544.059851] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [11544.076049] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11544.128674] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11544.142691] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11544.152195] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11544.160414] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11544.180818] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11544.184958] RAX: 0000000000006885 RBX: ffff96848811cae0 RCX: 0000000000000001 [11544.191636] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [11544.200053] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [11544.205053] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [11544.212852] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [11544.220614] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [11544.227778] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11544.234658] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [11544.238225] Call Trace: [11544.244003] [11544.244973] ? show_regs.cold.9+0x22/0x2f [11544.251950] ? watchdog_timer_fn.cold.6+0x62/0x10a [11544.253809] ? __hrtimer_run_queues+0x1c2/0x4d0 [11544.257651] ? watchdog+0x40/0x40 [11544.262129] ? hrtimer_interrupt+0x128/0x2c0 [11544.266328] ? smp_apic_timer_interrupt+0xa6/0x270 [11544.272082] ? apic_timer_interrupt+0xf/0x20 [11544.273922] [11544.274812] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11544.280974] do_raw_spin_lock+0x177/0x190 [11544.289122] _raw_spin_lock+0x21/0x30 [11544.294421] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11544.300638] ? do_raw_spin_unlock+0x75/0x190 [11544.303969] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11544.309981] ? do_wait_intr+0xf0/0xf0 [11544.315037] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11544.317755] mdc_close+0x53b/0xd70 [mdc] [11544.323097] lmv_close+0x18d/0x490 [lmv] [11544.325011] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [11544.330640] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11544.337732] ll_md_real_close+0xa5/0x2a0 [lustre] [11544.342927] ll_md_close+0x274/0x800 [lustre] [11544.345894] ll_file_release+0x128/0xc50 [lustre] [11544.348646] __fput+0x10d/0x440 [11544.354813] ____fput+0x12/0x20 [11544.356367] task_work_run+0xc6/0x110 [11544.359966] exit_to_usermode_loop+0x1dd/0x1f0 [11544.362096] do_syscall_64+0x3d6/0x440 [11544.364873] entry_SYSCALL_64_after_hwframe+0x49/0xae [11544.368896] RIP: 0033:0x7ff54af45cd8 [11544.370978] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11544.382121] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11544.388874] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [11544.396276] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [11544.402732] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [11544.409777] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [11544.413250] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11545.568840] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9586s! [11545.573761] Showing busy workqueues and worker pools: [11545.577346] workqueue events: flags=0x0 [11545.581797] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11545.588296] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [11545.598012] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [11545.601109] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [11545.614261] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11545.621795] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [11545.626765] workqueue events_long: flags=0x0 [11545.628493] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11545.635843] pending: lu_objects_flush [obdclass] [11545.639370] workqueue events_unbound: flags=0x2 [11545.645401] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [11545.654795] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11545.665129] workqueue events_power_efficient: flags=0x80 [11545.673745] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11545.679072] pending: neigh_periodic_work [11545.684624] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11545.689926] pending: do_cache_clean, check_lifetime [11545.695247] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11545.701867] pending: neigh_periodic_work [11545.705255] workqueue writeback: flags=0x4a [11545.706812] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11545.713579] pending: wb_workfn, wb_workfn [11545.717647] workqueue xprtiod: flags=0xa [11545.721768] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11545.726690] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [11545.733030] workqueue ipv6_addrconf: flags=0x40008 [11545.740712] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11545.747646] pending: addrconf_verify_work [11545.750867] workqueue knbd0-recv: flags=0x1a [11545.756769] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11545.760253] in-flight: 657:recv_work [11545.762163] workqueue ptlrpc_pinger: flags=0x2 [11545.767252] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11545.771109] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [11545.785259] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9586s workers=4 idle: 20831 12642 28859 [11545.793823] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11536s workers=2 idle: 659 [11548.060016] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [11548.066785] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11548.111826] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11548.126166] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11548.138148] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11548.140192] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11548.154525] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11548.158697] RAX: 0000000000002b22 RBX: ffff96848811cae0 RCX: 0000000000000001 [11548.162149] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [11548.169139] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [11548.173960] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [11548.180777] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [11548.184998] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [11548.192608] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11548.199741] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [11548.203972] Call Trace: [11548.205431] [11548.209808] ? show_regs.cold.9+0x22/0x2f [11548.212642] ? watchdog_timer_fn.cold.6+0x62/0x10a [11548.220649] ? __hrtimer_run_queues+0x1c2/0x4d0 [11548.223266] ? watchdog+0x40/0x40 [11548.225804] ? hrtimer_interrupt+0x128/0x2c0 [11548.230297] ? smp_apic_timer_interrupt+0xa6/0x270 [11548.235190] ? apic_timer_interrupt+0xf/0x20 [11548.237968] [11548.239760] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11548.247781] do_raw_spin_lock+0x177/0x190 [11548.256796] _raw_spin_lock+0x21/0x30 [11548.258980] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11548.263564] ? do_raw_spin_unlock+0x75/0x190 [11548.266632] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11548.269546] ? do_wait_intr+0xf0/0xf0 [11548.274009] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11548.284759] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11548.287177] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11548.290884] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11548.293459] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11548.295921] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11548.298325] mdc_intent_lock+0x353/0x7e0 [mdc] [11548.299805] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11548.302265] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11548.307957] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11548.311128] lmv_intent_open+0x338/0x1150 [lmv] [11548.313809] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11548.319580] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11548.323817] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11548.329913] ll_intent_lock+0xc5/0xaa0 [lustre] [11548.332795] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11548.335976] ll_lookup_it+0xe0e/0x2270 [lustre] [11548.338601] ? do_raw_spin_unlock+0x75/0x190 [11548.340164] ? _raw_spin_unlock+0x12/0x30 [11548.344596] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11548.348626] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11548.352690] ll_atomic_open+0x36a/0xd70 [lustre] [11548.360619] lookup_open+0x356/0xb40 [11548.361925] do_last+0x313/0xfc0 [11548.366478] ? nd_jump_root+0xe5/0x160 [11548.370944] ? path_init+0x437/0x520 [11548.376254] path_openat+0xf7/0x500 [11548.381146] ? do_raw_spin_unlock+0x75/0x190 [11548.385531] do_filp_open+0x99/0x140 [11548.386870] ? getname_flags+0x6e/0x330 [11548.396061] ? __check_object_size+0xff/0x256 [11548.398981] ? do_raw_spin_unlock+0x75/0x190 [11548.405179] ? _raw_spin_unlock+0x12/0x30 [11548.411952] do_sys_openat2+0x2b4/0x410 [11548.414864] do_sys_open+0x73/0xa0 [11548.416958] __x64_sys_openat+0x24/0x30 [11548.419060] do_syscall_64+0xc1/0x440 [11548.421136] entry_SYSCALL_64_after_hwframe+0x49/0xae [11548.424862] RIP: 0033:0x7ff54af45332 [11548.429072] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11548.443737] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11548.448483] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [11548.453823] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [11548.462214] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [11548.466979] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [11548.472034] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [11554.114831] rcu: INFO: rcu_sched self-detected stall on CPU [11554.122188] rcu: 0-...!: (5070614 ticks this GP) idle=98a/1/0x4000000000000002 softirq=125154/125154 fqs=959 [11554.140978] (t=9600274 jiffies g=253585 q=5974) [11554.152474] rcu: rcu_sched kthread starved for 9595234 jiffies! g253585 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [11554.165051] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [11554.172139] rcu: RCU grace-period kthread stack dump: [11554.175028] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [11554.194037] Call Trace: [11554.195375] __schedule+0x351/0xcb0 [11554.203168] schedule+0xc0/0x180 [11554.214581] schedule_timeout+0xb4/0x190 [11554.228600] ? do_raw_spin_unlock+0x75/0x190 [11554.238623] ? __next_timer_interrupt+0x160/0x160 [11554.250256] ? prepare_to_swait_event+0xf3/0x140 [11554.256198] rcu_gp_kthread+0x8dd/0x15b0 [11554.268126] ? force_qs_rnp+0x2c0/0x2c0 [11554.277944] kthread+0x1d1/0x200 [11554.283150] ? set_kthread_struct+0x70/0x70 [11554.291659] ret_from_fork+0x1f/0x30 [11554.297631] rcu: Stack dump where RCU GP kthread last ran: [11554.308081] Sending NMI from CPU 0 to CPUs 1: [11554.311142] NMI backtrace for cpu 1 [11554.311148] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11554.311150] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11554.311152] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11554.311154] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11554.311156] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 [11554.311158] RAX: 000000000000237a RBX: ffff96848811cae0 RCX: 0000000000000001 [11554.311159] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [11554.311160] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [11554.311161] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [11554.311162] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [11554.311163] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [11554.311164] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11554.311165] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [11554.311166] Call Trace: [11554.311167] [11554.311168] ? show_regs.cold.9+0x22/0x2f [11554.311169] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [11554.311170] ? nmi_cpu_backtrace_handler+0x15/0x40 [11554.311170] ? nmi_handle+0xbd/0x210 [11554.311171] ? default_do_nmi+0x4b/0x1b0 [11554.311172] ? do_nmi+0x280/0x380 [11554.311173] ? end_repeat_nmi+0x16/0x1f [11554.311174] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11554.311175] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11554.311175] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11554.311176] [11554.311177] do_raw_spin_lock+0x177/0x190 [11554.311178] _raw_spin_lock+0x21/0x30 [11554.311178] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11554.311179] ? do_raw_spin_unlock+0x75/0x190 [11554.311180] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11554.311181] ? do_wait_intr+0xf0/0xf0 [11554.311182] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11554.311182] mdc_close+0x53b/0xd70 [mdc] [11554.311183] lmv_close+0x18d/0x490 [lmv] [11554.311184] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [11554.311185] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11554.311186] ll_md_real_close+0xa5/0x2a0 [lustre] [11554.311187] ll_md_close+0x274/0x800 [lustre] [11554.311187] ll_file_release+0x128/0xc50 [lustre] [11554.311188] __fput+0x10d/0x440 [11554.311189] ____fput+0x12/0x20 [11554.311190] task_work_run+0xc6/0x110 [11554.311191] exit_to_usermode_loop+0x1dd/0x1f0 [11554.311192] do_syscall_64+0x3d6/0x440 [11554.311192] entry_SYSCALL_64_after_hwframe+0x49/0xae [11554.311193] RIP: 0033:0x7ff54af45cd8 [11554.311195] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11554.311197] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11554.311199] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [11554.311200] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [11554.311201] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [11554.311202] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [11554.311203] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11554.312439] NMI backtrace for cpu 0 [11554.618520] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11554.624443] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11554.630731] Call Trace: [11554.631869] [11554.633456] ? dump_stack+0xbb/0x10e [11554.635823] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [11554.642733] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [11554.650520] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [11554.656077] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [11554.665782] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [11554.670815] ? rcu_dump_cpu_stacks+0x11b/0x171 [11554.674760] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [11554.677792] ? update_process_times+0x6d/0xc0 [11554.681132] ? tick_sched_handle.isra.18+0x27/0x90 [11554.683665] ? tick_sched_timer+0x53/0xb0 [11554.690067] ? __hrtimer_run_queues+0x1c2/0x4d0 [11554.695414] ? tick_sched_handle.isra.18+0x90/0x90 [11554.700816] ? hrtimer_interrupt+0x128/0x2c0 [11554.705776] ? smp_apic_timer_interrupt+0xa6/0x270 [11554.718290] ? apic_timer_interrupt+0xf/0x20 [11554.720637] [11554.724091] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11554.728220] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11554.732287] ? do_raw_spin_lock+0x177/0x190 [11554.735200] ? _raw_spin_lock+0x21/0x30 [11554.737445] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11554.742693] ? obd_mod_free+0x46/0x140 [ptlrpc] [11554.746021] ? mdc_commit_open+0x91/0xb0 [mdc] [11554.751080] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11554.756125] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11554.762540] ? after_reply+0xe2c/0x1780 [ptlrpc] [11554.768697] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11554.772582] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [11554.777852] ? do_wait_intr+0xf0/0xf0 [11554.779218] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11554.788851] ? kthread+0x1d1/0x200 [11554.792901] ? set_kthread_struct+0x70/0x70 [11554.797152] ? ret_from_fork+0x1f/0x30 [11568.059009] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [11568.061920] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11568.094988] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11568.105799] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11568.111926] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11568.115949] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11568.120833] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11568.132489] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11568.145517] RAX: 0000000000002119 RBX: ffff96848811cae0 RCX: 0000000000000001 [11568.151990] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [11568.155942] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [11568.159912] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [11568.163329] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [11568.167833] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [11568.172001] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11568.174604] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [11568.178359] Call Trace: [11568.179779] [11568.181363] ? show_regs.cold.9+0x22/0x2f [11568.182808] ? watchdog_timer_fn.cold.6+0x62/0x10a [11568.184750] ? __hrtimer_run_queues+0x1c2/0x4d0 [11568.186447] ? watchdog+0x40/0x40 [11568.187907] ? hrtimer_interrupt+0x128/0x2c0 [11568.192187] ? smp_apic_timer_interrupt+0xa6/0x270 [11568.194329] ? apic_timer_interrupt+0xf/0x20 [11568.196362] [11568.198189] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11568.201537] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11568.205135] do_raw_spin_lock+0x177/0x190 [11568.207143] _raw_spin_lock+0x21/0x30 [11568.208965] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11568.211821] process_one_work+0x2c8/0x700 [11568.214123] worker_thread+0x296/0x6e0 [11568.215531] ? rescuer_thread+0x570/0x570 [11568.216914] kthread+0x1d1/0x200 [11568.218805] ? set_kthread_struct+0x70/0x70 [11568.221014] ret_from_fork+0x1f/0x30 [11576.062102] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [11576.069331] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11576.120875] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11576.139557] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11576.160358] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11576.168119] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11576.189482] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11576.199662] RAX: 00000000000078a4 RBX: ffff96848811cae0 RCX: 0000000000000001 [11576.204955] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [11576.213834] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [11576.217410] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [11576.227530] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [11576.264505] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [11576.275436] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11576.287996] BUG: workqueue lockup - pool [11576.290983] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [11576.290999] Call Trace: [11576.291008] [11576.291016] ? show_regs.cold.9+0x22/0x2f [11576.291032] ? watchdog_timer_fn.cold.6+0x62/0x10a [11576.291038] ? __hrtimer_run_queues+0x1c2/0x4d0 [11576.291043] ? watchdog+0x40/0x40 [11576.291046] ? hrtimer_interrupt+0x128/0x2c0 [11576.291050] ? smp_apic_timer_interrupt+0xa6/0x270 [11576.291058] ? apic_timer_interrupt+0xf/0x20 [11576.291062] [11576.291065] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11576.291073] do_raw_spin_lock+0x177/0x190 [11576.291077] _raw_spin_lock+0x21/0x30 [11576.291082] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11576.292877] cpus=0-3 flags=0x4 nice=0 stuck for 9617s! [11576.292894] Showing busy workqueues and worker pools: [11576.292897] workqueue events: flags=0x0 [11576.292910] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11576.292920] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [11576.293521] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [11576.293532] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [11576.293755] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11576.293763] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [11576.293997] workqueue events_long: flags=0x0 [11576.294021] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11576.294029] pending: lu_objects_flush [obdclass] [11576.294221] workqueue events_unbound: flags=0x2 [11576.294223] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [11576.294231] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11576.294255] workqueue events_power_efficient: flags=0x80 [11576.294258] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11576.294264] pending: neigh_periodic_work [11576.294273] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11576.294279] pending: do_cache_clean, check_lifetime [11576.294292] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11576.294299] pending: neigh_periodic_work [11576.294310] workqueue writeback: flags=0x4a [11576.294313] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11576.294318] pending: wb_workfn, wb_workfn [11576.294332] workqueue xprtiod: flags=0xa [11576.294335] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11576.294340] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [11576.294358] workqueue ipv6_addrconf: flags=0x40008 [11576.294361] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11576.294368] pending: addrconf_verify_work [11576.294377] workqueue knbd0-recv: flags=0x1a [11576.294379] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11576.294386] in-flight: 657:recv_work [11576.294395] workqueue ptlrpc_pinger: flags=0x2 [11576.294397] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11576.294402] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [11576.294650] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9617s workers=4 idle: 20831 12642 28859 [11576.294663] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11566s workers=2 idle: 659 [11576.607151] ? do_raw_spin_unlock+0x75/0x190 [11576.609531] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11576.613437] ? do_wait_intr+0xf0/0xf0 [11576.615933] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11576.620933] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11576.631586] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11576.636114] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11576.638536] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11576.641552] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11576.646944] mdc_intent_lock+0x353/0x7e0 [mdc] [11576.649946] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11576.652833] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11576.655790] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11576.659629] lmv_intent_open+0x338/0x1150 [lmv] [11576.662148] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11576.665116] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11576.668596] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11576.677254] ll_intent_lock+0xc5/0xaa0 [lustre] [11576.680895] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11576.691921] ll_lookup_it+0xe0e/0x2270 [lustre] [11576.704390] ? do_raw_spin_unlock+0x75/0x190 [11576.711621] ? _raw_spin_unlock+0x12/0x30 [11576.713118] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11576.715525] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11576.719101] ll_atomic_open+0x36a/0xd70 [lustre] [11576.721426] lookup_open+0x356/0xb40 [11576.726288] do_last+0x313/0xfc0 [11576.730384] ? nd_jump_root+0xe5/0x160 [11576.734444] ? path_init+0x437/0x520 [11576.737144] path_openat+0xf7/0x500 [11576.741522] ? do_raw_spin_unlock+0x75/0x190 [11576.745928] do_filp_open+0x99/0x140 [11576.748462] ? getname_flags+0x6e/0x330 [11576.750436] ? __check_object_size+0xff/0x256 [11576.757290] ? do_raw_spin_unlock+0x75/0x190 [11576.762408] ? _raw_spin_unlock+0x12/0x30 [11576.768251] do_sys_openat2+0x2b4/0x410 [11576.772883] do_sys_open+0x73/0xa0 [11576.776817] __x64_sys_openat+0x24/0x30 [11576.780874] do_syscall_64+0xc1/0x440 [11576.784868] entry_SYSCALL_64_after_hwframe+0x49/0xae [11576.789561] RIP: 0033:0x7ff54af45332 [11576.791912] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11576.801588] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11576.805568] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [11576.809309] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [11576.813983] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [11576.819886] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [11576.828850] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [11580.063423] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [11580.067110] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11580.114298] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11580.123662] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11580.129991] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11580.141855] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11580.157648] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11580.164694] RAX: 0000000000006116 RBX: ffff96848811cae0 RCX: 0000000000000001 [11580.171641] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [11580.176601] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [11580.179316] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [11580.184534] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [11580.190368] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [11580.197243] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11580.201196] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [11580.203797] Call Trace: [11580.204652] [11580.206439] ? show_regs.cold.9+0x22/0x2f [11580.210553] ? watchdog_timer_fn.cold.6+0x62/0x10a [11580.215660] ? __hrtimer_run_queues+0x1c2/0x4d0 [11580.220022] ? watchdog+0x40/0x40 [11580.221690] ? hrtimer_interrupt+0x128/0x2c0 [11580.227205] ? smp_apic_timer_interrupt+0xa6/0x270 [11580.233545] ? apic_timer_interrupt+0xf/0x20 [11580.239174] [11580.240482] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11580.244331] do_raw_spin_lock+0x177/0x190 [11580.246793] _raw_spin_lock+0x21/0x30 [11580.248705] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11580.254138] ? do_raw_spin_unlock+0x75/0x190 [11580.256513] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11580.263421] ? do_wait_intr+0xf0/0xf0 [11580.267262] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11580.271570] mdc_close+0x53b/0xd70 [mdc] [11580.274173] lmv_close+0x18d/0x490 [lmv] [11580.276915] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [11580.281368] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11580.287164] ll_md_real_close+0xa5/0x2a0 [lustre] [11580.290550] ll_md_close+0x274/0x800 [lustre] [11580.293117] ll_file_release+0x128/0xc50 [lustre] [11580.294655] __fput+0x10d/0x440 [11580.295582] ____fput+0x12/0x20 [11580.296904] task_work_run+0xc6/0x110 [11580.300340] exit_to_usermode_loop+0x1dd/0x1f0 [11580.303182] do_syscall_64+0x3d6/0x440 [11580.305373] entry_SYSCALL_64_after_hwframe+0x49/0xae [11580.307644] RIP: 0033:0x7ff54af45cd8 [11580.309346] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11580.318654] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11580.322440] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [11580.326347] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [11580.329954] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [11580.333485] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [11580.336451] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11596.063281] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [11596.067319] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11596.126502] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11596.140833] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11596.145870] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11596.153772] RIP: 0010:native_safe_halt+0x2e/0x50 [11596.155378] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11596.172334] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11596.184343] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [11596.194098] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c2134154 [11596.209075] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [11596.221050] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [11596.232347] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [11596.239640] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [11596.248116] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11596.255137] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [11596.264481] Call Trace: [11596.265407] [11596.266142] ? show_regs.cold.9+0x22/0x2f [11596.268538] ? watchdog_timer_fn.cold.6+0x62/0x10a [11596.281280] ? __hrtimer_run_queues+0x1c2/0x4d0 [11596.287624] ? watchdog+0x40/0x40 [11596.288437] ? hrtimer_interrupt+0x128/0x2c0 [11596.289435] ? smp_apic_timer_interrupt+0xa6/0x270 [11596.295310] ? apic_timer_interrupt+0xf/0x20 [11596.296879] [11596.297600] ? native_safe_halt+0x2e/0x50 [11596.311723] kvm_wait+0xcd/0xe0 [11596.314229] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11596.325305] do_raw_spin_lock+0x177/0x190 [11596.326732] _raw_spin_lock+0x21/0x30 [11596.336107] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11596.342510] process_one_work+0x2c8/0x700 [11596.347961] worker_thread+0x296/0x6e0 [11596.349481] ? rescuer_thread+0x570/0x570 [11596.355215] kthread+0x1d1/0x200 [11596.357056] ? set_kthread_struct+0x70/0x70 [11596.360051] ret_from_fork+0x1f/0x30 [11604.055212] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [11604.059972] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11604.064511] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [11604.107877] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11604.107894] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11604.110680] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11604.110979] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11604.110987] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11604.110991] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11604.111061] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11604.111130] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11604.111135] RAX: 000000000000760d RBX: ffff96848811cae0 RCX: 0000000000000001 [11604.111137] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [11604.111140] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [11604.111142] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [11604.111144] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [11604.111148] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [11604.111151] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11604.111154] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [11604.111161] Call Trace: [11604.111175] [11604.111181] ? show_regs.cold.9+0x22/0x2f [11604.111189] ? watchdog_timer_fn.cold.6+0x62/0x10a [11604.111196] ? __hrtimer_run_queues+0x1c2/0x4d0 [11604.111204] ? watchdog+0x40/0x40 [11604.111209] ? hrtimer_interrupt+0x128/0x2c0 [11604.111213] ? smp_apic_timer_interrupt+0xa6/0x270 [11604.111221] ? apic_timer_interrupt+0xf/0x20 [11604.111225] [11604.111228] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11604.111235] do_raw_spin_lock+0x177/0x190 [11604.111240] _raw_spin_lock+0x21/0x30 [11604.111244] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11604.124525] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11604.130720] ? do_raw_spin_unlock+0x75/0x190 [11604.130740] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11604.132189] ? do_wait_intr+0xf0/0xf0 [11604.132201] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11604.132412] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11604.132631] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11604.132853] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11604.133129] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11604.133170] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11604.133199] mdc_intent_lock+0x353/0x7e0 [mdc] [11604.133227] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11604.133357] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11604.133578] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11604.133608] lmv_intent_open+0x338/0x1150 [lmv] [11604.133635] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11604.135060] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11604.135147] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11604.135176] ll_intent_lock+0xc5/0xaa0 [lustre] [11604.135256] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11604.136331] ll_lookup_it+0xe0e/0x2270 [lustre] [11604.136415] ? do_raw_spin_unlock+0x75/0x190 [11604.136424] ? _raw_spin_unlock+0x12/0x30 [11604.136430] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11604.136455] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11604.136474] ll_atomic_open+0x36a/0xd70 [lustre] [11604.136558] lookup_open+0x356/0xb40 [11604.136568] do_last+0x313/0xfc0 [11604.136573] ? nd_jump_root+0xe5/0x160 [11604.136578] ? path_init+0x437/0x520 [11604.136582] path_openat+0xf7/0x500 [11604.136587] ? do_raw_spin_unlock+0x75/0x190 [11604.136592] do_filp_open+0x99/0x140 [11604.136598] ? getname_flags+0x6e/0x330 [11604.136602] ? __check_object_size+0xff/0x256 [11604.136609] ? do_raw_spin_unlock+0x75/0x190 [11604.136613] ? _raw_spin_unlock+0x12/0x30 [11604.136617] do_sys_openat2+0x2b4/0x410 [11604.136621] do_sys_open+0x73/0xa0 [11604.136625] __x64_sys_openat+0x24/0x30 [11604.136628] do_syscall_64+0xc1/0x440 [11604.136634] entry_SYSCALL_64_after_hwframe+0x49/0xae [11604.136641] RIP: 0033:0x7ff54af45332 [11604.136648] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11604.136652] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11604.136657] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [11604.136659] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [11604.136661] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [11604.136664] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [11604.136666] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [11604.517565] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11604.532237] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11604.540460] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [11604.545495] RDX: 00000000000021ac RSI: 0000000000000003 RDI: ffff96848811cae0 [11604.554370] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [11604.561159] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [11604.567925] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11604.573563] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [11604.581041] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11604.587153] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [11604.594081] Call Trace: [11604.595981] [11604.596945] ? show_regs.cold.9+0x22/0x2f [11604.600569] ? watchdog_timer_fn.cold.6+0x62/0x10a [11604.606436] ? __hrtimer_run_queues+0x1c2/0x4d0 [11604.608883] ? watchdog+0x40/0x40 [11604.611143] ? hrtimer_interrupt+0x128/0x2c0 [11604.616205] ? smp_apic_timer_interrupt+0xa6/0x270 [11604.619758] ? apic_timer_interrupt+0xf/0x20 [11604.624034] [11604.625290] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11604.630430] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11604.634884] do_raw_spin_lock+0x177/0x190 [11604.639352] _raw_spin_lock+0x21/0x30 [11604.640979] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11604.648153] obd_mod_free+0x46/0x140 [ptlrpc] [11604.650453] mdc_commit_open+0x91/0xb0 [mdc] [11604.662043] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11604.666988] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11604.670606] after_reply+0xe2c/0x1780 [ptlrpc] [11604.684890] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11604.687128] ptlrpcd+0xa82/0xe40 [ptlrpc] [11604.696008] ? do_wait_intr+0xf0/0xf0 [11604.697308] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11604.702508] kthread+0x1d1/0x200 [11604.709946] ? set_kthread_struct+0x70/0x70 [11604.711897] ret_from_fork+0x1f/0x30 [11606.495253] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9647s! [11606.498738] Showing busy workqueues and worker pools: [11606.508636] workqueue events: flags=0x0 [11606.515898] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11606.518614] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [11606.527360] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [11606.531759] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [11606.536698] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11606.539140] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [11606.541592] workqueue events_long: flags=0x0 [11606.542953] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11606.545964] pending: lu_objects_flush [obdclass] [11606.548112] workqueue events_unbound: flags=0x2 [11606.549654] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [11606.552370] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11606.556963] workqueue events_power_efficient: flags=0x80 [11606.558858] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11606.561495] pending: neigh_periodic_work [11606.562955] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11606.565167] pending: do_cache_clean, check_lifetime [11606.567477] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11606.571995] pending: neigh_periodic_work [11606.573855] workqueue writeback: flags=0x4a [11606.575975] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11606.577974] pending: wb_workfn, wb_workfn [11606.580557] workqueue xprtiod: flags=0xa [11606.582081] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11606.585546] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [11606.588613] workqueue ipv6_addrconf: flags=0x40008 [11606.590413] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11606.592564] pending: addrconf_verify_work [11606.595198] workqueue knbd0-recv: flags=0x1a [11606.596895] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11606.599832] in-flight: 657:recv_work [11606.601202] workqueue ptlrpc_pinger: flags=0x2 [11606.603498] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11606.605542] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [11606.608064] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9647s workers=4 idle: 20831 12642 28859 [11606.611430] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11596s workers=2 idle: 659 [11608.066147] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [11608.081683] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11608.127914] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11608.133261] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11608.136899] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11608.139127] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11608.147199] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11608.150286] RAX: 0000000000007e84 RBX: ffff96848811cae0 RCX: 0000000000000001 [11608.152854] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [11608.156242] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [11608.159295] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [11608.163761] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [11608.168185] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [11608.171883] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11608.174411] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [11608.177851] Call Trace: [11608.178998] [11608.180013] ? show_regs.cold.9+0x22/0x2f [11608.181932] ? watchdog_timer_fn.cold.6+0x62/0x10a [11608.184193] ? __hrtimer_run_queues+0x1c2/0x4d0 [11608.186729] ? watchdog+0x40/0x40 [11608.188381] ? hrtimer_interrupt+0x128/0x2c0 [11608.190187] ? smp_apic_timer_interrupt+0xa6/0x270 [11608.192422] ? apic_timer_interrupt+0xf/0x20 [11608.194506] [11608.197147] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11608.200590] do_raw_spin_lock+0x177/0x190 [11608.203206] _raw_spin_lock+0x21/0x30 [11608.204986] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11608.208263] ? do_raw_spin_unlock+0x75/0x190 [11608.212583] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11608.216815] ? do_wait_intr+0xf0/0xf0 [11608.220026] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11608.223861] mdc_close+0x53b/0xd70 [mdc] [11608.226232] lmv_close+0x18d/0x490 [lmv] [11608.231717] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [11608.236571] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11608.240420] ll_md_real_close+0xa5/0x2a0 [lustre] [11608.244537] ll_md_close+0x274/0x800 [lustre] [11608.247429] ll_file_release+0x128/0xc50 [lustre] [11608.256192] __fput+0x10d/0x440 [11608.257642] ____fput+0x12/0x20 [11608.259513] task_work_run+0xc6/0x110 [11608.261497] exit_to_usermode_loop+0x1dd/0x1f0 [11608.263724] do_syscall_64+0x3d6/0x440 [11608.266273] entry_SYSCALL_64_after_hwframe+0x49/0xae [11608.269789] RIP: 0033:0x7ff54af45cd8 [11608.274091] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11608.297474] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11608.300518] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [11608.307736] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [11608.314461] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [11608.321476] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [11608.334340] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11624.059022] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [11624.061413] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11624.080124] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11624.085168] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11624.091897] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11624.095169] RIP: 0010:native_safe_halt+0x2e/0x50 [11624.097251] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11624.106243] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11624.109691] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [11624.113341] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [11624.117123] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [11624.121886] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [11624.125251] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [11624.129149] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [11624.133992] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11624.138894] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [11624.142067] Call Trace: [11624.145081] [11624.145604] ? show_regs.cold.9+0x22/0x2f [11624.147141] ? watchdog_timer_fn.cold.6+0x62/0x10a [11624.149140] ? __hrtimer_run_queues+0x1c2/0x4d0 [11624.153113] ? watchdog+0x40/0x40 [11624.159425] ? hrtimer_interrupt+0x128/0x2c0 [11624.161984] ? smp_apic_timer_interrupt+0xa6/0x270 [11624.164815] ? apic_timer_interrupt+0xf/0x20 [11624.166913] [11624.168582] ? native_safe_halt+0x2e/0x50 [11624.170483] kvm_wait+0xcd/0xe0 [11624.172024] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11624.175152] do_raw_spin_lock+0x177/0x190 [11624.177279] _raw_spin_lock+0x21/0x30 [11624.179683] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11624.182819] process_one_work+0x2c8/0x700 [11624.184411] worker_thread+0x296/0x6e0 [11624.186919] ? rescuer_thread+0x570/0x570 [11624.189047] kthread+0x1d1/0x200 [11624.191760] ? set_kthread_struct+0x70/0x70 [11624.193855] ret_from_fork+0x1f/0x30 [11632.055045] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [11632.059445] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11632.060028] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [11632.095564] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11632.097926] Modules linked in: lustre(O) [11632.106746] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11632.106751] osp(O) ofd(O) lod(O) mdt(O) [11632.111881] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11632.114987] mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11632.115051] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11632.115058] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11632.115060] RIP: 0010:native_safe_halt+0x2e/0x50 [11632.115072] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11632.115077] RSP: 0000:ffffaf9fc15234f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11632.115081] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [11632.115083] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [11632.115085] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [11632.115087] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [11632.115089] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [11632.115092] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [11632.115095] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11632.115097] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [11632.115105] Call Trace: [11632.115110] [11632.115115] ? show_regs.cold.9+0x22/0x2f [11632.115124] ? watchdog_timer_fn.cold.6+0x62/0x10a [11632.115130] ? __hrtimer_run_queues+0x1c2/0x4d0 [11632.115134] ? watchdog+0x40/0x40 [11632.115138] ? hrtimer_interrupt+0x128/0x2c0 [11632.115143] ? smp_apic_timer_interrupt+0xa6/0x270 [11632.115149] ? apic_timer_interrupt+0xf/0x20 [11632.115154] [11632.115157] ? native_safe_halt+0x2e/0x50 [11632.115161] kvm_wait+0xcd/0xe0 [11632.115169] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11632.115177] do_raw_spin_lock+0x177/0x190 [11632.115182] _raw_spin_lock+0x21/0x30 [11632.115186] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11632.115633] ? do_raw_spin_unlock+0x75/0x190 [11632.115640] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11632.115857] ? do_wait_intr+0xf0/0xf0 [11632.115866] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11632.116081] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11632.116317] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11632.116549] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11632.116685] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11632.116725] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11632.116755] mdc_intent_lock+0x353/0x7e0 [mdc] [11632.116783] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11632.116869] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11632.117256] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11632.117288] lmv_intent_open+0x338/0x1150 [lmv] [11632.117321] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11632.117410] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11632.117493] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11632.117517] ll_intent_lock+0xc5/0xaa0 [lustre] [11632.117621] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11632.117712] ll_lookup_it+0xe0e/0x2270 [lustre] [11632.117802] ? do_raw_spin_unlock+0x75/0x190 [11632.117810] ? _raw_spin_unlock+0x12/0x30 [11632.117816] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11632.117844] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11632.117864] ll_atomic_open+0x36a/0xd70 [lustre] [11632.117955] lookup_open+0x356/0xb40 [11632.117964] do_last+0x313/0xfc0 [11632.117969] ? nd_jump_root+0xe5/0x160 [11632.117974] ? path_init+0x437/0x520 [11632.117978] path_openat+0xf7/0x500 [11632.117983] ? do_raw_spin_unlock+0x75/0x190 [11632.117988] do_filp_open+0x99/0x140 [11632.117994] ? getname_flags+0x6e/0x330 [11632.117998] ? __check_object_size+0xff/0x256 [11632.118004] ? do_raw_spin_unlock+0x75/0x190 [11632.118007] ? _raw_spin_unlock+0x12/0x30 [11632.118011] do_sys_openat2+0x2b4/0x410 [11632.118016] do_sys_open+0x73/0xa0 [11632.118021] __x64_sys_openat+0x24/0x30 [11632.118025] do_syscall_64+0xc1/0x440 [11632.118031] entry_SYSCALL_64_after_hwframe+0x49/0xae [11632.118037] RIP: 0033:0x7ff54af45332 [11632.118044] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11632.118048] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11632.118053] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [11632.118055] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [11632.118057] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [11632.118060] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [11632.118062] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [11632.705415] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11632.724299] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11632.736334] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [11632.741486] RDX: 0000000000002852 RSI: 0000000000000003 RDI: ffff96848811cae0 [11632.745880] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [11632.754266] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [11632.759302] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11632.764512] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [11632.770050] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11632.774537] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [11632.783300] Call Trace: [11632.786051] [11632.787918] ? show_regs.cold.9+0x22/0x2f [11632.793546] ? watchdog_timer_fn.cold.6+0x62/0x10a [11632.798874] ? __hrtimer_run_queues+0x1c2/0x4d0 [11632.803562] ? watchdog+0x40/0x40 [11632.806326] ? hrtimer_interrupt+0x128/0x2c0 [11632.814260] ? smp_apic_timer_interrupt+0xa6/0x270 [11632.816728] ? apic_timer_interrupt+0xf/0x20 [11632.822836] [11632.825885] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11632.841369] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11632.848365] do_raw_spin_lock+0x177/0x190 [11632.857282] _raw_spin_lock+0x21/0x30 [11632.863917] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11632.874116] obd_mod_free+0x46/0x140 [ptlrpc] [11632.880264] mdc_commit_open+0x91/0xb0 [mdc] [11632.883830] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11632.887318] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11632.894233] after_reply+0xe2c/0x1780 [ptlrpc] [11632.901316] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11632.910349] ptlrpcd+0xa82/0xe40 [ptlrpc] [11632.919078] ? do_wait_intr+0xf0/0xf0 [11632.924241] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11632.938579] kthread+0x1d1/0x200 [11632.942005] ? set_kthread_struct+0x70/0x70 [11632.947885] ret_from_fork+0x1f/0x30 [11636.068000] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [11636.074113] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11636.113127] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11636.121873] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11636.126749] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11636.137980] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11636.155116] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11636.160141] RAX: 0000000000004f8c RBX: ffff96848811cae0 RCX: 0000000000000001 [11636.172064] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [11636.178685] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [11636.181064] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [11636.186917] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [11636.193697] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [11636.198306] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11636.200248] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [11636.206114] Call Trace: [11636.218535] [11636.219303] ? show_regs.cold.9+0x22/0x2f [11636.220762] ? watchdog_timer_fn.cold.6+0x62/0x10a [11636.227413] ? __hrtimer_run_queues+0x1c2/0x4d0 [11636.229030] ? watchdog+0x40/0x40 [11636.240338] ? hrtimer_interrupt+0x128/0x2c0 [11636.246810] ? smp_apic_timer_interrupt+0xa6/0x270 [11636.251936] ? apic_timer_interrupt+0xf/0x20 [11636.254531] [11636.255646] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11636.257501] do_raw_spin_lock+0x177/0x190 [11636.259941] _raw_spin_lock+0x21/0x30 [11636.262498] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11636.266171] ? do_raw_spin_unlock+0x75/0x190 [11636.268742] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11636.270901] ? do_wait_intr+0xf0/0xf0 [11636.274025] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11636.276886] mdc_close+0x53b/0xd70 [mdc] [11636.278031] lmv_close+0x18d/0x490 [lmv] [11636.279117] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [11636.283220] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11636.285432] ll_md_real_close+0xa5/0x2a0 [lustre] [11636.293848] ll_md_close+0x274/0x800 [lustre] [11636.297078] ll_file_release+0x128/0xc50 [lustre] [11636.301660] __fput+0x10d/0x440 [11636.303206] ____fput+0x12/0x20 [11636.310212] task_work_run+0xc6/0x110 [11636.313155] exit_to_usermode_loop+0x1dd/0x1f0 [11636.318716] do_syscall_64+0x3d6/0x440 [11636.322733] entry_SYSCALL_64_after_hwframe+0x49/0xae [11636.325221] RIP: 0033:0x7ff54af45cd8 [11636.328883] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11636.340821] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11636.345175] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [11636.351922] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [11636.362136] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [11636.368188] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [11636.374215] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11636.703234] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9677s! [11636.712146] Showing busy workqueues and worker pools: [11636.715677] workqueue events: flags=0x0 [11636.720665] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11636.724073] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [11636.732533] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [11636.739340] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [11636.749180] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11636.753444] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [11636.756611] workqueue events_long: flags=0x0 [11636.759741] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11636.765895] pending: lu_objects_flush [obdclass] [11636.772796] workqueue events_unbound: flags=0x2 [11636.775783] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [11636.779806] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11636.792035] workqueue events_power_efficient: flags=0x80 [11636.794016] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11636.805772] pending: neigh_periodic_work [11636.808796] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11636.821854] pending: do_cache_clean, check_lifetime [11636.828434] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11636.836719] pending: neigh_periodic_work [11636.841786] workqueue writeback: flags=0x4a [11636.845305] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11636.859558] pending: wb_workfn, wb_workfn [11636.862705] workqueue xprtiod: flags=0xa [11636.865826] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11636.869187] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [11636.873709] workqueue ipv6_addrconf: flags=0x40008 [11636.876874] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11636.882083] pending: addrconf_verify_work [11636.893583] workqueue knbd0-recv: flags=0x1a [11636.896610] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11636.908599] in-flight: 657:recv_work [11636.914134] workqueue ptlrpc_pinger: flags=0x2 [11636.922688] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11636.934494] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [11636.943347] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9678s workers=4 idle: 20831 12642 28859 [11636.954322] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11627s workers=2 idle: 659 [11652.064232] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [11652.093373] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11652.172923] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11652.187981] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11652.202079] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11652.211749] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11652.222212] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11652.244201] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11652.254154] RAX: 0000000000007f11 RBX: ffff96848811cae0 RCX: 0000000000000001 [11652.259022] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [11652.268110] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [11652.282634] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [11652.291127] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [11652.298018] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [11652.303108] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11652.308823] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [11652.313964] Call Trace: [11652.314611] [11652.315070] ? show_regs.cold.9+0x22/0x2f [11652.316141] ? watchdog_timer_fn.cold.6+0x62/0x10a [11652.319755] ? __hrtimer_run_queues+0x1c2/0x4d0 [11652.325850] ? watchdog+0x40/0x40 [11652.326933] ? hrtimer_interrupt+0x128/0x2c0 [11652.329017] ? smp_apic_timer_interrupt+0xa6/0x270 [11652.332767] ? apic_timer_interrupt+0xf/0x20 [11652.336871] [11652.338183] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11652.343568] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11652.345539] do_raw_spin_lock+0x177/0x190 [11652.348864] _raw_spin_lock+0x21/0x30 [11652.352310] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11652.355139] process_one_work+0x2c8/0x700 [11652.357087] worker_thread+0x296/0x6e0 [11652.359123] ? rescuer_thread+0x570/0x570 [11652.361984] kthread+0x1d1/0x200 [11652.363059] ? set_kthread_struct+0x70/0x70 [11652.364869] ret_from_fork+0x1f/0x30 [11660.062337] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [11660.073442] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11660.165245] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11660.204005] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11660.218655] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11660.224850] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11660.252845] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11660.276047] RAX: 000000000000128d RBX: ffff96848811cae0 RCX: 0000000000000001 [11660.283024] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [11660.294484] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [11660.306022] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [11660.323711] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [11660.337852] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [11660.354184] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11660.362545] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [11660.382843] Call Trace: [11660.389439] [11660.392681] ? show_regs.cold.9+0x22/0x2f [11660.411510] ? watchdog_timer_fn.cold.6+0x62/0x10a [11660.416557] ? __hrtimer_run_queues+0x1c2/0x4d0 [11660.424412] ? watchdog+0x40/0x40 [11660.430066] ? hrtimer_interrupt+0x128/0x2c0 [11660.435767] ? smp_apic_timer_interrupt+0xa6/0x270 [11660.449393] ? apic_timer_interrupt+0xf/0x20 [11660.454281] [11660.455073] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11660.463521] do_raw_spin_lock+0x177/0x190 [11660.465388] _raw_spin_lock+0x21/0x30 [11660.466712] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11660.473559] ? do_raw_spin_unlock+0x75/0x190 [11660.481387] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11660.489417] ? do_wait_intr+0xf0/0xf0 [11660.492748] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11660.497000] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11660.501061] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11660.504585] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11660.510107] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11660.512008] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11660.518371] mdc_intent_lock+0x353/0x7e0 [mdc] [11660.521659] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11660.524965] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11660.534922] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11660.539430] lmv_intent_open+0x338/0x1150 [lmv] [11660.545151] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11660.556024] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11660.568230] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11660.573001] ll_intent_lock+0xc5/0xaa0 [lustre] [11660.574742] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11660.588763] ll_lookup_it+0xe0e/0x2270 [lustre] [11660.598515] ? do_raw_spin_unlock+0x75/0x190 [11660.611206] ? _raw_spin_unlock+0x12/0x30 [11660.614859] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11660.628469] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11660.630195] ll_atomic_open+0x36a/0xd70 [lustre] [11660.640950] lookup_open+0x356/0xb40 [11660.648119] do_last+0x313/0xfc0 [11660.650419] ? nd_jump_root+0xe5/0x160 [11660.654551] ? path_init+0x437/0x520 [11660.662053] path_openat+0xf7/0x500 [11660.666679] ? do_raw_spin_unlock+0x75/0x190 [11660.673246] do_filp_open+0x99/0x140 [11660.674446] ? getname_flags+0x6e/0x330 [11660.678935] ? __check_object_size+0xff/0x256 [11660.690682] ? do_raw_spin_unlock+0x75/0x190 [11660.698543] ? _raw_spin_unlock+0x12/0x30 [11660.699865] do_sys_openat2+0x2b4/0x410 [11660.718124] do_sys_open+0x73/0xa0 [11660.720682] __x64_sys_openat+0x24/0x30 [11660.727111] do_syscall_64+0xc1/0x440 [11660.730691] entry_SYSCALL_64_after_hwframe+0x49/0xae [11660.737764] RIP: 0033:0x7ff54af45332 [11660.741061] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11660.754765] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11660.766220] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [11660.777423] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [11660.781928] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [11660.794110] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [11660.805842] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [11664.055067] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [11664.060078] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11664.065652] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [11664.100985] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11664.101008] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11664.101011] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11664.101032] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11664.101036] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11664.101824] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [11664.101828] RDX: 0000000000007610 RSI: 0000000000000003 RDI: ffff96848811cae0 [11664.101830] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [11664.101833] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [11664.101836] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11664.101840] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [11664.101844] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11664.101846] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [11664.101856] Call Trace: [11664.101865] [11664.101874] ? show_regs.cold.9+0x22/0x2f [11664.101889] ? watchdog_timer_fn.cold.6+0x62/0x10a [11664.101897] ? __hrtimer_run_queues+0x1c2/0x4d0 [11664.101902] ? watchdog+0x40/0x40 [11664.101906] ? hrtimer_interrupt+0x128/0x2c0 [11664.101911] ? smp_apic_timer_interrupt+0xa6/0x270 [11664.101923] ? apic_timer_interrupt+0xf/0x20 [11664.101928] [11664.101931] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11664.101937] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11664.101942] do_raw_spin_lock+0x177/0x190 [11664.101947] _raw_spin_lock+0x21/0x30 [11664.101955] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11664.102821] obd_mod_free+0x46/0x140 [ptlrpc] [11664.103027] mdc_commit_open+0x91/0xb0 [mdc] [11664.103630] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11664.103842] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11664.104097] after_reply+0xe2c/0x1780 [ptlrpc] [11664.104681] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11664.104881] ptlrpcd+0xa82/0xe40 [ptlrpc] [11664.105376] ? do_wait_intr+0xf0/0xf0 [11664.105451] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11664.105767] kthread+0x1d1/0x200 [11664.105774] ? set_kthread_struct+0x70/0x70 [11664.105779] ret_from_fork+0x1f/0x30 [11664.247527] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11664.267988] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11664.273413] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11664.277668] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11664.280090] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11664.288753] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11664.292504] RAX: 0000000000006486 RBX: ffff96848811cae0 RCX: 0000000000000001 [11664.295762] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [11664.299411] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [11664.302250] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [11664.305320] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [11664.308483] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [11664.312070] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11664.314931] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [11664.318457] Call Trace: [11664.319702] [11664.320783] ? show_regs.cold.9+0x22/0x2f [11664.323110] ? watchdog_timer_fn.cold.6+0x62/0x10a [11664.325882] ? __hrtimer_run_queues+0x1c2/0x4d0 [11664.328007] ? watchdog+0x40/0x40 [11664.329664] ? hrtimer_interrupt+0x128/0x2c0 [11664.331466] ? smp_apic_timer_interrupt+0xa6/0x270 [11664.333740] ? apic_timer_interrupt+0xf/0x20 [11664.336049] [11664.336984] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11664.339151] do_raw_spin_lock+0x177/0x190 [11664.340766] _raw_spin_lock+0x21/0x30 [11664.342582] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11664.345842] ? do_raw_spin_unlock+0x75/0x190 [11664.347708] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11664.350538] ? do_wait_intr+0xf0/0xf0 [11664.352043] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11664.355138] mdc_close+0x53b/0xd70 [mdc] [11664.357383] lmv_close+0x18d/0x490 [lmv] [11664.359088] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [11664.362099] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11664.365086] ll_md_real_close+0xa5/0x2a0 [lustre] [11664.367516] ll_md_close+0x274/0x800 [lustre] [11664.369635] ll_file_release+0x128/0xc50 [lustre] [11664.371961] __fput+0x10d/0x440 [11664.373630] ____fput+0x12/0x20 [11664.374865] task_work_run+0xc6/0x110 [11664.376795] exit_to_usermode_loop+0x1dd/0x1f0 [11664.378864] do_syscall_64+0x3d6/0x440 [11664.380623] entry_SYSCALL_64_after_hwframe+0x49/0xae [11664.383008] RIP: 0033:0x7ff54af45cd8 [11664.384879] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11664.393462] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11664.396792] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [11664.400073] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [11664.403538] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [11664.407133] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [11664.410719] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11667.424197] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9708s! [11667.431674] Showing busy workqueues and worker pools: [11667.434568] workqueue events: flags=0x0 [11667.440565] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11667.448070] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [11667.457380] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [11667.464787] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [11667.476576] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11667.482421] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [11667.488896] workqueue events_long: flags=0x0 [11667.497200] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11667.502524] pending: lu_objects_flush [obdclass] [11667.508483] workqueue events_unbound: flags=0x2 [11667.519420] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [11667.528096] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11667.543948] workqueue events_power_efficient: flags=0x80 [11667.551356] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11667.555833] pending: neigh_periodic_work [11667.560489] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11667.564887] pending: do_cache_clean, check_lifetime [11667.569031] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11667.573148] pending: neigh_periodic_work [11667.576485] workqueue writeback: flags=0x4a [11667.578867] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11667.582568] pending: wb_workfn, wb_workfn [11667.585604] workqueue xprtiod: flags=0xa [11667.589713] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11667.597431] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [11667.607378] workqueue ipv6_addrconf: flags=0x40008 [11667.611533] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11667.619837] pending: addrconf_verify_work [11667.624157] workqueue knbd0-recv: flags=0x1a [11667.626968] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11667.631549] in-flight: 657:recv_work [11667.633932] workqueue ptlrpc_pinger: flags=0x2 [11667.637707] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11667.641127] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [11667.644710] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9708s workers=4 idle: 20831 12642 28859 [11667.650524] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11657s workers=2 idle: 659 [11680.065569] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [11680.073882] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11680.130451] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11680.142462] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11680.158637] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11680.164808] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11680.173429] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11680.198171] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [11680.206234] RAX: 0000000000005d59 RBX: ffff96848811cae0 RCX: 0000000000000001 [11680.210002] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [11680.214413] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [11680.220500] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [11680.223488] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [11680.232651] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [11680.241283] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11680.250343] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [11680.254462] Call Trace: [11680.255523] [11680.257486] ? show_regs.cold.9+0x22/0x2f [11680.265422] ? watchdog_timer_fn.cold.6+0x62/0x10a [11680.271333] ? __hrtimer_run_queues+0x1c2/0x4d0 [11680.273669] ? watchdog+0x40/0x40 [11680.275532] ? hrtimer_interrupt+0x128/0x2c0 [11680.277757] ? smp_apic_timer_interrupt+0xa6/0x270 [11680.285135] ? apic_timer_interrupt+0xf/0x20 [11680.287684] [11680.288815] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11680.300594] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11680.308275] do_raw_spin_lock+0x177/0x190 [11680.312777] _raw_spin_lock+0x21/0x30 [11680.314569] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11680.318215] process_one_work+0x2c8/0x700 [11680.321827] worker_thread+0x296/0x6e0 [11680.326388] ? rescuer_thread+0x570/0x570 [11680.332160] kthread+0x1d1/0x200 [11680.332916] ? set_kthread_struct+0x70/0x70 [11680.333882] ret_from_fork+0x1f/0x30 [11688.061410] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [11688.066698] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11688.098517] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11688.104410] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11688.111364] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11688.114087] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11688.128743] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11688.132633] RAX: 0000000000006a22 RBX: ffff96848811cae0 RCX: 0000000000000001 [11688.137737] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [11688.139852] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [11688.144132] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [11688.148145] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [11688.152472] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [11688.156419] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11688.160814] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [11688.169664] Call Trace: [11688.172994] [11688.173742] ? show_regs.cold.9+0x22/0x2f [11688.175031] ? watchdog_timer_fn.cold.6+0x62/0x10a [11688.183106] ? __hrtimer_run_queues+0x1c2/0x4d0 [11688.190498] ? watchdog+0x40/0x40 [11688.194915] ? hrtimer_interrupt+0x128/0x2c0 [11688.195918] ? smp_apic_timer_interrupt+0xa6/0x270 [11688.197254] ? apic_timer_interrupt+0xf/0x20 [11688.202137] [11688.203013] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11688.207836] do_raw_spin_lock+0x177/0x190 [11688.209401] _raw_spin_lock+0x21/0x30 [11688.213522] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11688.216035] ? do_raw_spin_unlock+0x75/0x190 [11688.219460] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11688.220943] ? do_wait_intr+0xf0/0xf0 [11688.224702] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11688.226584] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11688.233613] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11688.240793] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11688.246187] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11688.252025] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11688.254179] mdc_intent_lock+0x353/0x7e0 [mdc] [11688.256114] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11688.259415] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11688.266429] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11688.271451] lmv_intent_open+0x338/0x1150 [lmv] [11688.275117] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11688.280647] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11688.282584] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11688.291136] ll_intent_lock+0xc5/0xaa0 [lustre] [11688.296086] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11688.301346] ll_lookup_it+0xe0e/0x2270 [lustre] [11688.313643] ? do_raw_spin_unlock+0x75/0x190 [11688.317557] ? _raw_spin_unlock+0x12/0x30 [11688.319716] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11688.321118] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11688.322304] ll_atomic_open+0x36a/0xd70 [lustre] [11688.329407] lookup_open+0x356/0xb40 [11688.332372] do_last+0x313/0xfc0 [11688.335866] ? nd_jump_root+0xe5/0x160 [11688.341040] ? path_init+0x437/0x520 [11688.345654] path_openat+0xf7/0x500 [11688.346831] ? do_raw_spin_unlock+0x75/0x190 [11688.349693] do_filp_open+0x99/0x140 [11688.357945] ? getname_flags+0x6e/0x330 [11688.359298] ? __check_object_size+0xff/0x256 [11688.374277] ? do_raw_spin_unlock+0x75/0x190 [11688.375781] ? _raw_spin_unlock+0x12/0x30 [11688.380595] do_sys_openat2+0x2b4/0x410 [11688.381945] do_sys_open+0x73/0xa0 [11688.391270] __x64_sys_openat+0x24/0x30 [11688.392714] do_syscall_64+0xc1/0x440 [11688.396830] entry_SYSCALL_64_after_hwframe+0x49/0xae [11688.399818] RIP: 0033:0x7ff54af45332 [11688.403152] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11688.420485] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11688.431391] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [11688.435580] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [11688.448320] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [11688.450390] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [11688.468390] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [11692.055168] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [11692.058031] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [11692.059719] Modules linked in: [11692.063205] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11692.063284] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11692.063292] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11692.063295] RIP: 0010:native_safe_halt+0x2e/0x50 [11692.063309] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11692.063313] RSP: 0018:ffffaf9fc1543ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11692.063318] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [11692.063320] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c20b4154 [11692.063322] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [11692.063325] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [11692.063328] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [11692.063331] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [11692.063334] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11692.063337] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [11692.063346] Call Trace: [11692.063353] [11692.063360] ? show_regs.cold.9+0x22/0x2f [11692.063370] ? watchdog_timer_fn.cold.6+0x62/0x10a [11692.063378] ? __hrtimer_run_queues+0x1c2/0x4d0 [11692.063383] ? watchdog+0x40/0x40 [11692.063388] ? hrtimer_interrupt+0x128/0x2c0 [11692.063391] ? smp_apic_timer_interrupt+0xa6/0x270 [11692.063398] ? apic_timer_interrupt+0xf/0x20 [11692.063403] [11692.063408] ? native_safe_halt+0x2e/0x50 [11692.066161] lustre(O) [11692.083982] kvm_wait+0xcd/0xe0 [11692.089874] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11692.089953] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11692.089960] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11692.089963] RIP: 0010:native_safe_halt+0x2e/0x50 [11692.089979] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11692.089983] RSP: 0000:ffffaf9fc1b97bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11692.089987] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [11692.089989] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff96848811cae0 [11692.089991] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [11692.089993] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [11692.089995] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11692.089999] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [11692.090001] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11692.090003] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [11692.090018] Call Trace: [11692.090024] [11692.090031] ? show_regs.cold.9+0x22/0x2f [11692.090042] ? watchdog_timer_fn.cold.6+0x62/0x10a [11692.090049] ? __hrtimer_run_queues+0x1c2/0x4d0 [11692.090053] ? watchdog+0x40/0x40 [11692.090058] ? hrtimer_interrupt+0x128/0x2c0 [11692.090062] ? smp_apic_timer_interrupt+0xa6/0x270 [11692.090069] ? apic_timer_interrupt+0xf/0x20 [11692.090073] [11692.090076] ? native_safe_halt+0x2e/0x50 [11692.090080] kvm_wait+0xcd/0xe0 [11692.090089] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11692.090097] do_raw_spin_lock+0x177/0x190 [11692.090102] _raw_spin_lock+0x21/0x30 [11692.090107] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11692.090386] obd_mod_free+0x46/0x140 [ptlrpc] [11692.090584] mdc_commit_open+0x91/0xb0 [mdc] [11692.090620] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11692.090816] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11692.091026] after_reply+0xe2c/0x1780 [ptlrpc] [11692.091219] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11692.091414] ptlrpcd+0xa82/0xe40 [ptlrpc] [11692.091618] ? do_wait_intr+0xf0/0xf0 [11692.091628] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11692.091821] kthread+0x1d1/0x200 [11692.091827] ? set_kthread_struct+0x70/0x70 [11692.091831] ret_from_fork+0x1f/0x30 [11692.336690] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11692.340952] do_raw_spin_lock+0x177/0x190 [11692.342491] _raw_spin_lock+0x21/0x30 [11692.343841] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11692.353287] ? do_raw_spin_unlock+0x75/0x190 [11692.358331] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11692.364145] ? do_wait_intr+0xf0/0xf0 [11692.366938] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11692.370167] mdc_close+0x53b/0xd70 [mdc] [11692.372060] lmv_close+0x18d/0x490 [lmv] [11692.374167] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [11692.390119] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11692.404256] ll_md_real_close+0xa5/0x2a0 [lustre] [11692.406634] ll_md_close+0x274/0x800 [lustre] [11692.415072] ll_file_release+0x128/0xc50 [lustre] [11692.420065] __fput+0x10d/0x440 [11692.424262] ____fput+0x12/0x20 [11692.425335] task_work_run+0xc6/0x110 [11692.435657] exit_to_usermode_loop+0x1dd/0x1f0 [11692.439737] do_syscall_64+0x3d6/0x440 [11692.446624] entry_SYSCALL_64_after_hwframe+0x49/0xae [11692.458101] RIP: 0033:0x7ff54af45cd8 [11692.459280] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11692.475709] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11692.483107] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [11692.496384] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [11692.499800] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [11692.508157] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [11692.514386] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11698.144276] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9739s! [11698.152712] Showing busy workqueues and worker pools: [11698.154715] workqueue events: flags=0x0 [11698.157819] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11698.164483] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [11698.169286] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [11698.175548] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [11698.183301] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11698.188286] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [11698.193261] workqueue events_long: flags=0x0 [11698.198278] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11698.203020] pending: lu_objects_flush [obdclass] [11698.206350] workqueue events_unbound: flags=0x2 [11698.209183] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [11698.212327] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11698.217575] workqueue events_power_efficient: flags=0x80 [11698.223592] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11698.228620] pending: neigh_periodic_work [11698.231256] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11698.237546] pending: do_cache_clean, check_lifetime [11698.240681] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11698.246120] pending: neigh_periodic_work [11698.249594] workqueue writeback: flags=0x4a [11698.252685] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11698.255695] pending: wb_workfn, wb_workfn [11698.257833] workqueue xprtiod: flags=0xa [11698.259723] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11698.265349] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [11698.276548] workqueue ipv6_addrconf: flags=0x40008 [11698.283265] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11698.286081] pending: addrconf_verify_work [11698.287709] workqueue knbd0-recv: flags=0x1a [11698.289615] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11698.293280] in-flight: 657:recv_work [11698.295282] workqueue ptlrpc_pinger: flags=0x2 [11698.296943] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11698.300234] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [11698.306755] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9739s workers=4 idle: 20831 12642 28859 [11698.312738] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11688s workers=2 idle: 659 [11708.059021] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [11708.063108] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11708.105309] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11708.115774] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11708.124210] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11708.129818] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11708.140147] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11708.154522] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11708.160513] RAX: 00000000000025a7 RBX: ffff96848811cae0 RCX: 0000000000000001 [11708.167842] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [11708.172731] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [11708.176372] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [11708.184501] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [11708.189280] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [11708.197532] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11708.203275] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [11708.214460] Call Trace: [11708.218281] [11708.222059] ? show_regs.cold.9+0x22/0x2f [11708.227289] ? watchdog_timer_fn.cold.6+0x62/0x10a [11708.230028] ? __hrtimer_run_queues+0x1c2/0x4d0 [11708.235843] ? watchdog+0x40/0x40 [11708.241479] ? hrtimer_interrupt+0x128/0x2c0 [11708.253788] ? smp_apic_timer_interrupt+0xa6/0x270 [11708.257052] ? apic_timer_interrupt+0xf/0x20 [11708.258687] [11708.259533] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11708.273218] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11708.280164] do_raw_spin_lock+0x177/0x190 [11708.288117] _raw_spin_lock+0x21/0x30 [11708.290048] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11708.298536] process_one_work+0x2c8/0x700 [11708.301004] worker_thread+0x296/0x6e0 [11708.303364] ? rescuer_thread+0x570/0x570 [11708.306113] kthread+0x1d1/0x200 [11708.308212] ? set_kthread_struct+0x70/0x70 [11708.311260] ret_from_fork+0x1f/0x30 [11716.060906] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [11716.073172] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11716.120681] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11716.144081] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11716.154608] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11716.161335] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11716.190554] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11716.207269] RAX: 0000000000007304 RBX: ffff96848811cae0 RCX: 0000000000000001 [11716.216120] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [11716.226117] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [11716.234449] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [11716.248968] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [11716.251446] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [11716.269323] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11716.282236] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [11716.289224] Call Trace: [11716.290131] [11716.302567] ? show_regs.cold.9+0x22/0x2f [11716.304114] ? watchdog_timer_fn.cold.6+0x62/0x10a [11716.305752] ? __hrtimer_run_queues+0x1c2/0x4d0 [11716.320374] ? watchdog+0x40/0x40 [11716.323333] ? hrtimer_interrupt+0x128/0x2c0 [11716.330348] ? smp_apic_timer_interrupt+0xa6/0x270 [11716.342105] ? apic_timer_interrupt+0xf/0x20 [11716.347892] [11716.350827] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11716.358374] do_raw_spin_lock+0x177/0x190 [11716.370945] _raw_spin_lock+0x21/0x30 [11716.372170] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11716.380015] ? do_raw_spin_unlock+0x75/0x190 [11716.387549] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11716.392939] ? do_wait_intr+0xf0/0xf0 [11716.401332] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11716.405165] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11716.412302] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11716.421700] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11716.431127] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11716.436094] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11716.441890] mdc_intent_lock+0x353/0x7e0 [mdc] [11716.444537] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11716.450297] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11716.453965] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11716.463687] lmv_intent_open+0x338/0x1150 [lmv] [11716.468173] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11716.484187] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11716.488022] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11716.489626] ll_intent_lock+0xc5/0xaa0 [lustre] [11716.496754] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11716.501443] ll_lookup_it+0xe0e/0x2270 [lustre] [11716.503616] ? do_raw_spin_unlock+0x75/0x190 [11716.505122] ? _raw_spin_unlock+0x12/0x30 [11716.510891] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11716.514175] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11716.519079] ll_atomic_open+0x36a/0xd70 [lustre] [11716.523529] lookup_open+0x356/0xb40 [11716.524768] do_last+0x313/0xfc0 [11716.532845] ? nd_jump_root+0xe5/0x160 [11716.536170] ? path_init+0x437/0x520 [11716.540223] path_openat+0xf7/0x500 [11716.541349] ? do_raw_spin_unlock+0x75/0x190 [11716.550329] do_filp_open+0x99/0x140 [11716.553327] ? getname_flags+0x6e/0x330 [11716.554643] ? __check_object_size+0xff/0x256 [11716.562045] ? do_raw_spin_unlock+0x75/0x190 [11716.563859] ? _raw_spin_unlock+0x12/0x30 [11716.566084] do_sys_openat2+0x2b4/0x410 [11716.571873] do_sys_open+0x73/0xa0 [11716.574732] __x64_sys_openat+0x24/0x30 [11716.576925] do_syscall_64+0xc1/0x440 [11716.579060] entry_SYSCALL_64_after_hwframe+0x49/0xae [11716.587412] RIP: 0033:0x7ff54af45332 [11716.599017] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11716.620081] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11716.629094] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [11716.643339] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [11716.651011] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [11716.661014] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [11716.671340] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [11720.058027] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [11720.067294] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11720.119414] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11720.126024] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11720.132052] RIP: 0010:native_safe_halt+0x2e/0x50 [11720.136784] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11720.153082] RSP: 0018:ffffaf9fc1543ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11720.162758] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [11720.174897] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c20b4154 [11720.184507] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [11720.189887] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [11720.194059] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [11720.197612] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [11720.201010] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11720.204437] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [11720.207916] Call Trace: [11720.209139] [11720.210050] ? show_regs.cold.9+0x22/0x2f [11720.212698] ? watchdog_timer_fn.cold.6+0x62/0x10a [11720.214925] ? __hrtimer_run_queues+0x1c2/0x4d0 [11720.217041] ? watchdog+0x40/0x40 [11720.218547] ? hrtimer_interrupt+0x128/0x2c0 [11720.220759] ? smp_apic_timer_interrupt+0xa6/0x270 [11720.224028] ? apic_timer_interrupt+0xf/0x20 [11720.225943] [11720.227806] ? native_safe_halt+0x2e/0x50 [11720.229600] kvm_wait+0xcd/0xe0 [11720.231363] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11720.234725] do_raw_spin_lock+0x177/0x190 [11720.236696] _raw_spin_lock+0x21/0x30 [11720.238415] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11720.241490] ? do_raw_spin_unlock+0x75/0x190 [11720.242720] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11720.244800] ? do_wait_intr+0xf0/0xf0 [11720.246768] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11720.249541] mdc_close+0x53b/0xd70 [mdc] [11720.252772] lmv_close+0x18d/0x490 [lmv] [11720.255680] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [11720.262054] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11720.269532] ll_md_real_close+0xa5/0x2a0 [lustre] [11720.275464] ll_md_close+0x274/0x800 [lustre] [11720.279481] ll_file_release+0x128/0xc50 [lustre] [11720.286992] __fput+0x10d/0x440 [11720.292423] ____fput+0x12/0x20 [11720.295838] task_work_run+0xc6/0x110 [11720.300438] exit_to_usermode_loop+0x1dd/0x1f0 [11720.305820] do_syscall_64+0x3d6/0x440 [11720.307699] entry_SYSCALL_64_after_hwframe+0x49/0xae [11720.313529] RIP: 0033:0x7ff54af45cd8 [11720.315914] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11720.335094] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11720.341016] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [11720.347184] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [11720.352907] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [11720.358388] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [11720.361451] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11724.055341] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [11724.058180] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11724.086277] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11724.091950] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11724.095298] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11724.097764] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11724.105735] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11724.108912] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [11724.115276] RDX: 0000000000003517 RSI: 0000000000000003 RDI: ffff96848811cae0 [11724.117586] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [11724.122135] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [11724.126047] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11724.130826] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [11724.135387] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11724.141479] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [11724.144270] Call Trace: [11724.146661] [11724.148008] ? show_regs.cold.9+0x22/0x2f [11724.152284] ? watchdog_timer_fn.cold.6+0x62/0x10a [11724.154226] ? __hrtimer_run_queues+0x1c2/0x4d0 [11724.155907] ? watchdog+0x40/0x40 [11724.164193] ? hrtimer_interrupt+0x128/0x2c0 [11724.170765] ? smp_apic_timer_interrupt+0xa6/0x270 [11724.174134] ? apic_timer_interrupt+0xf/0x20 [11724.177416] [11724.180062] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11724.182287] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11724.185628] do_raw_spin_lock+0x177/0x190 [11724.187263] _raw_spin_lock+0x21/0x30 [11724.189215] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11724.191274] obd_mod_free+0x46/0x140 [ptlrpc] [11724.193988] mdc_commit_open+0x91/0xb0 [mdc] [11724.195846] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11724.200957] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11724.202835] after_reply+0xe2c/0x1780 [ptlrpc] [11724.205430] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11724.209884] ptlrpcd+0xa82/0xe40 [ptlrpc] [11724.212471] ? do_wait_intr+0xf0/0xf0 [11724.214372] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11724.218467] kthread+0x1d1/0x200 [11724.220567] ? set_kthread_struct+0x70/0x70 [11724.222572] ret_from_fork+0x1f/0x30 [11728.351243] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9769s! [11728.355974] Showing busy workqueues and worker pools: [11728.359224] workqueue events: flags=0x0 [11728.361763] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11728.365877] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [11728.371643] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [11728.376004] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [11728.384347] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11728.388912] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [11728.392725] workqueue events_long: flags=0x0 [11728.395147] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11728.399373] pending: lu_objects_flush [obdclass] [11728.402752] workqueue events_unbound: flags=0x2 [11728.405498] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [11728.410747] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11728.419427] workqueue events_power_efficient: flags=0x80 [11728.422947] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11728.427672] pending: neigh_periodic_work [11728.430281] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11728.435064] pending: do_cache_clean, check_lifetime [11728.439324] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11728.443677] pending: neigh_periodic_work [11728.446171] workqueue writeback: flags=0x4a [11728.448826] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11728.452670] pending: wb_workfn, wb_workfn [11728.454994] workqueue xprtiod: flags=0xa [11728.457712] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11728.461499] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [11728.466148] workqueue ipv6_addrconf: flags=0x40008 [11728.468841] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11728.472673] pending: addrconf_verify_work [11728.475325] workqueue knbd0-recv: flags=0x1a [11728.477774] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11728.481169] in-flight: 657:recv_work [11728.483544] workqueue ptlrpc_pinger: flags=0x2 [11728.485979] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11728.489253] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [11728.492312] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9769s workers=4 idle: 20831 12642 28859 [11728.495954] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11718s workers=2 idle: 659 [11734.117054] rcu: INFO: rcu_sched self-detected stall on CPU [11734.119626] rcu: 0-...!: (5165460 ticks this GP) idle=98a/1/0x4000000000000002 softirq=125154/125154 fqs=959 [11734.124135] (t=9780277 jiffies g=253585 q=6013) [11734.125974] rcu: rcu_sched kthread starved for 9775237 jiffies! g253585 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=3 [11734.130621] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [11734.136187] rcu: RCU grace-period kthread stack dump: [11734.138496] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [11734.142992] Call Trace: [11734.144044] __schedule+0x351/0xcb0 [11734.145974] schedule+0xc0/0x180 [11734.148138] schedule_timeout+0xb4/0x190 [11734.150790] ? do_raw_spin_unlock+0x75/0x190 [11734.153296] ? __next_timer_interrupt+0x160/0x160 [11734.155820] ? prepare_to_swait_event+0xf3/0x140 [11734.158218] rcu_gp_kthread+0x8dd/0x15b0 [11734.160719] ? force_qs_rnp+0x2c0/0x2c0 [11734.162369] kthread+0x1d1/0x200 [11734.163986] ? set_kthread_struct+0x70/0x70 [11734.165866] ret_from_fork+0x1f/0x30 [11734.167703] rcu: Stack dump where RCU GP kthread last ran: [11734.173169] Sending NMI from CPU 0 to CPUs 3: [11734.175701] NMI backtrace for cpu 3 skipped: idling at native_safe_halt+0x2e/0x50 [11734.178384] NMI backtrace for cpu 0 [11734.180779] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11734.186813] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11734.190755] Call Trace: [11734.192145] [11734.193147] ? dump_stack+0xbb/0x10e [11734.194331] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [11734.196699] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [11734.199107] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [11734.201403] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [11734.204365] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [11734.207147] ? rcu_dump_cpu_stacks+0x11b/0x171 [11734.209113] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [11734.211736] ? update_process_times+0x6d/0xc0 [11734.213766] ? tick_sched_handle.isra.18+0x27/0x90 [11734.215989] ? tick_sched_timer+0x53/0xb0 [11734.217947] ? __hrtimer_run_queues+0x1c2/0x4d0 [11734.220187] ? tick_sched_handle.isra.18+0x90/0x90 [11734.222246] ? hrtimer_interrupt+0x128/0x2c0 [11734.224357] ? smp_apic_timer_interrupt+0xa6/0x270 [11734.226673] ? apic_timer_interrupt+0xf/0x20 [11734.228532] [11734.229622] ? native_safe_halt+0x2e/0x50 [11734.231359] ? kvm_wait+0xcd/0xe0 [11734.232982] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11734.234835] ? do_raw_spin_lock+0x177/0x190 [11734.236931] ? _raw_spin_lock+0x21/0x30 [11734.238120] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11734.240324] ? obd_mod_free+0x46/0x140 [ptlrpc] [11734.242227] ? mdc_commit_open+0x91/0xb0 [mdc] [11734.243353] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11734.245251] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11734.247977] ? after_reply+0xe2c/0x1780 [ptlrpc] [11734.250297] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11734.252252] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [11734.254667] ? do_wait_intr+0xf0/0xf0 [11734.255693] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11734.257813] ? kthread+0x1d1/0x200 [11734.259400] ? set_kthread_struct+0x70/0x70 [11734.261358] ? ret_from_fork+0x1f/0x30 [11736.062718] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [11736.072472] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11736.118887] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11736.129535] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11736.139039] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11736.145761] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11736.156751] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11736.176238] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11736.189483] RAX: 0000000000004e8b RBX: ffff96848811cae0 RCX: 0000000000000001 [11736.196529] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [11736.201266] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [11736.208141] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [11736.227545] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [11736.237510] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [11736.244354] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11736.252400] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [11736.262020] Call Trace: [11736.263690] [11736.264349] ? show_regs.cold.9+0x22/0x2f [11736.267255] ? watchdog_timer_fn.cold.6+0x62/0x10a [11736.272163] ? __hrtimer_run_queues+0x1c2/0x4d0 [11736.275112] ? watchdog+0x40/0x40 [11736.277090] ? hrtimer_interrupt+0x128/0x2c0 [11736.280261] ? smp_apic_timer_interrupt+0xa6/0x270 [11736.283028] ? apic_timer_interrupt+0xf/0x20 [11736.284556] [11736.285338] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11736.288611] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11736.291608] do_raw_spin_lock+0x177/0x190 [11736.294005] _raw_spin_lock+0x21/0x30 [11736.296172] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11736.301274] process_one_work+0x2c8/0x700 [11736.305767] worker_thread+0x296/0x6e0 [11736.309318] ? rescuer_thread+0x570/0x570 [11736.313302] kthread+0x1d1/0x200 [11736.318162] ? set_kthread_struct+0x70/0x70 [11736.323107] ret_from_fork+0x1f/0x30 [11748.058019] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [11748.069164] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11748.133146] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11748.149912] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11748.166631] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11748.172309] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11748.196014] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11748.204057] RAX: 0000000000006010 RBX: ffff96848811cae0 RCX: 0000000000000001 [11748.212011] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [11748.220132] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [11748.225216] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [11748.233257] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [11748.241843] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [11748.252427] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11748.257577] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [11748.263766] Call Trace: [11748.267922] [11748.271942] ? show_regs.cold.9+0x22/0x2f [11748.277200] ? watchdog_timer_fn.cold.6+0x62/0x10a [11748.285719] ? __hrtimer_run_queues+0x1c2/0x4d0 [11748.291865] ? watchdog+0x40/0x40 [11748.294818] ? hrtimer_interrupt+0x128/0x2c0 [11748.303048] ? smp_apic_timer_interrupt+0xa6/0x270 [11748.314608] ? apic_timer_interrupt+0xf/0x20 [11748.318229] [11748.321175] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11748.331405] do_raw_spin_lock+0x177/0x190 [11748.334590] _raw_spin_lock+0x21/0x30 [11748.336943] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11748.340574] ? do_raw_spin_unlock+0x75/0x190 [11748.343987] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11748.349178] ? do_wait_intr+0xf0/0xf0 [11748.351793] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11748.359835] mdc_close+0x53b/0xd70 [mdc] [11748.364957] lmv_close+0x18d/0x490 [lmv] [11748.371032] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [11748.376275] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11748.380056] ll_md_real_close+0xa5/0x2a0 [lustre] [11748.385102] ll_md_close+0x274/0x800 [lustre] [11748.390610] ll_file_release+0x128/0xc50 [lustre] [11748.392898] __fput+0x10d/0x440 [11748.398883] ____fput+0x12/0x20 [11748.403209] task_work_run+0xc6/0x110 [11748.409158] exit_to_usermode_loop+0x1dd/0x1f0 [11748.411845] do_syscall_64+0x3d6/0x440 [11748.416965] entry_SYSCALL_64_after_hwframe+0x49/0xae [11748.427091] RIP: 0033:0x7ff54af45cd8 [11748.432047] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11748.454163] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11748.464674] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [11748.472050] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [11748.478991] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [11748.488459] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [11748.493229] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11758.559861] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9799s! [11758.566386] Showing busy workqueues and worker pools: [11758.570170] workqueue events: flags=0x0 [11758.573243] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11758.578852] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [11758.587866] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [11758.594402] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [11758.604576] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11758.610994] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [11758.620177] workqueue events_long: flags=0x0 [11758.624173] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11758.631334] pending: lu_objects_flush [obdclass] [11758.636114] workqueue events_unbound: flags=0x2 [11758.638943] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [11758.642897] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11758.659633] workqueue events_power_efficient: flags=0x80 [11758.664549] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11758.669944] pending: neigh_periodic_work [11758.672396] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11758.678913] pending: do_cache_clean, check_lifetime [11758.682305] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11758.687048] pending: neigh_periodic_work [11758.690775] workqueue writeback: flags=0x4a [11758.693144] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11758.698562] pending: wb_workfn, wb_workfn [11758.701646] workqueue xprtiod: flags=0xa [11758.703944] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11758.707917] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [11758.714512] workqueue ipv6_addrconf: flags=0x40008 [11758.718823] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11758.725680] pending: addrconf_verify_work [11758.727943] workqueue knbd0-recv: flags=0x1a [11758.729983] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11758.737957] in-flight: 657:recv_work [11758.743395] workqueue ptlrpc_pinger: flags=0x2 [11758.748890] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11758.754833] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [11758.762481] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9799s workers=4 idle: 20831 12642 28859 [11758.767149] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11748s workers=2 idle: 659 [11760.065919] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [11760.072415] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11760.116801] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11760.121883] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11760.132710] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11760.135892] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11760.151591] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11760.158852] RAX: 0000000000005115 RBX: ffff96848811cae0 RCX: 0000000000000001 [11760.170875] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [11760.184819] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [11760.197839] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [11760.203619] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [11760.208674] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [11760.216915] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11760.230597] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [11760.235883] Call Trace: [11760.239628] [11760.241196] ? show_regs.cold.9+0x22/0x2f [11760.243529] ? watchdog_timer_fn.cold.6+0x62/0x10a [11760.246181] ? __hrtimer_run_queues+0x1c2/0x4d0 [11760.255710] ? watchdog+0x40/0x40 [11760.256802] ? hrtimer_interrupt+0x128/0x2c0 [11760.262017] ? smp_apic_timer_interrupt+0xa6/0x270 [11760.268609] ? apic_timer_interrupt+0xf/0x20 [11760.275127] [11760.276819] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11760.286409] do_raw_spin_lock+0x177/0x190 [11760.291114] _raw_spin_lock+0x21/0x30 [11760.295571] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11760.299762] ? do_raw_spin_unlock+0x75/0x190 [11760.302523] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11760.306588] ? do_wait_intr+0xf0/0xf0 [11760.308750] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11760.311728] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11760.313774] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11760.315826] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11760.318641] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11760.324031] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11760.328510] mdc_intent_lock+0x353/0x7e0 [mdc] [11760.331586] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11760.334763] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11760.340601] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11760.343737] lmv_intent_open+0x338/0x1150 [lmv] [11760.347011] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11760.353065] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11760.363733] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11760.365174] ll_intent_lock+0xc5/0xaa0 [lustre] [11760.369744] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11760.372596] ll_lookup_it+0xe0e/0x2270 [lustre] [11760.374025] ? do_raw_spin_unlock+0x75/0x190 [11760.375734] ? _raw_spin_unlock+0x12/0x30 [11760.377449] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11760.379965] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11760.386576] ll_atomic_open+0x36a/0xd70 [lustre] [11760.389748] lookup_open+0x356/0xb40 [11760.391551] do_last+0x313/0xfc0 [11760.392968] ? nd_jump_root+0xe5/0x160 [11760.395022] ? path_init+0x437/0x520 [11760.397613] path_openat+0xf7/0x500 [11760.400867] ? do_raw_spin_unlock+0x75/0x190 [11760.403616] do_filp_open+0x99/0x140 [11760.406981] ? getname_flags+0x6e/0x330 [11760.409851] ? __check_object_size+0xff/0x256 [11760.411896] ? do_raw_spin_unlock+0x75/0x190 [11760.416515] ? _raw_spin_unlock+0x12/0x30 [11760.417967] do_sys_openat2+0x2b4/0x410 [11760.419465] do_sys_open+0x73/0xa0 [11760.420617] __x64_sys_openat+0x24/0x30 [11760.422056] do_syscall_64+0xc1/0x440 [11760.423912] entry_SYSCALL_64_after_hwframe+0x49/0xae [11760.425507] RIP: 0033:0x7ff54af45332 [11760.426368] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11760.432669] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11760.434608] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [11760.436334] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [11760.438564] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [11760.440846] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [11760.443334] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [11764.061758] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [11764.065830] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11764.112405] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11764.123044] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11764.130296] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11764.134879] RIP: 0010:native_safe_halt+0x2e/0x50 [11764.142300] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11764.164829] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11764.173158] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [11764.176921] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c2134154 [11764.183647] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [11764.192267] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [11764.200692] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [11764.217538] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [11764.226674] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11764.234462] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [11764.247166] Call Trace: [11764.248903] [11764.252989] ? show_regs.cold.9+0x22/0x2f [11764.256272] ? watchdog_timer_fn.cold.6+0x62/0x10a [11764.263495] ? __hrtimer_run_queues+0x1c2/0x4d0 [11764.266173] ? watchdog+0x40/0x40 [11764.267174] ? hrtimer_interrupt+0x128/0x2c0 [11764.271162] ? smp_apic_timer_interrupt+0xa6/0x270 [11764.272925] ? apic_timer_interrupt+0xf/0x20 [11764.275622] [11764.276855] ? native_safe_halt+0x2e/0x50 [11764.289485] kvm_wait+0xcd/0xe0 [11764.290833] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11764.296491] do_raw_spin_lock+0x177/0x190 [11764.298778] _raw_spin_lock+0x21/0x30 [11764.305864] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11764.315994] process_one_work+0x2c8/0x700 [11764.318357] worker_thread+0x296/0x6e0 [11764.323674] ? rescuer_thread+0x570/0x570 [11764.327056] kthread+0x1d1/0x200 [11764.328944] ? set_kthread_struct+0x70/0x70 [11764.330593] ret_from_fork+0x1f/0x30 [11776.058011] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [11776.066818] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11776.092347] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11776.097226] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11776.100452] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11776.103000] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11776.111851] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11776.115931] RAX: 0000000000001b32 RBX: ffff96848811cae0 RCX: 0000000000000001 [11776.123601] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [11776.126959] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [11776.131660] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [11776.141261] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [11776.144228] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [11776.149811] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11776.153779] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [11776.156931] Call Trace: [11776.159173] [11776.159867] ? show_regs.cold.9+0x22/0x2f [11776.161986] ? watchdog_timer_fn.cold.6+0x62/0x10a [11776.164154] ? __hrtimer_run_queues+0x1c2/0x4d0 [11776.165748] ? watchdog+0x40/0x40 [11776.167247] ? hrtimer_interrupt+0x128/0x2c0 [11776.169300] ? smp_apic_timer_interrupt+0xa6/0x270 [11776.172050] ? apic_timer_interrupt+0xf/0x20 [11776.173927] [11776.174954] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11776.177730] do_raw_spin_lock+0x177/0x190 [11776.179888] _raw_spin_lock+0x21/0x30 [11776.183305] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11776.186997] ? do_raw_spin_unlock+0x75/0x190 [11776.188364] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11776.191294] ? do_wait_intr+0xf0/0xf0 [11776.192988] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11776.195832] mdc_close+0x53b/0xd70 [mdc] [11776.197003] lmv_close+0x18d/0x490 [lmv] [11776.198544] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [11776.199738] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11776.202310] ll_md_real_close+0xa5/0x2a0 [lustre] [11776.205291] ll_md_close+0x274/0x800 [lustre] [11776.206541] ll_file_release+0x128/0xc50 [lustre] [11776.208742] __fput+0x10d/0x440 [11776.210128] ____fput+0x12/0x20 [11776.211600] task_work_run+0xc6/0x110 [11776.212875] exit_to_usermode_loop+0x1dd/0x1f0 [11776.214253] do_syscall_64+0x3d6/0x440 [11776.215377] entry_SYSCALL_64_after_hwframe+0x49/0xae [11776.217556] RIP: 0033:0x7ff54af45cd8 [11776.219732] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11776.228574] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11776.232442] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [11776.236732] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [11776.238663] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [11776.241133] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [11776.243785] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11784.055152] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [11784.059310] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11784.090584] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11784.097516] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11784.103755] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11784.107670] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11784.116836] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11784.119794] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [11784.124617] RDX: 0000000000000e88 RSI: 0000000000000003 RDI: ffff96848811cae0 [11784.128675] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [11784.130786] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [11784.135068] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11784.138331] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [11784.142986] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11784.146635] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [11784.149587] Call Trace: [11784.152014] [11784.152788] ? show_regs.cold.9+0x22/0x2f [11784.155810] ? watchdog_timer_fn.cold.6+0x62/0x10a [11784.157837] ? __hrtimer_run_queues+0x1c2/0x4d0 [11784.160628] ? watchdog+0x40/0x40 [11784.162522] ? hrtimer_interrupt+0x128/0x2c0 [11784.163963] ? smp_apic_timer_interrupt+0xa6/0x270 [11784.167452] ? apic_timer_interrupt+0xf/0x20 [11784.169609] [11784.170428] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11784.172462] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11784.176112] do_raw_spin_lock+0x177/0x190 [11784.177217] _raw_spin_lock+0x21/0x30 [11784.178157] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11784.181435] obd_mod_free+0x46/0x140 [ptlrpc] [11784.186146] mdc_commit_open+0x91/0xb0 [mdc] [11784.190064] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11784.195743] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11784.199137] after_reply+0xe2c/0x1780 [ptlrpc] [11784.204316] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11784.209917] ptlrpcd+0xa82/0xe40 [ptlrpc] [11784.212471] ? do_wait_intr+0xf0/0xf0 [11784.214017] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11784.219646] kthread+0x1d1/0x200 [11784.221145] ? set_kthread_struct+0x70/0x70 [11784.224921] ret_from_fork+0x1f/0x30 [11788.060067] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [11788.063744] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11788.088685] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11788.094466] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11788.099219] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11788.101227] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11788.110741] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [11788.114583] RAX: 0000000000006f85 RBX: ffff96848811cae0 RCX: 0000000000000001 [11788.118043] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [11788.121253] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [11788.124691] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [11788.130632] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [11788.140054] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [11788.148674] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11788.153072] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [11788.157055] Call Trace: [11788.158410] [11788.159266] ? show_regs.cold.9+0x22/0x2f [11788.162818] ? watchdog_timer_fn.cold.6+0x62/0x10a [11788.170208] ? __hrtimer_run_queues+0x1c2/0x4d0 [11788.173421] ? watchdog+0x40/0x40 [11788.175172] ? hrtimer_interrupt+0x128/0x2c0 [11788.178203] ? smp_apic_timer_interrupt+0xa6/0x270 [11788.180637] ? apic_timer_interrupt+0xf/0x20 [11788.185586] [11788.186969] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11788.189379] do_raw_spin_lock+0x177/0x190 [11788.191419] _raw_spin_lock+0x21/0x30 [11788.193276] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11788.197065] ? do_raw_spin_unlock+0x75/0x190 [11788.199202] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11788.202578] ? do_wait_intr+0xf0/0xf0 [11788.204531] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11788.209683] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11788.215165] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11788.220067] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11788.223020] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11788.226019] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11788.229785] mdc_intent_lock+0x353/0x7e0 [mdc] [11788.231746] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11788.234492] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11788.236648] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11788.238690] lmv_intent_open+0x338/0x1150 [lmv] [11788.240883] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11788.243003] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11788.245746] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11788.247590] ll_intent_lock+0xc5/0xaa0 [lustre] [11788.249690] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11788.252005] ll_lookup_it+0xe0e/0x2270 [lustre] [11788.254206] ? do_raw_spin_unlock+0x75/0x190 [11788.255719] ? _raw_spin_unlock+0x12/0x30 [11788.257541] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11788.259648] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11788.261605] ll_atomic_open+0x36a/0xd70 [lustre] [11788.264027] lookup_open+0x356/0xb40 [11788.265340] do_last+0x313/0xfc0 [11788.266761] ? nd_jump_root+0xe5/0x160 [11788.267933] ? path_init+0x437/0x520 [11788.269223] path_openat+0xf7/0x500 [11788.270594] ? do_raw_spin_unlock+0x75/0x190 [11788.271890] do_filp_open+0x99/0x140 [11788.273246] ? getname_flags+0x6e/0x330 [11788.274602] ? __check_object_size+0xff/0x256 [11788.276011] ? do_raw_spin_unlock+0x75/0x190 [11788.277279] ? _raw_spin_unlock+0x12/0x30 [11788.279021] do_sys_openat2+0x2b4/0x410 [11788.280560] do_sys_open+0x73/0xa0 [11788.282000] __x64_sys_openat+0x24/0x30 [11788.283867] do_syscall_64+0xc1/0x440 [11788.285281] entry_SYSCALL_64_after_hwframe+0x49/0xae [11788.287283] RIP: 0033:0x7ff54af45332 [11788.288719] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11788.297167] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11788.300137] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [11788.303011] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [11788.306496] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [11788.310000] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [11788.312876] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [11789.279111] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9830s! [11789.286234] Showing busy workqueues and worker pools: [11789.290114] workqueue events: flags=0x0 [11789.292881] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11789.297616] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [11789.303211] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [11789.306375] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [11789.314502] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11789.320364] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [11789.324740] workqueue events_long: flags=0x0 [11789.327274] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11789.330787] pending: lu_objects_flush [obdclass] [11789.334867] workqueue events_unbound: flags=0x2 [11789.336580] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [11789.340114] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11789.345445] workqueue events_power_efficient: flags=0x80 [11789.349359] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11789.353857] pending: neigh_periodic_work [11789.356517] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11789.361784] pending: do_cache_clean, check_lifetime [11789.365340] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11789.368913] pending: neigh_periodic_work [11789.372710] workqueue writeback: flags=0x4a [11789.375412] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11789.379290] pending: wb_workfn, wb_workfn [11789.382861] workqueue xprtiod: flags=0xa [11789.385798] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11789.389783] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [11789.394431] workqueue ipv6_addrconf: flags=0x40008 [11789.398175] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11789.402592] pending: addrconf_verify_work [11789.406275] workqueue knbd0-recv: flags=0x1a [11789.409012] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11789.414000] in-flight: 657:recv_work [11789.417233] workqueue ptlrpc_pinger: flags=0x2 [11789.420245] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11789.424882] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [11789.429159] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9830s workers=4 idle: 20831 12642 28859 [11789.433385] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11779s workers=2 idle: 659 [11792.062756] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [11792.070158] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11792.112526] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11792.121233] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11792.126173] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11792.130934] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11792.137017] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11792.159517] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11792.166452] RAX: 0000000000006386 RBX: ffff96848811cae0 RCX: 0000000000000001 [11792.172831] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [11792.178917] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [11792.182223] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [11792.185434] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [11792.188633] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [11792.193450] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11792.204032] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [11792.207640] Call Trace: [11792.208458] [11792.209330] ? show_regs.cold.9+0x22/0x2f [11792.211609] ? watchdog_timer_fn.cold.6+0x62/0x10a [11792.214544] ? __hrtimer_run_queues+0x1c2/0x4d0 [11792.217119] ? watchdog+0x40/0x40 [11792.218839] ? hrtimer_interrupt+0x128/0x2c0 [11792.220843] ? smp_apic_timer_interrupt+0xa6/0x270 [11792.223651] ? apic_timer_interrupt+0xf/0x20 [11792.227375] [11792.228567] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11792.230654] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11792.233821] do_raw_spin_lock+0x177/0x190 [11792.236809] _raw_spin_lock+0x21/0x30 [11792.239451] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11792.242706] process_one_work+0x2c8/0x700 [11792.244165] worker_thread+0x296/0x6e0 [11792.245921] ? rescuer_thread+0x570/0x570 [11792.251197] kthread+0x1d1/0x200 [11792.253719] ? set_kthread_struct+0x70/0x70 [11792.257412] ret_from_fork+0x1f/0x30 [11804.060670] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [11804.076616] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11804.147689] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11804.154493] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11804.164403] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11804.172802] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11804.188142] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11804.198805] RAX: 000000000000620c RBX: ffff96848811cae0 RCX: 0000000000000001 [11804.210575] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [11804.218736] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [11804.227157] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [11804.239683] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [11804.248198] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [11804.257322] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11804.263638] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [11804.269228] Call Trace: [11804.271169] [11804.273601] ? show_regs.cold.9+0x22/0x2f [11804.277814] ? watchdog_timer_fn.cold.6+0x62/0x10a [11804.281658] ? __hrtimer_run_queues+0x1c2/0x4d0 [11804.286087] ? watchdog+0x40/0x40 [11804.289052] ? hrtimer_interrupt+0x128/0x2c0 [11804.292687] ? smp_apic_timer_interrupt+0xa6/0x270 [11804.296147] ? apic_timer_interrupt+0xf/0x20 [11804.297520] [11804.298436] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11804.304076] do_raw_spin_lock+0x177/0x190 [11804.305883] _raw_spin_lock+0x21/0x30 [11804.308590] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11804.313568] ? do_raw_spin_unlock+0x75/0x190 [11804.318577] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11804.327102] ? do_wait_intr+0xf0/0xf0 [11804.331571] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11804.339713] mdc_close+0x53b/0xd70 [mdc] [11804.354340] lmv_close+0x18d/0x490 [lmv] [11804.363715] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [11804.372570] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11804.387342] ll_md_real_close+0xa5/0x2a0 [lustre] [11804.402582] ll_md_close+0x274/0x800 [lustre] [11804.415018] ll_file_release+0x128/0xc50 [lustre] [11804.425317] __fput+0x10d/0x440 [11804.427373] ____fput+0x12/0x20 [11804.432855] task_work_run+0xc6/0x110 [11804.437965] exit_to_usermode_loop+0x1dd/0x1f0 [11804.442293] do_syscall_64+0x3d6/0x440 [11804.448209] entry_SYSCALL_64_after_hwframe+0x49/0xae [11804.462758] RIP: 0033:0x7ff54af45cd8 [11804.470538] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11804.501497] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11804.513163] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [11804.520704] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [11804.532542] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [11804.542096] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [11804.558781] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11816.055032] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [11816.061092] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [11816.063444] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) [11816.068425] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11816.068908] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11816.068915] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11816.068918] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11816.068933] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11816.068937] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11816.068942] RAX: 0000000000007c13 RBX: ffff96848811cae0 RCX: 0000000000000001 [11816.068945] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [11816.068948] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [11816.068950] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [11816.068952] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [11816.068956] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [11816.068959] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11816.068962] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [11816.068970] Call Trace: [11816.068975] [11816.068982] ? show_regs.cold.9+0x22/0x2f [11816.068993] ? watchdog_timer_fn.cold.6+0x62/0x10a [11816.069000] ? __hrtimer_run_queues+0x1c2/0x4d0 [11816.069005] ? watchdog+0x40/0x40 [11816.069009] ? hrtimer_interrupt+0x128/0x2c0 [11816.069014] ? smp_apic_timer_interrupt+0xa6/0x270 [11816.069021] ? apic_timer_interrupt+0xf/0x20 [11816.069026] [11816.069029] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11816.069036] do_raw_spin_lock+0x177/0x190 [11816.069041] _raw_spin_lock+0x21/0x30 [11816.069045] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11816.081051] lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11816.081100] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11816.081108] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11816.081110] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11816.081124] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11816.081129] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11816.081133] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [11816.081136] RDX: 0000000000003fde RSI: 0000000000000003 RDI: ffff96848811cae0 [11816.081138] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [11816.081140] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [11816.081143] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11816.081146] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [11816.081149] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11816.081151] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [11816.081159] Call Trace: [11816.081165] [11816.081170] ? show_regs.cold.9+0x22/0x2f [11816.081179] ? watchdog_timer_fn.cold.6+0x62/0x10a [11816.081185] ? __hrtimer_run_queues+0x1c2/0x4d0 [11816.081190] ? watchdog+0x40/0x40 [11816.081194] ? hrtimer_interrupt+0x128/0x2c0 [11816.081198] ? smp_apic_timer_interrupt+0xa6/0x270 [11816.081409] ? apic_timer_interrupt+0xf/0x20 [11816.081415] [11816.081418] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11816.081423] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11816.081428] do_raw_spin_lock+0x177/0x190 [11816.081432] _raw_spin_lock+0x21/0x30 [11816.081439] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11816.082067] obd_mod_free+0x46/0x140 [ptlrpc] [11816.082365] mdc_commit_open+0x91/0xb0 [mdc] [11816.082408] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11816.086775] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11816.124260] ? do_raw_spin_unlock+0x75/0x190 [11816.124281] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11816.124566] ? do_wait_intr+0xf0/0xf0 [11816.124576] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11816.124850] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11816.125070] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11816.125309] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11816.125427] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11816.125465] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11816.125497] mdc_intent_lock+0x353/0x7e0 [mdc] [11816.125527] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11816.125617] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11816.126050] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11816.126083] lmv_intent_open+0x338/0x1150 [lmv] [11816.126114] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11816.126210] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11816.126290] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11816.126313] ll_intent_lock+0xc5/0xaa0 [lustre] [11816.126402] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11816.126489] ll_lookup_it+0xe0e/0x2270 [lustre] [11816.126573] ? do_raw_spin_unlock+0x75/0x190 [11816.126580] ? _raw_spin_unlock+0x12/0x30 [11816.126587] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11816.126613] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11816.132996] after_reply+0xe2c/0x1780 [ptlrpc] [11816.133303] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11816.133518] ptlrpcd+0xa82/0xe40 [ptlrpc] [11816.133782] ? do_wait_intr+0xf0/0xf0 [11816.133793] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11816.134000] kthread+0x1d1/0x200 [11816.134006] ? set_kthread_struct+0x70/0x70 [11816.134010] ret_from_fork+0x1f/0x30 [11816.539730] ll_atomic_open+0x36a/0xd70 [lustre] [11816.544307] lookup_open+0x356/0xb40 [11816.546910] do_last+0x313/0xfc0 [11816.548766] ? nd_jump_root+0xe5/0x160 [11816.552406] ? path_init+0x437/0x520 [11816.555284] path_openat+0xf7/0x500 [11816.557775] ? do_raw_spin_unlock+0x75/0x190 [11816.561132] do_filp_open+0x99/0x140 [11816.562680] ? getname_flags+0x6e/0x330 [11816.565269] ? __check_object_size+0xff/0x256 [11816.569201] ? do_raw_spin_unlock+0x75/0x190 [11816.571534] ? _raw_spin_unlock+0x12/0x30 [11816.574064] do_sys_openat2+0x2b4/0x410 [11816.576604] do_sys_open+0x73/0xa0 [11816.578358] __x64_sys_openat+0x24/0x30 [11816.581934] do_syscall_64+0xc1/0x440 [11816.584202] entry_SYSCALL_64_after_hwframe+0x49/0xae [11816.587202] RIP: 0033:0x7ff54af45332 [11816.589319] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11816.601936] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11816.607616] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [11816.613888] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [11816.618503] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [11816.622291] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [11816.629268] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [11819.490668] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9860s! [11819.500586] Showing busy workqueues and worker pools: [11819.504806] workqueue events: flags=0x0 [11819.507049] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11819.519866] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [11819.536502] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [11819.549963] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [11819.570297] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11819.580683] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [11819.585307] workqueue events_long: flags=0x0 [11819.592017] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11819.603839] pending: lu_objects_flush [obdclass] [11819.606965] workqueue events_unbound: flags=0x2 [11819.614453] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [11819.623150] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11819.634310] workqueue events_power_efficient: flags=0x80 [11819.641300] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11819.646984] pending: neigh_periodic_work [11819.652039] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11819.658477] pending: do_cache_clean, check_lifetime [11819.663024] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11819.671275] pending: neigh_periodic_work [11819.677735] workqueue writeback: flags=0x4a [11819.686592] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11819.696091] pending: wb_workfn, wb_workfn [11819.700308] workqueue xprtiod: flags=0xa [11819.704056] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11819.709525] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [11819.715736] workqueue ipv6_addrconf: flags=0x40008 [11819.719117] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11819.724963] pending: addrconf_verify_work [11819.728112] workqueue knbd0-recv: flags=0x1a [11819.731559] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11819.739967] in-flight: 657:recv_work [11819.746889] workqueue ptlrpc_pinger: flags=0x2 [11819.750283] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11819.760173] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [11819.765167] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9860s workers=4 idle: 20831 12642 28859 [11819.780655] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11809s workers=2 idle: 659 [11820.059270] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [11820.063149] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11820.107486] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11820.122677] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11820.132035] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11820.136926] RIP: 0010:native_safe_halt+0x2e/0x50 [11820.144425] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11820.163482] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11820.167816] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [11820.175735] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [11820.185108] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [11820.191309] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [11820.195987] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [11820.202688] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [11820.210941] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11820.214892] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [11820.217934] Call Trace: [11820.219793] [11820.221720] ? show_regs.cold.9+0x22/0x2f [11820.226029] ? watchdog_timer_fn.cold.6+0x62/0x10a [11820.227934] ? __hrtimer_run_queues+0x1c2/0x4d0 [11820.230330] ? watchdog+0x40/0x40 [11820.233021] ? hrtimer_interrupt+0x128/0x2c0 [11820.237974] ? smp_apic_timer_interrupt+0xa6/0x270 [11820.242398] ? apic_timer_interrupt+0xf/0x20 [11820.247114] [11820.250166] ? native_safe_halt+0x2e/0x50 [11820.255335] kvm_wait+0xcd/0xe0 [11820.256499] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11820.260236] do_raw_spin_lock+0x177/0x190 [11820.261728] _raw_spin_lock+0x21/0x30 [11820.264780] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11820.270030] process_one_work+0x2c8/0x700 [11820.278803] worker_thread+0x296/0x6e0 [11820.282532] ? rescuer_thread+0x570/0x570 [11820.288916] kthread+0x1d1/0x200 [11820.291714] ? set_kthread_struct+0x70/0x70 [11820.297680] ret_from_fork+0x1f/0x30 [11832.058213] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [11832.061422] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11832.098177] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11832.104411] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11832.110364] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11832.119845] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11832.142696] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11832.153271] RAX: 0000000000007b88 RBX: ffff96848811cae0 RCX: 0000000000000001 [11832.160016] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [11832.164352] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [11832.170751] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [11832.174228] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [11832.182390] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [11832.189611] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11832.192743] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [11832.197818] Call Trace: [11832.201314] [11832.205597] ? show_regs.cold.9+0x22/0x2f [11832.208747] ? watchdog_timer_fn.cold.6+0x62/0x10a [11832.212566] ? __hrtimer_run_queues+0x1c2/0x4d0 [11832.214197] ? watchdog+0x40/0x40 [11832.216723] ? hrtimer_interrupt+0x128/0x2c0 [11832.220368] ? smp_apic_timer_interrupt+0xa6/0x270 [11832.223273] ? apic_timer_interrupt+0xf/0x20 [11832.225694] [11832.227289] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11832.230344] do_raw_spin_lock+0x177/0x190 [11832.232599] _raw_spin_lock+0x21/0x30 [11832.234381] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11832.237712] ? do_raw_spin_unlock+0x75/0x190 [11832.240417] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11832.243124] ? do_wait_intr+0xf0/0xf0 [11832.246597] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11832.249454] mdc_close+0x53b/0xd70 [mdc] [11832.251782] lmv_close+0x18d/0x490 [lmv] [11832.254465] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [11832.258076] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11832.261371] ll_md_real_close+0xa5/0x2a0 [lustre] [11832.263748] ll_md_close+0x274/0x800 [lustre] [11832.267223] ll_file_release+0x128/0xc50 [lustre] [11832.271598] __fput+0x10d/0x440 [11832.273025] ____fput+0x12/0x20 [11832.274317] task_work_run+0xc6/0x110 [11832.276365] exit_to_usermode_loop+0x1dd/0x1f0 [11832.279175] do_syscall_64+0x3d6/0x440 [11832.280498] entry_SYSCALL_64_after_hwframe+0x49/0xae [11832.285961] RIP: 0033:0x7ff54af45cd8 [11832.290212] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11832.303406] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11832.307688] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [11832.316523] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [11832.322513] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [11832.332027] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [11832.340544] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11844.055020] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [11844.057294] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11844.060673] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [11844.072510] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11844.079315] Modules linked in: [11844.085637] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11844.085649] RIP: 0010:native_safe_halt+0x2e/0x50 [11844.085693] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11844.085697] RSP: 0000:ffffaf9fc1b97bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11844.085703] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [11844.085705] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff96848811cae0 [11844.085708] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [11844.085710] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [11844.085713] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11844.085716] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [11844.085720] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11844.085722] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [11844.085744] Call Trace: [11844.085762] [11844.085769] ? show_regs.cold.9+0x22/0x2f [11844.086165] ? watchdog_timer_fn.cold.6+0x62/0x10a [11844.086172] ? __hrtimer_run_queues+0x1c2/0x4d0 [11844.086177] ? watchdog+0x40/0x40 [11844.086180] ? hrtimer_interrupt+0x128/0x2c0 [11844.086184] ? smp_apic_timer_interrupt+0xa6/0x270 [11844.086191] ? apic_timer_interrupt+0xf/0x20 [11844.086195] [11844.086199] ? native_safe_halt+0x2e/0x50 [11844.086203] kvm_wait+0xcd/0xe0 [11844.086213] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11844.086221] do_raw_spin_lock+0x177/0x190 [11844.086226] _raw_spin_lock+0x21/0x30 [11844.086233] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11844.086561] obd_mod_free+0x46/0x140 [ptlrpc] [11844.087262] mdc_commit_open+0x91/0xb0 [mdc] [11844.087299] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11844.087667] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11844.088332] after_reply+0xe2c/0x1780 [ptlrpc] [11844.093082] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11844.093156] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11844.093164] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11844.093166] RIP: 0010:native_safe_halt+0x2e/0x50 [11844.093181] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11844.093185] RSP: 0000:ffffaf9fc15234f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11844.093189] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [11844.093192] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c21b4154 [11844.093194] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [11844.093196] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [11844.093198] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [11844.093201] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [11844.093204] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11844.093207] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [11844.093214] Call Trace: [11844.093220] [11844.093226] ? show_regs.cold.9+0x22/0x2f [11844.093236] ? watchdog_timer_fn.cold.6+0x62/0x10a [11844.093242] ? __hrtimer_run_queues+0x1c2/0x4d0 [11844.093246] ? watchdog+0x40/0x40 [11844.093250] ? hrtimer_interrupt+0x128/0x2c0 [11844.093255] ? smp_apic_timer_interrupt+0xa6/0x270 [11844.093260] ? apic_timer_interrupt+0xf/0x20 [11844.093265] [11844.093267] ? native_safe_halt+0x2e/0x50 [11844.093272] kvm_wait+0xcd/0xe0 [11844.093279] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11844.093287] do_raw_spin_lock+0x177/0x190 [11844.093292] _raw_spin_lock+0x21/0x30 [11844.093297] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11844.093691] ? do_raw_spin_unlock+0x75/0x190 [11844.093697] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11844.109088] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11844.112575] ? do_wait_intr+0xf0/0xf0 [11844.112666] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11844.113216] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11844.130504] ptlrpcd+0xa82/0xe40 [ptlrpc] [11844.138719] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11844.139271] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11844.143546] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11844.143954] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11844.144047] mdc_intent_lock+0x353/0x7e0 [mdc] [11844.144082] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11844.144185] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11844.150104] ? do_wait_intr+0xf0/0xf0 [11844.150129] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11844.150424] kthread+0x1d1/0x200 [11844.150431] ? set_kthread_struct+0x70/0x70 [11844.150435] ret_from_fork+0x1f/0x30 [11844.655030] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11844.657629] lmv_intent_open+0x338/0x1150 [lmv] [11844.659585] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11844.662161] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11844.664740] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11844.666655] ll_intent_lock+0xc5/0xaa0 [lustre] [11844.669255] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11844.672095] ll_lookup_it+0xe0e/0x2270 [lustre] [11844.674269] ? do_raw_spin_unlock+0x75/0x190 [11844.676265] ? _raw_spin_unlock+0x12/0x30 [11844.678001] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11844.680865] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11844.686282] ll_atomic_open+0x36a/0xd70 [lustre] [11844.691270] lookup_open+0x356/0xb40 [11844.695731] do_last+0x313/0xfc0 [11844.697883] ? nd_jump_root+0xe5/0x160 [11844.699935] ? path_init+0x437/0x520 [11844.705408] path_openat+0xf7/0x500 [11844.706757] ? do_raw_spin_unlock+0x75/0x190 [11844.708260] do_filp_open+0x99/0x140 [11844.713259] ? getname_flags+0x6e/0x330 [11844.717559] ? __check_object_size+0xff/0x256 [11844.722568] ? do_raw_spin_unlock+0x75/0x190 [11844.729794] ? _raw_spin_unlock+0x12/0x30 [11844.731782] do_sys_openat2+0x2b4/0x410 [11844.733067] do_sys_open+0x73/0xa0 [11844.736161] __x64_sys_openat+0x24/0x30 [11844.738144] do_syscall_64+0xc1/0x440 [11844.743523] entry_SYSCALL_64_after_hwframe+0x49/0xae [11844.745386] RIP: 0033:0x7ff54af45332 [11844.748779] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11844.759086] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11844.763471] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [11844.767068] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [11844.771604] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [11844.780516] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [11844.788582] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [11848.059614] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [11848.063889] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11848.105051] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11848.113739] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11848.121909] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11848.127075] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11848.129137] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11848.141315] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11848.145917] RAX: 0000000000007c36 RBX: ffff96848811cae0 RCX: 0000000000000001 [11848.154312] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [11848.160117] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [11848.164370] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [11848.178745] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [11848.192136] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [11848.194249] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11848.201201] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [11848.210214] Call Trace: [11848.211279] [11848.212136] ? show_regs.cold.9+0x22/0x2f [11848.222152] ? watchdog_timer_fn.cold.6+0x62/0x10a [11848.223973] ? __hrtimer_run_queues+0x1c2/0x4d0 [11848.230482] ? watchdog+0x40/0x40 [11848.233820] ? hrtimer_interrupt+0x128/0x2c0 [11848.236381] ? smp_apic_timer_interrupt+0xa6/0x270 [11848.240732] ? apic_timer_interrupt+0xf/0x20 [11848.242551] [11848.243487] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11848.250889] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11848.255890] do_raw_spin_lock+0x177/0x190 [11848.261205] _raw_spin_lock+0x21/0x30 [11848.266688] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11848.273420] process_one_work+0x2c8/0x700 [11848.277061] worker_thread+0x296/0x6e0 [11848.281168] ? rescuer_thread+0x570/0x570 [11848.284238] kthread+0x1d1/0x200 [11848.286126] ? set_kthread_struct+0x70/0x70 [11848.290881] ret_from_fork+0x1f/0x30 [11850.207112] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9891s! [11850.212579] Showing busy workqueues and worker pools: [11850.215197] workqueue events: flags=0x0 [11850.217734] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11850.225107] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [11850.237176] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [11850.247798] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [11850.267118] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11850.277825] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [11850.309407] workqueue events_long: flags=0x0 [11850.318381] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11850.328866] pending: lu_objects_flush [obdclass] [11850.334593] workqueue events_unbound: flags=0x2 [11850.339135] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [11850.346596] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11850.355644] workqueue events_power_efficient: flags=0x80 [11850.358834] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11850.368706] pending: neigh_periodic_work [11850.373237] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11850.376703] pending: do_cache_clean, check_lifetime [11850.380385] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11850.386240] pending: neigh_periodic_work [11850.390598] workqueue writeback: flags=0x4a [11850.393223] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11850.401294] pending: wb_workfn, wb_workfn [11850.408226] workqueue xprtiod: flags=0xa [11850.412095] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11850.418750] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [11850.426293] workqueue ipv6_addrconf: flags=0x40008 [11850.430166] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11850.436660] pending: addrconf_verify_work [11850.445916] workqueue knbd0-recv: flags=0x1a [11850.447742] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11850.456862] in-flight: 657:recv_work [11850.462894] workqueue ptlrpc_pinger: flags=0x2 [11850.472387] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11850.480766] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [11850.495220] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9891s workers=4 idle: 20831 12642 28859 [11850.509447] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11840s workers=2 idle: 659 [11860.058024] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [11860.062944] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11860.114113] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11860.130070] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11860.138120] RIP: 0010:native_safe_halt+0x2e/0x50 [11860.145696] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11860.165847] RSP: 0018:ffffaf9fc1543ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11860.169655] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [11860.175714] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [11860.179313] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [11860.188650] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [11860.198115] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [11860.203064] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [11860.213340] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11860.217629] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [11860.221396] Call Trace: [11860.222567] [11860.226002] ? show_regs.cold.9+0x22/0x2f [11860.228022] ? watchdog_timer_fn.cold.6+0x62/0x10a [11860.230367] ? __hrtimer_run_queues+0x1c2/0x4d0 [11860.232145] ? watchdog+0x40/0x40 [11860.233845] ? hrtimer_interrupt+0x128/0x2c0 [11860.238074] ? smp_apic_timer_interrupt+0xa6/0x270 [11860.240092] ? apic_timer_interrupt+0xf/0x20 [11860.241637] [11860.242797] ? native_safe_halt+0x2e/0x50 [11860.244652] kvm_wait+0xcd/0xe0 [11860.248537] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11860.251107] do_raw_spin_lock+0x177/0x190 [11860.252767] _raw_spin_lock+0x21/0x30 [11860.254500] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11860.256928] ? do_raw_spin_unlock+0x75/0x190 [11860.258963] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11860.261352] ? do_wait_intr+0xf0/0xf0 [11860.263002] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11860.267883] mdc_close+0x53b/0xd70 [mdc] [11860.269718] lmv_close+0x18d/0x490 [lmv] [11860.271581] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [11860.274156] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11860.276732] ll_md_real_close+0xa5/0x2a0 [lustre] [11860.281819] ll_md_close+0x274/0x800 [lustre] [11860.284689] ll_file_release+0x128/0xc50 [lustre] [11860.286809] __fput+0x10d/0x440 [11860.288597] ____fput+0x12/0x20 [11860.289955] task_work_run+0xc6/0x110 [11860.291722] exit_to_usermode_loop+0x1dd/0x1f0 [11860.294073] do_syscall_64+0x3d6/0x440 [11860.297765] entry_SYSCALL_64_after_hwframe+0x49/0xae [11860.300113] RIP: 0033:0x7ff54af45cd8 [11860.302701] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11860.323047] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11860.331043] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [11860.341064] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [11860.348048] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [11860.355973] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [11860.361109] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11872.060059] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [11872.065024] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11872.133430] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11872.146919] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11872.157415] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11872.167300] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11872.185989] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11872.198830] RAX: 0000000000007333 RBX: ffff96848811cae0 RCX: 0000000000000001 [11872.204463] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [11872.213355] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [11872.220235] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [11872.225376] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [11872.228438] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [11872.232526] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11872.234712] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [11872.238145] Call Trace: [11872.239242] [11872.240226] ? show_regs.cold.9+0x22/0x2f [11872.242304] ? watchdog_timer_fn.cold.6+0x62/0x10a [11872.244497] ? __hrtimer_run_queues+0x1c2/0x4d0 [11872.246510] ? watchdog+0x40/0x40 [11872.247651] ? hrtimer_interrupt+0x128/0x2c0 [11872.249586] ? smp_apic_timer_interrupt+0xa6/0x270 [11872.251697] ? apic_timer_interrupt+0xf/0x20 [11872.255561] [11872.257388] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11872.259190] do_raw_spin_lock+0x177/0x190 [11872.260963] _raw_spin_lock+0x21/0x30 [11872.261861] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11872.265603] ? do_raw_spin_unlock+0x75/0x190 [11872.270934] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11872.278206] ? do_wait_intr+0xf0/0xf0 [11872.279958] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11872.283703] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11872.293422] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11872.298025] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11872.301980] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11872.304257] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11872.305712] mdc_intent_lock+0x353/0x7e0 [mdc] [11872.309656] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11872.313645] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11872.319646] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11872.322449] lmv_intent_open+0x338/0x1150 [lmv] [11872.331353] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11872.333191] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11872.336588] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11872.341317] ll_intent_lock+0xc5/0xaa0 [lustre] [11872.347443] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11872.349163] ll_lookup_it+0xe0e/0x2270 [lustre] [11872.357478] ? do_raw_spin_unlock+0x75/0x190 [11872.363904] ? _raw_spin_unlock+0x12/0x30 [11872.378619] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11872.387842] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11872.394712] ll_atomic_open+0x36a/0xd70 [lustre] [11872.397584] lookup_open+0x356/0xb40 [11872.398890] do_last+0x313/0xfc0 [11872.407452] ? nd_jump_root+0xe5/0x160 [11872.413657] ? path_init+0x437/0x520 [11872.415748] path_openat+0xf7/0x500 [11872.417492] ? do_raw_spin_unlock+0x75/0x190 [11872.418975] do_filp_open+0x99/0x140 [11872.432625] ? getname_flags+0x6e/0x330 [11872.434048] ? __check_object_size+0xff/0x256 [11872.446191] ? do_raw_spin_unlock+0x75/0x190 [11872.447726] ? _raw_spin_unlock+0x12/0x30 [11872.449145] do_sys_openat2+0x2b4/0x410 [11872.458724] do_sys_open+0x73/0xa0 [11872.461897] __x64_sys_openat+0x24/0x30 [11872.467607] do_syscall_64+0xc1/0x440 [11872.468979] entry_SYSCALL_64_after_hwframe+0x49/0xae [11872.474477] RIP: 0033:0x7ff54af45332 [11872.475826] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11872.493356] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11872.501790] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [11872.509071] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [11872.518329] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [11872.524348] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [11872.529599] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [11876.057738] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [11876.068172] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [11876.068786] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11876.068857] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11876.068863] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11876.068865] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11876.068879] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11876.068884] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11876.068889] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [11876.068892] RDX: 0000000000003437 RSI: 0000000000000003 RDI: ffff96848811cae0 [11876.068894] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [11876.068896] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [11876.068899] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11876.068902] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [11876.068905] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11876.068908] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [11876.069911] Call Trace: [11876.069934] [11876.069940] ? show_regs.cold.9+0x22/0x2f [11876.069954] ? watchdog_timer_fn.cold.6+0x62/0x10a [11876.069960] ? __hrtimer_run_queues+0x1c2/0x4d0 [11876.069965] ? watchdog+0x40/0x40 [11876.069970] ? hrtimer_interrupt+0x128/0x2c0 [11876.069974] ? smp_apic_timer_interrupt+0xa6/0x270 [11876.069984] ? apic_timer_interrupt+0xf/0x20 [11876.069988] [11876.069992] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11876.069998] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11876.070003] do_raw_spin_lock+0x177/0x190 [11876.070007] _raw_spin_lock+0x21/0x30 [11876.070014] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11876.070462] obd_mod_free+0x46/0x140 [ptlrpc] [11876.070670] mdc_commit_open+0x91/0xb0 [mdc] [11876.070708] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11876.071340] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11876.071546] after_reply+0xe2c/0x1780 [ptlrpc] [11876.073334] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11876.073535] ptlrpcd+0xa82/0xe40 [ptlrpc] [11876.073754] ? do_wait_intr+0xf0/0xf0 [11876.073765] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11876.074610] kthread+0x1d1/0x200 [11876.074617] ? set_kthread_struct+0x70/0x70 [11876.074621] ret_from_fork+0x1f/0x30 [11876.401714] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11876.434786] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11876.440441] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11876.444470] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11876.447400] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11876.451192] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11876.460008] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11876.463530] RAX: 0000000000002691 RBX: ffff96848811cae0 RCX: 0000000000000001 [11876.466546] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [11876.469536] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [11876.472586] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [11876.475895] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [11876.479043] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [11876.482909] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11876.486019] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [11876.489270] Call Trace: [11876.490438] [11876.491359] ? show_regs.cold.9+0x22/0x2f [11876.493200] ? watchdog_timer_fn.cold.6+0x62/0x10a [11876.495332] ? __hrtimer_run_queues+0x1c2/0x4d0 [11876.497934] ? watchdog+0x40/0x40 [11876.499430] ? hrtimer_interrupt+0x128/0x2c0 [11876.500909] ? smp_apic_timer_interrupt+0xa6/0x270 [11876.507158] ? apic_timer_interrupt+0xf/0x20 [11876.509886] [11876.511026] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11876.513559] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11876.516414] do_raw_spin_lock+0x177/0x190 [11876.518392] _raw_spin_lock+0x21/0x30 [11876.520311] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11876.525732] process_one_work+0x2c8/0x700 [11876.529248] worker_thread+0x296/0x6e0 [11876.531233] ? rescuer_thread+0x570/0x570 [11876.536854] kthread+0x1d1/0x200 [11876.540997] ? set_kthread_struct+0x70/0x70 [11876.545053] ret_from_fork+0x1f/0x30 [11880.927166] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9922s! [11880.930712] Showing busy workqueues and worker pools: [11880.933645] workqueue events: flags=0x0 [11880.935322] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11880.937842] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [11880.942321] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [11880.945115] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [11880.949225] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11880.952309] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [11880.955348] workqueue events_long: flags=0x0 [11880.957448] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11880.960492] pending: lu_objects_flush [obdclass] [11880.963492] workqueue events_unbound: flags=0x2 [11880.965404] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [11880.968316] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11880.974133] workqueue events_power_efficient: flags=0x80 [11880.976302] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11880.979525] pending: neigh_periodic_work [11880.981298] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11880.983469] pending: do_cache_clean, check_lifetime [11880.985521] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11880.987875] pending: neigh_periodic_work [11880.989725] workqueue writeback: flags=0x4a [11880.991835] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11880.994373] pending: wb_workfn, wb_workfn [11880.995819] workqueue xprtiod: flags=0xa [11880.997400] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11880.999481] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [11881.001734] workqueue ipv6_addrconf: flags=0x40008 [11881.003655] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11881.005990] pending: addrconf_verify_work [11881.007531] workqueue knbd0-recv: flags=0x1a [11881.008876] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11881.011106] in-flight: 657:recv_work [11881.012697] workqueue ptlrpc_pinger: flags=0x2 [11881.014391] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11881.016544] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [11881.018882] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9922s workers=4 idle: 20831 12642 28859 [11881.021973] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11871s workers=2 idle: 659 [11888.058007] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [11888.061064] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11888.088884] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11888.093907] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11888.099269] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11888.102084] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11888.109500] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11888.115853] RAX: 0000000000007ae4 RBX: ffff96848811cae0 RCX: 0000000000000001 [11888.119254] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [11888.124144] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [11888.127845] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [11888.130966] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [11888.134162] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [11888.137316] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11888.139198] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [11888.143682] Call Trace: [11888.144727] [11888.145894] ? show_regs.cold.9+0x22/0x2f [11888.147790] ? watchdog_timer_fn.cold.6+0x62/0x10a [11888.153779] ? __hrtimer_run_queues+0x1c2/0x4d0 [11888.156009] ? watchdog+0x40/0x40 [11888.158204] ? hrtimer_interrupt+0x128/0x2c0 [11888.159756] ? smp_apic_timer_interrupt+0xa6/0x270 [11888.162439] ? apic_timer_interrupt+0xf/0x20 [11888.168809] [11888.169869] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11888.173230] do_raw_spin_lock+0x177/0x190 [11888.174958] _raw_spin_lock+0x21/0x30 [11888.177206] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11888.179157] ? do_raw_spin_unlock+0x75/0x190 [11888.182789] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11888.185311] ? do_wait_intr+0xf0/0xf0 [11888.186581] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11888.189020] mdc_close+0x53b/0xd70 [mdc] [11888.190316] lmv_close+0x18d/0x490 [lmv] [11888.191620] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [11888.197128] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11888.200161] ll_md_real_close+0xa5/0x2a0 [lustre] [11888.203362] ll_md_close+0x274/0x800 [lustre] [11888.205465] ll_file_release+0x128/0xc50 [lustre] [11888.207870] __fput+0x10d/0x440 [11888.209181] ____fput+0x12/0x20 [11888.210451] task_work_run+0xc6/0x110 [11888.212371] exit_to_usermode_loop+0x1dd/0x1f0 [11888.214407] do_syscall_64+0x3d6/0x440 [11888.217664] entry_SYSCALL_64_after_hwframe+0x49/0xae [11888.220705] RIP: 0033:0x7ff54af45cd8 [11888.222774] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11888.232893] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11888.237977] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [11888.242241] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [11888.245827] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [11888.249341] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [11888.252446] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11900.063487] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [11900.075262] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11900.131132] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11900.150567] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11900.165420] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11900.172556] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11900.202338] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11900.208207] RAX: 0000000000007828 RBX: ffff96848811cae0 RCX: 0000000000000001 [11900.212483] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [11900.224179] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [11900.231304] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [11900.237721] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [11900.242590] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [11900.251382] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11900.259847] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [11900.265080] Call Trace: [11900.266453] [11900.269473] ? show_regs.cold.9+0x22/0x2f [11900.273553] ? watchdog_timer_fn.cold.6+0x62/0x10a [11900.275751] ? __hrtimer_run_queues+0x1c2/0x4d0 [11900.281294] ? watchdog+0x40/0x40 [11900.294078] ? hrtimer_interrupt+0x128/0x2c0 [11900.302489] ? smp_apic_timer_interrupt+0xa6/0x270 [11900.311552] ? apic_timer_interrupt+0xf/0x20 [11900.321845] [11900.324414] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11900.335734] do_raw_spin_lock+0x177/0x190 [11900.338636] _raw_spin_lock+0x21/0x30 [11900.341534] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11900.347370] ? do_raw_spin_unlock+0x75/0x190 [11900.351173] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11900.354532] ? do_wait_intr+0xf0/0xf0 [11900.357932] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11900.364412] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11900.375209] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11900.386266] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11900.390256] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11900.394217] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11900.401780] mdc_intent_lock+0x353/0x7e0 [mdc] [11900.407428] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11900.415615] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11900.423665] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11900.428271] lmv_intent_open+0x338/0x1150 [lmv] [11900.432003] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11900.442702] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11900.447668] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11900.453447] ll_intent_lock+0xc5/0xaa0 [lustre] [11900.457427] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11900.463297] ll_lookup_it+0xe0e/0x2270 [lustre] [11900.466395] ? do_raw_spin_unlock+0x75/0x190 [11900.469215] ? _raw_spin_unlock+0x12/0x30 [11900.475189] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11900.480637] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11900.483925] ll_atomic_open+0x36a/0xd70 [lustre] [11900.488575] lookup_open+0x356/0xb40 [11900.490489] do_last+0x313/0xfc0 [11900.495579] ? nd_jump_root+0xe5/0x160 [11900.503967] ? path_init+0x437/0x520 [11900.507899] path_openat+0xf7/0x500 [11900.509475] ? do_raw_spin_unlock+0x75/0x190 [11900.516274] do_filp_open+0x99/0x140 [11900.518339] ? getname_flags+0x6e/0x330 [11900.523749] ? __check_object_size+0xff/0x256 [11900.528334] ? do_raw_spin_unlock+0x75/0x190 [11900.531237] ? _raw_spin_unlock+0x12/0x30 [11900.533745] do_sys_openat2+0x2b4/0x410 [11900.536280] do_sys_open+0x73/0xa0 [11900.538749] __x64_sys_openat+0x24/0x30 [11900.541421] do_syscall_64+0xc1/0x440 [11900.544157] entry_SYSCALL_64_after_hwframe+0x49/0xae [11900.547563] RIP: 0033:0x7ff54af45332 [11900.550288] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11900.559729] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11900.564264] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [11900.568028] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [11900.572661] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [11900.576731] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [11900.582725] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [11904.066569] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [11904.070567] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11904.112411] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11904.124543] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11904.130417] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11904.135891] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11904.142068] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11904.173759] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11904.181674] RAX: 000000000000418b RBX: ffff96848811cae0 RCX: 0000000000000001 [11904.193582] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [11904.204609] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [11904.218786] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [11904.230757] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [11904.234965] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [11904.239884] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11904.242331] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [11904.245657] Call Trace: [11904.247085] [11904.249764] ? show_regs.cold.9+0x22/0x2f [11904.252417] ? watchdog_timer_fn.cold.6+0x62/0x10a [11904.254962] ? __hrtimer_run_queues+0x1c2/0x4d0 [11904.257880] ? watchdog+0x40/0x40 [11904.259345] ? hrtimer_interrupt+0x128/0x2c0 [11904.262722] ? smp_apic_timer_interrupt+0xa6/0x270 [11904.264760] ? apic_timer_interrupt+0xf/0x20 [11904.268304] [11904.269412] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11904.273683] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11904.278047] do_raw_spin_lock+0x177/0x190 [11904.282019] _raw_spin_lock+0x21/0x30 [11904.284262] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11904.292449] process_one_work+0x2c8/0x700 [11904.299658] worker_thread+0x296/0x6e0 [11904.300853] ? rescuer_thread+0x570/0x570 [11904.310313] kthread+0x1d1/0x200 [11904.311421] ? set_kthread_struct+0x70/0x70 [11904.313578] ret_from_fork+0x1f/0x30 [11908.055183] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [11908.058675] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11908.090573] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11908.096493] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11908.101185] RIP: 0010:native_safe_halt+0x2e/0x50 [11908.103428] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11908.111303] RSP: 0000:ffffaf9fc1b97bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11908.113435] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [11908.117593] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff96848811cae0 [11908.119660] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [11908.122807] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [11908.125401] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11908.132929] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [11908.137157] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11908.149385] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [11908.152932] Call Trace: [11908.154161] [11908.155240] ? show_regs.cold.9+0x22/0x2f [11908.156780] ? watchdog_timer_fn.cold.6+0x62/0x10a [11908.159413] ? __hrtimer_run_queues+0x1c2/0x4d0 [11908.161350] ? watchdog+0x40/0x40 [11908.163341] ? hrtimer_interrupt+0x128/0x2c0 [11908.165386] ? smp_apic_timer_interrupt+0xa6/0x270 [11908.167549] ? apic_timer_interrupt+0xf/0x20 [11908.170014] [11908.171227] ? native_safe_halt+0x2e/0x50 [11908.174335] kvm_wait+0xcd/0xe0 [11908.176763] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11908.180399] do_raw_spin_lock+0x177/0x190 [11908.185006] _raw_spin_lock+0x21/0x30 [11908.187079] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11908.191824] obd_mod_free+0x46/0x140 [ptlrpc] [11908.194691] mdc_commit_open+0x91/0xb0 [mdc] [11908.196358] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11908.205045] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11908.209287] after_reply+0xe2c/0x1780 [ptlrpc] [11908.213338] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11908.217372] ptlrpcd+0xa82/0xe40 [ptlrpc] [11908.220488] ? do_wait_intr+0xf0/0xf0 [11908.222175] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11908.226543] kthread+0x1d1/0x200 [11908.228780] ? set_kthread_struct+0x70/0x70 [11908.230442] ret_from_fork+0x1f/0x30 [11911.135953] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9952s! [11911.144848] Showing busy workqueues and worker pools: [11911.152272] workqueue events: flags=0x0 [11911.154506] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11911.157836] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [11911.162072] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [11911.172054] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [11911.183952] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11911.190324] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [11911.195602] workqueue events_long: flags=0x0 [11911.197458] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11911.202767] pending: lu_objects_flush [obdclass] [11911.206013] workqueue events_unbound: flags=0x2 [11911.208414] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [11911.211641] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11911.217388] workqueue events_power_efficient: flags=0x80 [11911.219163] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11911.221497] pending: neigh_periodic_work [11911.225520] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11911.229464] pending: do_cache_clean, check_lifetime [11911.235878] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11911.239297] pending: neigh_periodic_work [11911.244695] workqueue writeback: flags=0x4a [11911.251393] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11911.253729] pending: wb_workfn, wb_workfn [11911.259157] workqueue xprtiod: flags=0xa [11911.260769] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11911.264298] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [11911.274308] workqueue ipv6_addrconf: flags=0x40008 [11911.278259] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11911.281980] pending: addrconf_verify_work [11911.284159] workqueue knbd0-recv: flags=0x1a [11911.286664] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11911.290430] in-flight: 657:recv_work [11911.292546] workqueue ptlrpc_pinger: flags=0x2 [11911.295918] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11911.306311] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [11911.310595] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9952s workers=4 idle: 20831 12642 28859 [11911.314483] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11901s workers=2 idle: 659 [11914.120416] rcu: INFO: rcu_sched self-detected stall on CPU [11914.122562] rcu: 0-...!: (5272941 ticks this GP) idle=98a/1/0x4000000000000002 softirq=125154/125154 fqs=959 [11914.126674] (t=9960280 jiffies g=253585 q=6065) [11914.128819] rcu: rcu_sched kthread starved for 9955240 jiffies! g253585 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=3 [11914.133026] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [11914.136603] rcu: RCU grace-period kthread stack dump: [11914.138947] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [11914.144337] Call Trace: [11914.145400] __schedule+0x351/0xcb0 [11914.146889] schedule+0xc0/0x180 [11914.148850] schedule_timeout+0xb4/0x190 [11914.150923] ? do_raw_spin_unlock+0x75/0x190 [11914.152244] ? __next_timer_interrupt+0x160/0x160 [11914.153640] ? prepare_to_swait_event+0xf3/0x140 [11914.154904] rcu_gp_kthread+0x8dd/0x15b0 [11914.156286] ? force_qs_rnp+0x2c0/0x2c0 [11914.157513] kthread+0x1d1/0x200 [11914.158648] ? set_kthread_struct+0x70/0x70 [11914.160004] ret_from_fork+0x1f/0x30 [11914.161112] rcu: Stack dump where RCU GP kthread last ran: [11914.162612] Sending NMI from CPU 0 to CPUs 3: [11914.163961] NMI backtrace for cpu 3 skipped: idling at native_safe_halt+0x2e/0x50 [11914.164745] NMI backtrace for cpu 0 [11914.169190] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11914.172619] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11914.174914] Call Trace: [11914.175493] [11914.176131] ? dump_stack+0xbb/0x10e [11914.177656] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [11914.179827] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [11914.181561] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [11914.183756] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [11914.186400] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [11914.188566] ? rcu_dump_cpu_stacks+0x11b/0x171 [11914.190905] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [11914.193109] ? update_process_times+0x6d/0xc0 [11914.194988] ? tick_sched_handle.isra.18+0x27/0x90 [11914.197133] ? tick_sched_timer+0x53/0xb0 [11914.198798] ? __hrtimer_run_queues+0x1c2/0x4d0 [11914.200741] ? tick_sched_handle.isra.18+0x90/0x90 [11914.202864] ? hrtimer_interrupt+0x128/0x2c0 [11914.204650] ? smp_apic_timer_interrupt+0xa6/0x270 [11914.206810] ? apic_timer_interrupt+0xf/0x20 [11914.208893] [11914.209646] ? native_safe_halt+0x2e/0x50 [11914.211430] ? kvm_wait+0xcd/0xe0 [11914.213267] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11914.216333] ? do_raw_spin_lock+0x177/0x190 [11914.218638] ? _raw_spin_lock+0x21/0x30 [11914.220519] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11914.225566] ? obd_mod_free+0x46/0x140 [ptlrpc] [11914.228302] ? mdc_commit_open+0x91/0xb0 [mdc] [11914.229938] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11914.232178] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11914.235295] ? after_reply+0xe2c/0x1780 [ptlrpc] [11914.237604] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11914.240360] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [11914.242554] ? do_wait_intr+0xf0/0xf0 [11914.245301] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11914.247538] ? kthread+0x1d1/0x200 [11914.251373] ? set_kthread_struct+0x70/0x70 [11914.253206] ? ret_from_fork+0x1f/0x30 [11916.058310] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [11916.065853] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11916.098146] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11916.105442] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11916.112875] RIP: 0010:native_safe_halt+0x2e/0x50 [11916.114775] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11916.124555] RSP: 0018:ffffaf9fc1543ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11916.130072] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [11916.135417] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c20b4154 [11916.143043] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [11916.145309] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [11916.147160] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [11916.152110] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [11916.155826] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11916.159132] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [11916.163956] Call Trace: [11916.165461] [11916.166317] ? show_regs.cold.9+0x22/0x2f [11916.168073] ? watchdog_timer_fn.cold.6+0x62/0x10a [11916.170314] ? __hrtimer_run_queues+0x1c2/0x4d0 [11916.172409] ? watchdog+0x40/0x40 [11916.173766] ? hrtimer_interrupt+0x128/0x2c0 [11916.175376] ? smp_apic_timer_interrupt+0xa6/0x270 [11916.177960] ? apic_timer_interrupt+0xf/0x20 [11916.179561] [11916.180676] ? native_safe_halt+0x2e/0x50 [11916.182336] kvm_wait+0xcd/0xe0 [11916.183768] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11916.185572] do_raw_spin_lock+0x177/0x190 [11916.186749] _raw_spin_lock+0x21/0x30 [11916.188434] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11916.191319] ? do_raw_spin_unlock+0x75/0x190 [11916.193422] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11916.196092] ? do_wait_intr+0xf0/0xf0 [11916.197956] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11916.200746] mdc_close+0x53b/0xd70 [mdc] [11916.202599] lmv_close+0x18d/0x490 [lmv] [11916.204576] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [11916.207398] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11916.210603] ll_md_real_close+0xa5/0x2a0 [lustre] [11916.213576] ll_md_close+0x274/0x800 [lustre] [11916.217194] ll_file_release+0x128/0xc50 [lustre] [11916.219492] __fput+0x10d/0x440 [11916.220872] ____fput+0x12/0x20 [11916.221987] task_work_run+0xc6/0x110 [11916.223386] exit_to_usermode_loop+0x1dd/0x1f0 [11916.226101] do_syscall_64+0x3d6/0x440 [11916.227185] entry_SYSCALL_64_after_hwframe+0x49/0xae [11916.229311] RIP: 0033:0x7ff54af45cd8 [11916.231126] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11916.240270] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11916.244351] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [11916.249050] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [11916.255341] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [11916.259212] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [11916.262769] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11932.059781] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [11932.072097] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11932.120133] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11932.131204] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11932.135377] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11932.138437] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11932.141350] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11932.153366] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11932.158688] RAX: 000000000000570a RBX: ffff96848811cae0 RCX: 0000000000000001 [11932.168266] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [11932.177760] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [11932.185148] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [11932.191244] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [11932.197465] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [11932.201141] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11932.211070] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [11932.216114] Call Trace: [11932.217959] [11932.218783] ? show_regs.cold.9+0x22/0x2f [11932.220950] ? watchdog_timer_fn.cold.6+0x62/0x10a [11932.223019] ? __hrtimer_run_queues+0x1c2/0x4d0 [11932.226483] ? watchdog+0x40/0x40 [11932.229715] ? hrtimer_interrupt+0x128/0x2c0 [11932.232369] ? smp_apic_timer_interrupt+0xa6/0x270 [11932.238119] ? apic_timer_interrupt+0xf/0x20 [11932.239500] [11932.240178] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11932.249868] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11932.259787] do_raw_spin_lock+0x177/0x190 [11932.262448] _raw_spin_lock+0x21/0x30 [11932.265258] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11932.269318] process_one_work+0x2c8/0x700 [11932.272243] worker_thread+0x296/0x6e0 [11932.274233] ? rescuer_thread+0x570/0x570 [11932.280700] kthread+0x1d1/0x200 [11932.285439] ? set_kthread_struct+0x70/0x70 [11932.291059] ret_from_fork+0x1f/0x30 [11940.055040] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [11940.058587] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11940.060031] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [11940.070281] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11940.070301] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11940.070305] RIP: 0010:native_safe_halt+0x2e/0x50 [11940.070321] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11940.070326] RSP: 0000:ffffaf9fc1b97bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11940.070333] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [11940.070335] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff96848811cae0 [11940.070337] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [11940.070339] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [11940.070342] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11940.070345] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [11940.070348] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11940.070350] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [11940.070359] Call Trace: [11940.070366] [11940.070374] ? show_regs.cold.9+0x22/0x2f [11940.070386] ? watchdog_timer_fn.cold.6+0x62/0x10a [11940.070394] ? __hrtimer_run_queues+0x1c2/0x4d0 [11940.070398] ? watchdog+0x40/0x40 [11940.070402] ? hrtimer_interrupt+0x128/0x2c0 [11940.070406] ? smp_apic_timer_interrupt+0xa6/0x270 [11940.070418] ? apic_timer_interrupt+0xf/0x20 [11940.070423] [11940.070426] ? native_safe_halt+0x2e/0x50 [11940.070490] kvm_wait+0xcd/0xe0 [11940.070499] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11940.070507] do_raw_spin_lock+0x177/0x190 [11940.070511] _raw_spin_lock+0x21/0x30 [11940.070518] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11940.071088] obd_mod_free+0x46/0x140 [ptlrpc] [11940.071283] mdc_commit_open+0x91/0xb0 [mdc] [11940.071327] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11940.071635] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11940.071820] after_reply+0xe2c/0x1780 [ptlrpc] [11940.072022] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11940.072214] ptlrpcd+0xa82/0xe40 [ptlrpc] [11940.072414] ? do_wait_intr+0xf0/0xf0 [11940.072426] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11940.074917] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11940.074996] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11940.075006] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11940.075009] RIP: 0010:native_safe_halt+0x2e/0x50 [11940.075027] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11940.075032] RSP: 0000:ffffaf9fc15234f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11940.075037] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [11940.075040] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c21b4154 [11940.075042] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [11940.075044] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [11940.075047] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [11940.075051] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [11940.075055] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11940.075057] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [11940.075067] Call Trace: [11940.075075] [11940.075083] ? show_regs.cold.9+0x22/0x2f [11940.075092] ? watchdog_timer_fn.cold.6+0x62/0x10a [11940.075099] ? __hrtimer_run_queues+0x1c2/0x4d0 [11940.075102] ? watchdog+0x40/0x40 [11940.075105] ? hrtimer_interrupt+0x128/0x2c0 [11940.075108] ? smp_apic_timer_interrupt+0xa6/0x270 [11940.075115] ? apic_timer_interrupt+0xf/0x20 [11940.075120] [11940.075124] ? native_safe_halt+0x2e/0x50 [11940.075128] kvm_wait+0xcd/0xe0 [11940.075138] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11940.075148] do_raw_spin_lock+0x177/0x190 [11940.075153] _raw_spin_lock+0x21/0x30 [11940.075157] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11940.080330] kthread+0x1d1/0x200 [11940.080354] ? set_kthread_struct+0x70/0x70 [11940.080358] ret_from_fork+0x1f/0x30 [11940.218352] ? do_raw_spin_unlock+0x75/0x190 [11940.219780] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11940.221697] ? do_wait_intr+0xf0/0xf0 [11940.222983] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11940.224978] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11940.226883] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11940.228930] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11940.230775] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11940.232397] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11940.233870] mdc_intent_lock+0x353/0x7e0 [mdc] [11940.235311] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11940.237041] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11940.239587] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11940.241330] lmv_intent_open+0x338/0x1150 [lmv] [11940.243097] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11940.245642] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11940.247527] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11940.249210] ll_intent_lock+0xc5/0xaa0 [lustre] [11940.251233] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11940.253510] ll_lookup_it+0xe0e/0x2270 [lustre] [11940.255331] ? do_raw_spin_unlock+0x75/0x190 [11940.256820] ? _raw_spin_unlock+0x12/0x30 [11940.258260] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11940.260165] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11940.261635] ll_atomic_open+0x36a/0xd70 [lustre] [11940.263323] lookup_open+0x356/0xb40 [11940.265313] do_last+0x313/0xfc0 [11940.266279] ? nd_jump_root+0xe5/0x160 [11940.267422] ? path_init+0x437/0x520 [11940.268516] path_openat+0xf7/0x500 [11940.269796] ? do_raw_spin_unlock+0x75/0x190 [11940.271086] do_filp_open+0x99/0x140 [11940.272333] ? getname_flags+0x6e/0x330 [11940.273594] ? __check_object_size+0xff/0x256 [11940.274938] ? do_raw_spin_unlock+0x75/0x190 [11940.276405] ? _raw_spin_unlock+0x12/0x30 [11940.277641] do_sys_openat2+0x2b4/0x410 [11940.278733] do_sys_open+0x73/0xa0 [11940.279757] __x64_sys_openat+0x24/0x30 [11940.281096] do_syscall_64+0xc1/0x440 [11940.282125] entry_SYSCALL_64_after_hwframe+0x49/0xae [11940.283716] RIP: 0033:0x7ff54af45332 [11940.284559] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11940.290475] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11940.293079] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [11940.295410] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [11940.297912] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [11940.300806] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [11940.303670] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [11941.343057] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9982s! [11941.345950] Showing busy workqueues and worker pools: [11941.348079] workqueue events: flags=0x0 [11941.349585] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11941.352276] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [11941.356281] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [11941.358924] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [11941.363855] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11941.366779] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [11941.369837] workqueue events_long: flags=0x0 [11941.371769] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11941.374173] pending: lu_objects_flush [obdclass] [11941.376464] workqueue events_unbound: flags=0x2 [11941.378542] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [11941.381199] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11941.385300] workqueue events_power_efficient: flags=0x80 [11941.387817] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11941.391127] pending: neigh_periodic_work [11941.394114] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11941.398296] pending: do_cache_clean, check_lifetime [11941.400771] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11941.403904] pending: neigh_periodic_work [11941.405927] workqueue writeback: flags=0x4a [11941.408318] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11941.411828] pending: wb_workfn, wb_workfn [11941.414279] workqueue xprtiod: flags=0xa [11941.416334] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11941.419755] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [11941.423972] workqueue ipv6_addrconf: flags=0x40008 [11941.426949] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11941.433629] pending: addrconf_verify_work [11941.436909] workqueue knbd0-recv: flags=0x1a [11941.440868] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11941.445680] in-flight: 657:recv_work [11941.449642] workqueue ptlrpc_pinger: flags=0x2 [11941.452636] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11941.456663] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [11941.459963] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9982s workers=4 idle: 20831 12642 28859 [11941.463890] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11931s workers=2 idle: 659 [11944.058385] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [11944.062125] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11944.093127] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11944.097590] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11944.104643] RIP: 0010:native_safe_halt+0x2e/0x50 [11944.111938] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11944.125694] RSP: 0018:ffffaf9fc1543ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11944.133658] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [11944.138582] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c20b4154 [11944.143983] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [11944.149293] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [11944.157649] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [11944.163370] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [11944.168309] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11944.171557] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [11944.176143] Call Trace: [11944.177326] [11944.178352] ? show_regs.cold.9+0x22/0x2f [11944.180276] ? watchdog_timer_fn.cold.6+0x62/0x10a [11944.184011] ? __hrtimer_run_queues+0x1c2/0x4d0 [11944.188961] ? watchdog+0x40/0x40 [11944.191059] ? hrtimer_interrupt+0x128/0x2c0 [11944.198684] ? smp_apic_timer_interrupt+0xa6/0x270 [11944.204056] ? apic_timer_interrupt+0xf/0x20 [11944.206016] [11944.211135] ? native_safe_halt+0x2e/0x50 [11944.213104] kvm_wait+0xcd/0xe0 [11944.214708] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11944.217647] do_raw_spin_lock+0x177/0x190 [11944.220048] _raw_spin_lock+0x21/0x30 [11944.222388] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11944.231005] ? do_raw_spin_unlock+0x75/0x190 [11944.235585] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11944.239121] ? do_wait_intr+0xf0/0xf0 [11944.248399] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11944.256917] mdc_close+0x53b/0xd70 [mdc] [11944.267238] lmv_close+0x18d/0x490 [lmv] [11944.271327] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [11944.277488] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11944.280248] ll_md_real_close+0xa5/0x2a0 [lustre] [11944.282787] ll_md_close+0x274/0x800 [lustre] [11944.285532] ll_file_release+0x128/0xc50 [lustre] [11944.287860] __fput+0x10d/0x440 [11944.289377] ____fput+0x12/0x20 [11944.290759] task_work_run+0xc6/0x110 [11944.292578] exit_to_usermode_loop+0x1dd/0x1f0 [11944.294634] do_syscall_64+0x3d6/0x440 [11944.296334] entry_SYSCALL_64_after_hwframe+0x49/0xae [11944.298208] RIP: 0033:0x7ff54af45cd8 [11944.299541] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11944.305368] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11944.308287] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [11944.311362] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [11944.324834] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [11944.334934] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [11944.341341] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11960.063521] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [11960.074917] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11960.128058] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11960.137934] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11960.146894] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11960.154480] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11960.162048] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11960.180179] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11960.184945] RAX: 0000000000003512 RBX: ffff96848811cae0 RCX: 0000000000000001 [11960.190471] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [11960.196213] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [11960.200983] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [11960.205666] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [11960.208748] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [11960.212946] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11960.216345] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [11960.220170] Call Trace: [11960.221066] [11960.221810] ? show_regs.cold.9+0x22/0x2f [11960.223825] ? watchdog_timer_fn.cold.6+0x62/0x10a [11960.227524] ? __hrtimer_run_queues+0x1c2/0x4d0 [11960.230932] ? watchdog+0x40/0x40 [11960.232078] ? hrtimer_interrupt+0x128/0x2c0 [11960.233884] ? smp_apic_timer_interrupt+0xa6/0x270 [11960.237154] ? apic_timer_interrupt+0xf/0x20 [11960.239180] [11960.240089] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11960.245076] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11960.249571] do_raw_spin_lock+0x177/0x190 [11960.251852] _raw_spin_lock+0x21/0x30 [11960.254044] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11960.263446] process_one_work+0x2c8/0x700 [11960.264312] worker_thread+0x296/0x6e0 [11960.274910] ? rescuer_thread+0x570/0x570 [11960.277483] kthread+0x1d1/0x200 [11960.280033] ? set_kthread_struct+0x70/0x70 [11960.282135] ret_from_fork+0x1f/0x30 [11968.055034] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [11968.060056] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [11968.061483] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11968.061966] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11968.061974] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11968.061978] RIP: 0010:native_safe_halt+0x2e/0x50 [11968.061996] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11968.062000] RSP: 0000:ffffaf9fc1b97bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11968.062005] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [11968.062008] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff96848811cae0 [11968.062010] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [11968.062012] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [11968.062015] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11968.062019] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [11968.062022] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11968.062025] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [11968.062036] Call Trace: [11968.062044] [11968.062052] ? show_regs.cold.9+0x22/0x2f [11968.062066] ? watchdog_timer_fn.cold.6+0x62/0x10a [11968.062073] ? __hrtimer_run_queues+0x1c2/0x4d0 [11968.062077] ? watchdog+0x40/0x40 [11968.062082] ? hrtimer_interrupt+0x128/0x2c0 [11968.062086] ? smp_apic_timer_interrupt+0xa6/0x270 [11968.062094] ? apic_timer_interrupt+0xf/0x20 [11968.062099] [11968.062101] ? native_safe_halt+0x2e/0x50 [11968.062105] kvm_wait+0xcd/0xe0 [11968.062116] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11968.062124] do_raw_spin_lock+0x177/0x190 [11968.062128] _raw_spin_lock+0x21/0x30 [11968.062134] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11968.062648] obd_mod_free+0x46/0x140 [ptlrpc] [11968.062845] mdc_commit_open+0x91/0xb0 [mdc] [11968.062891] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11968.063090] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11968.068663] Modules linked in: lustre(O) [11968.115135] after_reply+0xe2c/0x1780 [ptlrpc] [11968.115760] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11968.115967] ptlrpcd+0xa82/0xe40 [ptlrpc] [11968.134679] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11968.134756] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11968.134763] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11968.134767] RIP: 0010:native_safe_halt+0x2e/0x50 [11968.134780] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11968.134784] RSP: 0000:ffffaf9fc15234f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11968.134788] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [11968.134791] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c21b4154 [11968.134793] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [11968.134796] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [11968.134811] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [11968.134814] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [11968.134818] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11968.134820] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [11968.134828] Call Trace: [11968.134834] [11968.134840] ? show_regs.cold.9+0x22/0x2f [11968.134850] ? watchdog_timer_fn.cold.6+0x62/0x10a [11968.134858] ? __hrtimer_run_queues+0x1c2/0x4d0 [11968.134862] ? watchdog+0x40/0x40 [11968.134866] ? hrtimer_interrupt+0x128/0x2c0 [11968.134870] ? smp_apic_timer_interrupt+0xa6/0x270 [11968.134877] ? apic_timer_interrupt+0xf/0x20 [11968.134881] [11968.134884] ? native_safe_halt+0x2e/0x50 [11968.134889] kvm_wait+0xcd/0xe0 [11968.134897] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11968.134906] do_raw_spin_lock+0x177/0x190 [11968.134910] _raw_spin_lock+0x21/0x30 [11968.134914] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11968.142969] ? do_wait_intr+0xf0/0xf0 [11968.148523] ? do_raw_spin_unlock+0x75/0x190 [11968.148547] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11968.148858] ? do_wait_intr+0xf0/0xf0 [11968.148868] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11968.149063] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11968.174667] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11968.175383] kthread+0x1d1/0x200 [11968.175396] ? set_kthread_struct+0x70/0x70 [11968.175402] ret_from_fork+0x1f/0x30 [11968.419792] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11968.422878] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11968.425612] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11968.428653] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11968.430920] mdc_intent_lock+0x353/0x7e0 [mdc] [11968.433192] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11968.435724] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11968.438097] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11968.441057] lmv_intent_open+0x338/0x1150 [lmv] [11968.443438] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11968.445931] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11968.450410] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11968.454422] ll_intent_lock+0xc5/0xaa0 [lustre] [11968.457244] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11968.460252] ll_lookup_it+0xe0e/0x2270 [lustre] [11968.463492] ? do_raw_spin_unlock+0x75/0x190 [11968.466761] ? _raw_spin_unlock+0x12/0x30 [11968.472432] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11968.476506] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11968.482847] ll_atomic_open+0x36a/0xd70 [lustre] [11968.488137] lookup_open+0x356/0xb40 [11968.494559] do_last+0x313/0xfc0 [11968.497230] ? nd_jump_root+0xe5/0x160 [11968.498861] ? path_init+0x437/0x520 [11968.500139] path_openat+0xf7/0x500 [11968.506399] ? do_raw_spin_unlock+0x75/0x190 [11968.508878] do_filp_open+0x99/0x140 [11968.512947] ? getname_flags+0x6e/0x330 [11968.516778] ? __check_object_size+0xff/0x256 [11968.521992] ? do_raw_spin_unlock+0x75/0x190 [11968.525199] ? _raw_spin_unlock+0x12/0x30 [11968.527387] do_sys_openat2+0x2b4/0x410 [11968.528924] do_sys_open+0x73/0xa0 [11968.530445] __x64_sys_openat+0x24/0x30 [11968.532480] do_syscall_64+0xc1/0x440 [11968.534071] entry_SYSCALL_64_after_hwframe+0x49/0xae [11968.536211] RIP: 0033:0x7ff54af45332 [11968.538288] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11968.549732] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11968.553043] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [11968.555760] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [11968.563692] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [11968.567656] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [11968.571515] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [11971.551705] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10012s! [11971.555848] Showing busy workqueues and worker pools: [11971.558342] workqueue events: flags=0x0 [11971.562478] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11971.566269] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [11971.571454] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [11971.576707] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [11971.590990] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11971.600788] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [11971.608576] workqueue events_long: flags=0x0 [11971.613165] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11971.617545] pending: lu_objects_flush [obdclass] [11971.623556] workqueue events_unbound: flags=0x2 [11971.631261] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [11971.634058] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11971.645851] workqueue events_power_efficient: flags=0x80 [11971.652121] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11971.656928] pending: neigh_periodic_work [11971.661986] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11971.674319] pending: do_cache_clean, check_lifetime [11971.679975] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11971.685562] pending: neigh_periodic_work [11971.690599] workqueue writeback: flags=0x4a [11971.693826] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11971.699617] pending: wb_workfn, wb_workfn [11971.709960] workqueue xprtiod: flags=0xa [11971.711433] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11971.717362] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [11971.725106] workqueue ipv6_addrconf: flags=0x40008 [11971.728886] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11971.735500] pending: addrconf_verify_work [11971.741350] workqueue knbd0-recv: flags=0x1a [11971.745064] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11971.755689] in-flight: 657:recv_work [11971.757227] workqueue ptlrpc_pinger: flags=0x2 [11971.763607] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11971.766998] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [11971.775722] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10012s workers=4 idle: 20831 12642 28859 [11971.782476] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11962s workers=2 idle: 659 [11972.058018] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [11972.061840] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11972.094351] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11972.098966] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11972.102737] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11972.105895] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11972.114089] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11972.117806] RAX: 0000000000005215 RBX: ffff96848811cae0 RCX: 0000000000000001 [11972.121002] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [11972.124698] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [11972.127582] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [11972.130479] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [11972.132766] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [11972.135806] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11972.138452] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [11972.140949] Call Trace: [11972.141895] [11972.142756] ? show_regs.cold.9+0x22/0x2f [11972.144348] ? watchdog_timer_fn.cold.6+0x62/0x10a [11972.146489] ? __hrtimer_run_queues+0x1c2/0x4d0 [11972.148012] ? watchdog+0x40/0x40 [11972.149392] ? hrtimer_interrupt+0x128/0x2c0 [11972.151207] ? smp_apic_timer_interrupt+0xa6/0x270 [11972.153378] ? apic_timer_interrupt+0xf/0x20 [11972.155102] [11972.156078] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11972.158391] do_raw_spin_lock+0x177/0x190 [11972.159950] _raw_spin_lock+0x21/0x30 [11972.161472] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11972.163568] ? do_raw_spin_unlock+0x75/0x190 [11972.165107] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11972.167520] ? do_wait_intr+0xf0/0xf0 [11972.169250] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11972.171645] mdc_close+0x53b/0xd70 [mdc] [11972.173001] lmv_close+0x18d/0x490 [lmv] [11972.174890] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [11972.179003] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11972.181574] ll_md_real_close+0xa5/0x2a0 [lustre] [11972.183731] ll_md_close+0x274/0x800 [lustre] [11972.185877] ll_file_release+0x128/0xc50 [lustre] [11972.189514] __fput+0x10d/0x440 [11972.199525] ____fput+0x12/0x20 [11972.201539] task_work_run+0xc6/0x110 [11972.206435] exit_to_usermode_loop+0x1dd/0x1f0 [11972.208677] do_syscall_64+0x3d6/0x440 [11972.211993] entry_SYSCALL_64_after_hwframe+0x49/0xae [11972.213873] RIP: 0033:0x7ff54af45cd8 [11972.215080] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11972.233694] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11972.239543] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [11972.243377] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [11972.247204] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [11972.250840] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [11972.253690] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11988.059012] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [11988.064120] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11988.092698] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11988.100715] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11988.109459] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11988.112834] RIP: 0010:__pv_queued_spin_lock_slowpath+0x320/0x6d0 [11988.117376] Code: 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 <48> 83 05 18 d4 dc 02 01 eb e0 48 83 05 06 d4 dc 02 01 48 83 05 be [11988.135492] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000286 ORIG_RAX: ffffffffffffff13 [11988.149588] RAX: 0000000000001182 RBX: ffff96848811cae0 RCX: 0000000000000001 [11988.153869] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [11988.161865] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [11988.167950] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [11988.178807] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [11988.186244] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [11988.191911] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11988.195780] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [11988.199748] Call Trace: [11988.201664] [11988.202122] ? show_regs.cold.9+0x22/0x2f [11988.203480] ? watchdog_timer_fn.cold.6+0x62/0x10a [11988.206810] ? __hrtimer_run_queues+0x1c2/0x4d0 [11988.208549] ? watchdog+0x40/0x40 [11988.214352] ? hrtimer_interrupt+0x128/0x2c0 [11988.216760] ? smp_apic_timer_interrupt+0xa6/0x270 [11988.218488] ? apic_timer_interrupt+0xf/0x20 [11988.220613] [11988.221378] ? __pv_queued_spin_lock_slowpath+0x320/0x6d0 [11988.223758] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11988.225358] do_raw_spin_lock+0x177/0x190 [11988.227073] _raw_spin_lock+0x21/0x30 [11988.228150] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11988.235323] process_one_work+0x2c8/0x700 [11988.237784] worker_thread+0x296/0x6e0 [11988.240059] ? rescuer_thread+0x570/0x570 [11988.242640] kthread+0x1d1/0x200 [11988.244476] ? set_kthread_struct+0x70/0x70 [11988.246573] ret_from_fork+0x1f/0x30 [11996.055022] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [11996.057604] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11996.061112] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [11996.073288] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11996.073313] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11996.073317] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11996.073337] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11996.073345] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11996.073353] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [11996.073356] RDX: 0000000000002322 RSI: 0000000000000003 RDI: ffff96848811cae0 [11996.073358] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [11996.073361] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [11996.073364] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11996.073368] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [11996.073371] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11996.073374] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [11996.073382] Call Trace: [11996.073393] [11996.073400] ? show_regs.cold.9+0x22/0x2f [11996.073412] ? watchdog_timer_fn.cold.6+0x62/0x10a [11996.073419] ? __hrtimer_run_queues+0x1c2/0x4d0 [11996.073432] ? watchdog+0x40/0x40 [11996.078029] Modules linked in: lustre(O) [11996.081510] ? hrtimer_interrupt+0x128/0x2c0 [11996.081537] ? smp_apic_timer_interrupt+0xa6/0x270 [11996.086339] osp(O) [11996.089672] ? apic_timer_interrupt+0xf/0x20 [11996.100089] ofd(O) [11996.103404] [11996.103416] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11996.103429] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11996.103434] do_raw_spin_lock+0x177/0x190 [11996.103443] _raw_spin_lock+0x21/0x30 [11996.103451] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11996.109384] lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11996.109462] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11996.109532] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11996.109535] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11996.109549] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11996.109553] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11996.109557] RAX: 0000000000005662 RBX: ffff96848811cae0 RCX: 0000000000000001 [11996.109559] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [11996.109561] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [11996.109563] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [11996.109566] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [11996.109569] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [11996.109577] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11996.109580] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [11996.109587] Call Trace: [11996.109594] [11996.109600] ? show_regs.cold.9+0x22/0x2f [11996.109610] ? watchdog_timer_fn.cold.6+0x62/0x10a [11996.109616] ? __hrtimer_run_queues+0x1c2/0x4d0 [11996.109620] ? watchdog+0x40/0x40 [11996.109625] ? hrtimer_interrupt+0x128/0x2c0 [11996.109629] ? smp_apic_timer_interrupt+0xa6/0x270 [11996.109637] ? apic_timer_interrupt+0xf/0x20 [11996.109642] [11996.109645] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11996.109652] do_raw_spin_lock+0x177/0x190 [11996.109657] _raw_spin_lock+0x21/0x30 [11996.109661] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11996.110146] ? do_raw_spin_unlock+0x75/0x190 [11996.110158] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11996.114687] obd_mod_free+0x46/0x140 [ptlrpc] [11996.115334] mdc_commit_open+0x91/0xb0 [mdc] [11996.115378] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11996.115680] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11996.116036] after_reply+0xe2c/0x1780 [ptlrpc] [11996.116302] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11996.116565] ptlrpcd+0xa82/0xe40 [ptlrpc] [11996.116882] ? do_wait_intr+0xf0/0xf0 [11996.116955] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11996.117455] kthread+0x1d1/0x200 [11996.117461] ? set_kthread_struct+0x70/0x70 [11996.117464] ret_from_fork+0x1f/0x30 [11996.389413] ? do_wait_intr+0xf0/0xf0 [11996.390664] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11996.395514] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11996.398210] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11996.401155] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11996.402961] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11996.406561] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11996.409074] mdc_intent_lock+0x353/0x7e0 [mdc] [11996.410725] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11996.416074] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11996.421788] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11996.428694] lmv_intent_open+0x338/0x1150 [lmv] [11996.432190] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11996.435209] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11996.438215] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11996.441173] ll_intent_lock+0xc5/0xaa0 [lustre] [11996.445219] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11996.450643] ll_lookup_it+0xe0e/0x2270 [lustre] [11996.458701] ? do_raw_spin_unlock+0x75/0x190 [11996.461444] ? _raw_spin_unlock+0x12/0x30 [11996.462858] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11996.474481] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11996.480094] ll_atomic_open+0x36a/0xd70 [lustre] [11996.483429] lookup_open+0x356/0xb40 [11996.485644] do_last+0x313/0xfc0 [11996.487786] ? nd_jump_root+0xe5/0x160 [11996.490587] ? path_init+0x437/0x520 [11996.492515] path_openat+0xf7/0x500 [11996.494153] ? do_raw_spin_unlock+0x75/0x190 [11996.495852] do_filp_open+0x99/0x140 [11996.496856] ? getname_flags+0x6e/0x330 [11996.503512] ? __check_object_size+0xff/0x256 [11996.506672] ? do_raw_spin_unlock+0x75/0x190 [11996.509196] ? _raw_spin_unlock+0x12/0x30 [11996.512522] do_sys_openat2+0x2b4/0x410 [11996.516123] do_sys_open+0x73/0xa0 [11996.517290] __x64_sys_openat+0x24/0x30 [11996.526037] do_syscall_64+0xc1/0x440 [11996.529686] entry_SYSCALL_64_after_hwframe+0x49/0xae [11996.531990] RIP: 0033:0x7ff54af45332 [11996.533170] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11996.545662] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11996.548877] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [11996.552154] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [11996.555105] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [11996.558182] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [11996.560457] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [12000.058038] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [12000.061449] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12000.096076] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12000.101137] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12000.105500] RIP: 0010:native_safe_halt+0x2e/0x50 [12000.110328] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12000.130096] RSP: 0018:ffffaf9fc1543ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12000.134589] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [12000.141875] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c20b4154 [12000.149176] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [12000.156194] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [12000.160088] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [12000.165946] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [12000.168734] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12000.175827] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [12000.183836] Call Trace: [12000.188030] [12000.188768] ? show_regs.cold.9+0x22/0x2f [12000.191037] ? watchdog_timer_fn.cold.6+0x62/0x10a [12000.196134] ? __hrtimer_run_queues+0x1c2/0x4d0 [12000.198238] ? watchdog+0x40/0x40 [12000.202305] ? hrtimer_interrupt+0x128/0x2c0 [12000.207417] ? smp_apic_timer_interrupt+0xa6/0x270 [12000.209371] ? apic_timer_interrupt+0xf/0x20 [12000.217345] [12000.221936] ? native_safe_halt+0x2e/0x50 [12000.222967] kvm_wait+0xcd/0xe0 [12000.224138] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12000.234674] do_raw_spin_lock+0x177/0x190 [12000.237949] _raw_spin_lock+0x21/0x30 [12000.241542] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12000.249478] ? do_raw_spin_unlock+0x75/0x190 [12000.251052] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12000.259248] ? do_wait_intr+0xf0/0xf0 [12000.261767] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12000.267420] mdc_close+0x53b/0xd70 [mdc] [12000.271928] lmv_close+0x18d/0x490 [lmv] [12000.275758] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [12000.283073] ll_close_inode_openhandle+0x493/0xea0 [lustre] [12000.284802] ll_md_real_close+0xa5/0x2a0 [lustre] [12000.289282] ll_md_close+0x274/0x800 [lustre] [12000.291842] ll_file_release+0x128/0xc50 [lustre] [12000.294514] __fput+0x10d/0x440 [12000.297262] ____fput+0x12/0x20 [12000.299577] task_work_run+0xc6/0x110 [12000.301511] exit_to_usermode_loop+0x1dd/0x1f0 [12000.305398] do_syscall_64+0x3d6/0x440 [12000.307293] entry_SYSCALL_64_after_hwframe+0x49/0xae [12000.316568] RIP: 0033:0x7ff54af45cd8 [12000.321355] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [12000.335489] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [12000.340101] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [12000.350099] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [12000.356379] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [12000.362747] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [12000.366825] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12002.271080] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10043s! [12002.274045] Showing busy workqueues and worker pools: [12002.275655] workqueue events: flags=0x0 [12002.280167] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12002.286659] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [12002.293411] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [12002.299780] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [12002.308788] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12002.312813] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [12002.316691] workqueue events_long: flags=0x0 [12002.319105] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12002.325649] pending: lu_objects_flush [obdclass] [12002.331734] workqueue events_unbound: flags=0x2 [12002.334024] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [12002.340090] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [12002.346154] workqueue events_power_efficient: flags=0x80 [12002.352732] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12002.356684] pending: neigh_periodic_work [12002.360030] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12002.368274] pending: do_cache_clean, check_lifetime [12002.370056] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12002.373498] pending: neigh_periodic_work [12002.376306] workqueue writeback: flags=0x4a [12002.378961] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [12002.385997] pending: wb_workfn, wb_workfn [12002.387077] workqueue xprtiod: flags=0xa [12002.390093] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [12002.399083] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [12002.403068] workqueue ipv6_addrconf: flags=0x40008 [12002.405938] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12002.408347] pending: addrconf_verify_work [12002.414724] workqueue knbd0-recv: flags=0x1a [12002.417278] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12002.421483] in-flight: 657:recv_work [12002.423416] workqueue ptlrpc_pinger: flags=0x2 [12002.426372] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12002.430713] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [12002.433849] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10043s workers=4 idle: 20831 12642 28859 [12002.437771] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11992s workers=2 idle: 659 [12016.060850] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [12016.068016] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12016.117725] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12016.138212] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12016.154269] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12016.161149] RIP: 0010:native_safe_halt+0x2e/0x50 [12016.166675] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12016.197737] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12016.208483] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [12016.218407] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c2134154 [12016.231407] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [12016.245544] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [12016.264546] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [12016.278031] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [12016.293998] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12016.300657] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [12016.309990] Call Trace: [12016.314604] [12016.321188] ? show_regs.cold.9+0x22/0x2f [12016.329359] ? watchdog_timer_fn.cold.6+0x62/0x10a [12016.341567] ? __hrtimer_run_queues+0x1c2/0x4d0 [12016.350045] ? watchdog+0x40/0x40 [12016.351198] ? hrtimer_interrupt+0x128/0x2c0 [12016.364797] ? smp_apic_timer_interrupt+0xa6/0x270 [12016.367415] ? apic_timer_interrupt+0xf/0x20 [12016.371212] [12016.372655] ? native_safe_halt+0x2e/0x50 [12016.373906] kvm_wait+0xcd/0xe0 [12016.374837] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12016.377537] do_raw_spin_lock+0x177/0x190 [12016.379763] _raw_spin_lock+0x21/0x30 [12016.381513] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12016.384129] process_one_work+0x2c8/0x700 [12016.385360] worker_thread+0x296/0x6e0 [12016.386743] ? rescuer_thread+0x570/0x570 [12016.388544] kthread+0x1d1/0x200 [12016.390323] ? set_kthread_struct+0x70/0x70 [12016.392052] ret_from_fork+0x1f/0x30 [12024.060160] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [12024.072525] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12024.154010] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12024.170659] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12024.186603] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12024.199275] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12024.224757] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12024.240976] RAX: 0000000000007708 RBX: ffff96848811cae0 RCX: 0000000000000001 [12024.249400] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [12024.260527] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [12024.270748] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [12024.281306] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [12024.292229] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [12024.300037] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12024.308235] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [12024.314019] Call Trace: [12024.326163] [12024.329251] ? show_regs.cold.9+0x22/0x2f [12024.341459] ? watchdog_timer_fn.cold.6+0x62/0x10a [12024.351307] ? __hrtimer_run_queues+0x1c2/0x4d0 [12024.361091] ? watchdog+0x40/0x40 [12024.362366] ? hrtimer_interrupt+0x128/0x2c0 [12024.379778] ? smp_apic_timer_interrupt+0xa6/0x270 [12024.395454] ? apic_timer_interrupt+0xf/0x20 [12024.396977] [12024.397609] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12024.418462] do_raw_spin_lock+0x177/0x190 [12024.427566] _raw_spin_lock+0x21/0x30 [12024.437554] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12024.447837] ? do_raw_spin_unlock+0x75/0x190 [12024.465725] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12024.481399] ? do_wait_intr+0xf0/0xf0 [12024.491415] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12024.505509] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [12024.510852] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12024.516388] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12024.520088] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12024.527454] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [12024.532536] mdc_intent_lock+0x353/0x7e0 [mdc] [12024.538488] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12024.542780] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12024.547376] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12024.554984] lmv_intent_open+0x338/0x1150 [lmv] [12024.557529] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12024.563493] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12024.572307] lmv_intent_lock+0x1dd/0x4c0 [lmv] [12024.577938] ll_intent_lock+0xc5/0xaa0 [lustre] [12024.579486] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12024.598804] ll_lookup_it+0xe0e/0x2270 [lustre] [12024.608551] ? do_raw_spin_unlock+0x75/0x190 [12024.615076] ? _raw_spin_unlock+0x12/0x30 [12024.622372] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [12024.637743] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [12024.644580] ll_atomic_open+0x36a/0xd70 [lustre] [12024.648964] lookup_open+0x356/0xb40 [12024.652936] do_last+0x313/0xfc0 [12024.657818] ? nd_jump_root+0xe5/0x160 [12024.662122] ? path_init+0x437/0x520 [12024.663357] path_openat+0xf7/0x500 [12024.669498] ? do_raw_spin_unlock+0x75/0x190 [12024.674322] do_filp_open+0x99/0x140 [12024.680400] ? getname_flags+0x6e/0x330 [12024.684422] ? __check_object_size+0xff/0x256 [12024.690377] ? do_raw_spin_unlock+0x75/0x190 [12024.695924] ? _raw_spin_unlock+0x12/0x30 [12024.701756] do_sys_openat2+0x2b4/0x410 [12024.703081] do_sys_open+0x73/0xa0 [12024.705853] __x64_sys_openat+0x24/0x30 [12024.707609] do_syscall_64+0xc1/0x440 [12024.708775] entry_SYSCALL_64_after_hwframe+0x49/0xae [12024.711482] RIP: 0033:0x7ff54af45332 [12024.717165] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [12024.738399] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [12024.749048] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [12024.759360] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [12024.777706] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [12024.789372] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [12024.803014] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [12028.057673] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [12028.059319] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [12028.061274] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12028.061373] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12028.061388] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12028.061393] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [12028.061420] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [12028.061425] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12028.061431] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [12028.061433] RDX: 0000000000006761 RSI: 0000000000000003 RDI: ffff96848811cae0 [12028.061436] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [12028.061439] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [12028.061441] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12028.061446] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [12028.061449] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12028.061451] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [12028.061461] Call Trace: [12028.061471] [12028.061478] ? show_regs.cold.9+0x22/0x2f [12028.061491] ? watchdog_timer_fn.cold.6+0x62/0x10a [12028.061499] ? __hrtimer_run_queues+0x1c2/0x4d0 [12028.061504] ? watchdog+0x40/0x40 [12028.061509] ? hrtimer_interrupt+0x128/0x2c0 [12028.061513] ? smp_apic_timer_interrupt+0xa6/0x270 [12028.061527] ? apic_timer_interrupt+0xf/0x20 [12028.061532] [12028.061536] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [12028.061547] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12028.061552] do_raw_spin_lock+0x177/0x190 [12028.061556] _raw_spin_lock+0x21/0x30 [12028.061563] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12028.061952] obd_mod_free+0x46/0x140 [ptlrpc] [12028.062158] mdc_commit_open+0x91/0xb0 [mdc] [12028.062219] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12028.062447] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12028.062656] after_reply+0xe2c/0x1780 [ptlrpc] [12028.062888] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12028.063291] ptlrpcd+0xa82/0xe40 [ptlrpc] [12028.063509] ? do_wait_intr+0xf0/0xf0 [12028.063520] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12028.063763] kthread+0x1d1/0x200 [12028.063769] ? set_kthread_struct+0x70/0x70 [12028.063773] ret_from_fork+0x1f/0x30 [12028.198030] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12028.218615] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12028.222534] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12028.225790] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12028.227998] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12028.234552] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12028.237516] RAX: 0000000000004889 RBX: ffff96848811cae0 RCX: 0000000000000001 [12028.240078] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [12028.242804] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [12028.245365] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [12028.247969] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [12028.250450] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [12028.253407] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12028.255447] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [12028.258224] Call Trace: [12028.259155] [12028.259972] ? show_regs.cold.9+0x22/0x2f [12028.261425] ? watchdog_timer_fn.cold.6+0x62/0x10a [12028.263183] ? __hrtimer_run_queues+0x1c2/0x4d0 [12028.264897] ? watchdog+0x40/0x40 [12028.265954] ? hrtimer_interrupt+0x128/0x2c0 [12028.267535] ? smp_apic_timer_interrupt+0xa6/0x270 [12028.269153] ? apic_timer_interrupt+0xf/0x20 [12028.270836] [12028.271657] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12028.273635] do_raw_spin_lock+0x177/0x190 [12028.275201] _raw_spin_lock+0x21/0x30 [12028.276652] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12028.278810] ? do_raw_spin_unlock+0x75/0x190 [12028.280282] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12028.282274] ? do_wait_intr+0xf0/0xf0 [12028.283674] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12028.285655] mdc_close+0x53b/0xd70 [mdc] [12028.287114] lmv_close+0x18d/0x490 [lmv] [12028.288629] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [12028.290848] ll_close_inode_openhandle+0x493/0xea0 [lustre] [12028.292983] ll_md_real_close+0xa5/0x2a0 [lustre] [12028.294854] ll_md_close+0x274/0x800 [lustre] [12028.296508] ll_file_release+0x128/0xc50 [lustre] [12028.298398] __fput+0x10d/0x440 [12028.299493] ____fput+0x12/0x20 [12028.300663] task_work_run+0xc6/0x110 [12028.302059] exit_to_usermode_loop+0x1dd/0x1f0 [12028.303791] do_syscall_64+0x3d6/0x440 [12028.305206] entry_SYSCALL_64_after_hwframe+0x49/0xae [12028.307092] RIP: 0033:0x7ff54af45cd8 [12028.308438] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [12028.314953] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [12028.317651] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [12028.320330] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [12028.323038] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [12028.325811] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [12028.328440] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12032.481304] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10073s! [12032.492782] Showing busy workqueues and worker pools: [12032.497882] workqueue events: flags=0x0 [12032.505406] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12032.521728] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [12032.531527] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [12032.542470] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [12032.557404] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12032.561846] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [12032.563962] workqueue events_long: flags=0x0 [12032.564967] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12032.572527] pending: lu_objects_flush [obdclass] [12032.576222] workqueue events_unbound: flags=0x2 [12032.586916] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [12032.603337] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [12032.616089] workqueue events_power_efficient: flags=0x80 [12032.626102] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12032.646830] pending: neigh_periodic_work [12032.663109] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12032.674857] pending: do_cache_clean, check_lifetime [12032.694554] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12032.709536] pending: neigh_periodic_work [12032.719586] workqueue writeback: flags=0x4a [12032.721789] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [12032.731895] pending: wb_workfn, wb_workfn [12032.738843] workqueue xprtiod: flags=0xa [12032.745259] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [12032.759982] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [12032.785264] workqueue ipv6_addrconf: flags=0x40008 [12032.796309] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12032.811259] pending: addrconf_verify_work [12032.816941] workqueue knbd0-recv: flags=0x1a [12032.818495] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12032.847951] in-flight: 657:recv_work [12032.858087] workqueue ptlrpc_pinger: flags=0x2 [12032.876134] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12032.878474] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [12032.904087] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10074s workers=4 idle: 20831 12642 28859 [12032.916847] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12023s workers=2 idle: 659 [12044.059073] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [12044.065234] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12044.135886] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12044.157140] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12044.169758] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12044.177855] RIP: 0010:native_safe_halt+0x2e/0x50 [12044.181543] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12044.201876] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12044.213502] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [12044.227183] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [12044.233917] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [12044.240201] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [12044.249366] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [12044.256908] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [12044.265837] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12044.275992] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [12044.293772] Call Trace: [12044.295334] [12044.298939] ? show_regs.cold.9+0x22/0x2f [12044.302646] ? watchdog_timer_fn.cold.6+0x62/0x10a [12044.305883] ? __hrtimer_run_queues+0x1c2/0x4d0 [12044.309932] ? watchdog+0x40/0x40 [12044.311971] ? hrtimer_interrupt+0x128/0x2c0 [12044.315940] ? smp_apic_timer_interrupt+0xa6/0x270 [12044.318781] ? apic_timer_interrupt+0xf/0x20 [12044.321331] [12044.322305] ? native_safe_halt+0x2e/0x50 [12044.324850] kvm_wait+0xcd/0xe0 [12044.329909] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12044.334902] do_raw_spin_lock+0x177/0x190 [12044.336481] _raw_spin_lock+0x21/0x30 [12044.338745] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12044.342151] process_one_work+0x2c8/0x700 [12044.345786] worker_thread+0x296/0x6e0 [12044.347931] ? rescuer_thread+0x570/0x570 [12044.351283] kthread+0x1d1/0x200 [12044.352563] ? set_kthread_struct+0x70/0x70 [12044.358291] ret_from_fork+0x1f/0x30 [12052.060015] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [12052.063332] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12052.097468] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12052.102253] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12052.108694] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12052.115205] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12052.128064] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12052.134174] RAX: 0000000000003a97 RBX: ffff96848811cae0 RCX: 0000000000000001 [12052.137441] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [12052.144597] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [12052.146927] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [12052.151620] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [12052.154743] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [12052.162926] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12052.166643] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [12052.169720] Call Trace: [12052.170745] [12052.171642] ? show_regs.cold.9+0x22/0x2f [12052.175770] ? watchdog_timer_fn.cold.6+0x62/0x10a [12052.177132] ? __hrtimer_run_queues+0x1c2/0x4d0 [12052.178267] ? watchdog+0x40/0x40 [12052.180143] ? hrtimer_interrupt+0x128/0x2c0 [12052.185249] ? smp_apic_timer_interrupt+0xa6/0x270 [12052.186930] ? apic_timer_interrupt+0xf/0x20 [12052.188132] [12052.188937] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12052.202163] do_raw_spin_lock+0x177/0x190 [12052.203609] _raw_spin_lock+0x21/0x30 [12052.205306] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12052.208649] ? do_raw_spin_unlock+0x75/0x190 [12052.210926] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12052.218368] ? do_wait_intr+0xf0/0xf0 [12052.219721] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12052.223164] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [12052.224907] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12052.229088] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12052.231257] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12052.233650] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [12052.235113] mdc_intent_lock+0x353/0x7e0 [mdc] [12052.237198] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12052.239525] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12052.242952] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12052.244568] lmv_intent_open+0x338/0x1150 [lmv] [12052.246032] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12052.248264] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12052.250167] lmv_intent_lock+0x1dd/0x4c0 [lmv] [12052.251727] ll_intent_lock+0xc5/0xaa0 [lustre] [12052.253272] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12052.254978] ll_lookup_it+0xe0e/0x2270 [lustre] [12052.256160] ? do_raw_spin_unlock+0x75/0x190 [12052.257223] ? _raw_spin_unlock+0x12/0x30 [12052.258577] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [12052.260632] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [12052.262561] ll_atomic_open+0x36a/0xd70 [lustre] [12052.264321] lookup_open+0x356/0xb40 [12052.265917] do_last+0x313/0xfc0 [12052.267049] ? nd_jump_root+0xe5/0x160 [12052.268450] ? path_init+0x437/0x520 [12052.269544] path_openat+0xf7/0x500 [12052.270439] ? do_raw_spin_unlock+0x75/0x190 [12052.271163] do_filp_open+0x99/0x140 [12052.271978] ? getname_flags+0x6e/0x330 [12052.273040] ? __check_object_size+0xff/0x256 [12052.274176] ? do_raw_spin_unlock+0x75/0x190 [12052.275303] ? _raw_spin_unlock+0x12/0x30 [12052.276329] do_sys_openat2+0x2b4/0x410 [12052.277649] do_sys_open+0x73/0xa0 [12052.278675] __x64_sys_openat+0x24/0x30 [12052.279853] do_syscall_64+0xc1/0x440 [12052.280920] entry_SYSCALL_64_after_hwframe+0x49/0xae [12052.282393] RIP: 0033:0x7ff54af45332 [12052.283500] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [12052.288531] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [12052.290489] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [12052.292325] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [12052.294573] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [12052.297024] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [12052.299839] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [12056.063198] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [12056.069817] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12056.092934] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12056.101134] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12056.107951] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12056.111872] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12056.126900] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12056.133093] RAX: 0000000000007877 RBX: ffff96848811cae0 RCX: 0000000000000001 [12056.137662] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [12056.142026] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [12056.149199] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [12056.154151] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [12056.160926] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [12056.169950] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12056.173909] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [12056.183174] Call Trace: [12056.185725] [12056.188880] ? show_regs.cold.9+0x22/0x2f [12056.194460] ? watchdog_timer_fn.cold.6+0x62/0x10a [12056.197058] ? __hrtimer_run_queues+0x1c2/0x4d0 [12056.202893] ? watchdog+0x40/0x40 [12056.207068] ? hrtimer_interrupt+0x128/0x2c0 [12056.210550] ? smp_apic_timer_interrupt+0xa6/0x270 [12056.212610] ? apic_timer_interrupt+0xf/0x20 [12056.216723] [12056.218007] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12056.221698] do_raw_spin_lock+0x177/0x190 [12056.225777] _raw_spin_lock+0x21/0x30 [12056.228297] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12056.235630] ? do_raw_spin_unlock+0x75/0x190 [12056.237830] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12056.240743] ? do_wait_intr+0xf0/0xf0 [12056.246041] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12056.250702] mdc_close+0x53b/0xd70 [mdc] [12056.255985] lmv_close+0x18d/0x490 [lmv] [12056.257932] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [12056.261743] ll_close_inode_openhandle+0x493/0xea0 [lustre] [12056.265075] ll_md_real_close+0xa5/0x2a0 [lustre] [12056.267605] ll_md_close+0x274/0x800 [lustre] [12056.271613] ll_file_release+0x128/0xc50 [lustre] [12056.274857] __fput+0x10d/0x440 [12056.278724] ____fput+0x12/0x20 [12056.282380] task_work_run+0xc6/0x110 [12056.286687] exit_to_usermode_loop+0x1dd/0x1f0 [12056.290697] do_syscall_64+0x3d6/0x440 [12056.294161] entry_SYSCALL_64_after_hwframe+0x49/0xae [12056.296859] RIP: 0033:0x7ff54af45cd8 [12056.301698] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [12056.314086] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [12056.321034] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [12056.325804] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [12056.329595] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [12056.334119] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [12056.346811] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12060.060285] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [12060.069958] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12060.130709] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12060.139403] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12060.153646] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [12060.165531] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [12060.183109] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12060.186471] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [12060.189873] RDX: 00000000000068d0 RSI: 0000000000000003 RDI: ffff96848811cae0 [12060.197619] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [12060.200880] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [12060.211046] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12060.217044] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [12060.225773] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12060.232823] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [12060.236059] Call Trace: [12060.238064] [12060.241474] ? show_regs.cold.9+0x22/0x2f [12060.246542] ? watchdog_timer_fn.cold.6+0x62/0x10a [12060.251950] ? __hrtimer_run_queues+0x1c2/0x4d0 [12060.256076] ? watchdog+0x40/0x40 [12060.258820] ? hrtimer_interrupt+0x128/0x2c0 [12060.261681] ? smp_apic_timer_interrupt+0xa6/0x270 [12060.264917] ? apic_timer_interrupt+0xf/0x20 [12060.268598] [12060.270668] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [12060.274486] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12060.278069] do_raw_spin_lock+0x177/0x190 [12060.280220] _raw_spin_lock+0x21/0x30 [12060.285220] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12060.289182] obd_mod_free+0x46/0x140 [ptlrpc] [12060.296240] mdc_commit_open+0x91/0xb0 [mdc] [12060.301327] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12060.305710] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12060.310517] after_reply+0xe2c/0x1780 [ptlrpc] [12060.319528] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12060.325277] ptlrpcd+0xa82/0xe40 [ptlrpc] [12060.328756] ? do_wait_intr+0xf0/0xf0 [12060.333939] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12060.337540] kthread+0x1d1/0x200 [12060.340665] ? set_kthread_struct+0x70/0x70 [12060.347590] ret_from_fork+0x1f/0x30 [12063.201362] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10104s! [12063.208045] Showing busy workqueues and worker pools: [12063.210360] workqueue events: flags=0x0 [12063.211932] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12063.215327] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [12063.219723] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [12063.222459] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [12063.226904] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12063.229522] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [12063.232088] workqueue events_long: flags=0x0 [12063.233524] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12063.235094] pending: lu_objects_flush [obdclass] [12063.236643] workqueue events_unbound: flags=0x2 [12063.237904] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [12063.239790] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [12063.242943] workqueue events_power_efficient: flags=0x80 [12063.244991] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12063.247723] pending: neigh_periodic_work [12063.249335] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12063.251172] pending: do_cache_clean, check_lifetime [12063.253644] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12063.255590] pending: neigh_periodic_work [12063.258788] workqueue writeback: flags=0x4a [12063.260044] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [12063.262221] pending: wb_workfn, wb_workfn [12063.269778] workqueue xprtiod: flags=0xa [12063.271083] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [12063.272781] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [12063.275314] workqueue ipv6_addrconf: flags=0x40008 [12063.276545] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12063.278602] pending: addrconf_verify_work [12063.280103] workqueue knbd0-recv: flags=0x1a [12063.281389] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12063.283152] in-flight: 657:recv_work [12063.284200] workqueue ptlrpc_pinger: flags=0x2 [12063.285604] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12063.287461] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [12063.289387] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10104s workers=4 idle: 20831 12642 28859 [12063.292604] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12053s workers=2 idle: 659 [12072.059049] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [12072.063135] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12072.084279] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12072.089765] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12072.096466] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12072.100264] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12072.103390] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12072.113214] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12072.117112] RAX: 000000000000495d RBX: ffff96848811cae0 RCX: 0000000000000001 [12072.120740] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [12072.124476] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [12072.129350] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [12072.132995] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [12072.136889] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [12072.140862] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12072.143379] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [12072.147102] Call Trace: [12072.148346] [12072.149411] ? show_regs.cold.9+0x22/0x2f [12072.151507] ? watchdog_timer_fn.cold.6+0x62/0x10a [12072.154279] ? __hrtimer_run_queues+0x1c2/0x4d0 [12072.156613] ? watchdog+0x40/0x40 [12072.158395] ? hrtimer_interrupt+0x128/0x2c0 [12072.160526] ? smp_apic_timer_interrupt+0xa6/0x270 [12072.162945] ? apic_timer_interrupt+0xf/0x20 [12072.167012] [12072.168147] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12072.171993] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12072.177123] do_raw_spin_lock+0x177/0x190 [12072.179410] _raw_spin_lock+0x21/0x30 [12072.183404] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12072.188607] process_one_work+0x2c8/0x700 [12072.191238] worker_thread+0x296/0x6e0 [12072.193815] ? rescuer_thread+0x570/0x570 [12072.196393] kthread+0x1d1/0x200 [12072.198536] ? set_kthread_struct+0x70/0x70 [12072.200892] ret_from_fork+0x1f/0x30 [12080.065026] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [12080.075812] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12080.114192] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12080.127939] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12080.138940] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12080.143806] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12080.162261] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12080.170962] RAX: 0000000000006d84 RBX: ffff96848811cae0 RCX: 0000000000000001 [12080.176896] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [12080.183313] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [12080.191004] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [12080.200578] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [12080.205089] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [12080.211339] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12080.214320] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [12080.217823] Call Trace: [12080.218982] [12080.220416] ? show_regs.cold.9+0x22/0x2f [12080.222889] ? watchdog_timer_fn.cold.6+0x62/0x10a [12080.224891] ? __hrtimer_run_queues+0x1c2/0x4d0 [12080.226758] ? watchdog+0x40/0x40 [12080.227876] ? hrtimer_interrupt+0x128/0x2c0 [12080.229900] ? smp_apic_timer_interrupt+0xa6/0x270 [12080.232643] ? apic_timer_interrupt+0xf/0x20 [12080.234765] [12080.236015] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12080.237509] do_raw_spin_lock+0x177/0x190 [12080.238976] _raw_spin_lock+0x21/0x30 [12080.240602] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12080.242791] ? do_raw_spin_unlock+0x75/0x190 [12080.245193] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12080.247518] ? do_wait_intr+0xf0/0xf0 [12080.248498] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12080.250852] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [12080.253942] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12080.256473] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12080.258905] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12080.261636] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [12080.264379] mdc_intent_lock+0x353/0x7e0 [mdc] [12080.266344] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12080.273789] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12080.277166] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12080.280802] lmv_intent_open+0x338/0x1150 [lmv] [12080.283617] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12080.286411] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12080.288976] lmv_intent_lock+0x1dd/0x4c0 [lmv] [12080.290722] ll_intent_lock+0xc5/0xaa0 [lustre] [12080.292623] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12080.296526] ll_lookup_it+0xe0e/0x2270 [lustre] [12080.299354] ? do_raw_spin_unlock+0x75/0x190 [12080.301634] ? _raw_spin_unlock+0x12/0x30 [12080.303980] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [12080.307383] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [12080.310632] ll_atomic_open+0x36a/0xd70 [lustre] [12080.313525] lookup_open+0x356/0xb40 [12080.314516] do_last+0x313/0xfc0 [12080.315289] ? nd_jump_root+0xe5/0x160 [12080.316215] ? path_init+0x437/0x520 [12080.322976] path_openat+0xf7/0x500 [12080.326066] ? do_raw_spin_unlock+0x75/0x190 [12080.332010] do_filp_open+0x99/0x140 [12080.333225] ? getname_flags+0x6e/0x330 [12080.334152] ? __check_object_size+0xff/0x256 [12080.337354] ? do_raw_spin_unlock+0x75/0x190 [12080.341727] ? _raw_spin_unlock+0x12/0x30 [12080.343184] do_sys_openat2+0x2b4/0x410 [12080.345333] do_sys_open+0x73/0xa0 [12080.347916] __x64_sys_openat+0x24/0x30 [12080.348885] do_syscall_64+0xc1/0x440 [12080.354596] entry_SYSCALL_64_after_hwframe+0x49/0xae [12080.361964] RIP: 0033:0x7ff54af45332 [12080.364210] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [12080.376331] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [12080.379628] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [12080.381632] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [12080.384440] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [12080.387658] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [12080.396543] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [12084.058226] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [12084.061618] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12084.090471] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12084.098295] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12084.104075] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12084.106861] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12084.116458] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12084.126464] RAX: 000000000000783d RBX: ffff96848811cae0 RCX: 0000000000000001 [12084.138891] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [12084.141908] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [12084.153013] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [12084.164323] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [12084.174079] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [12084.191912] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12084.195481] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [12084.206107] Call Trace: [12084.210920] [12084.211994] ? show_regs.cold.9+0x22/0x2f [12084.213934] ? watchdog_timer_fn.cold.6+0x62/0x10a [12084.222429] ? __hrtimer_run_queues+0x1c2/0x4d0 [12084.226172] ? watchdog+0x40/0x40 [12084.228332] ? hrtimer_interrupt+0x128/0x2c0 [12084.230582] ? smp_apic_timer_interrupt+0xa6/0x270 [12084.232921] ? apic_timer_interrupt+0xf/0x20 [12084.241316] [12084.247515] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12084.254912] do_raw_spin_lock+0x177/0x190 [12084.261565] _raw_spin_lock+0x21/0x30 [12084.268788] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12084.276197] ? do_raw_spin_unlock+0x75/0x190 [12084.282999] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12084.286436] ? do_wait_intr+0xf0/0xf0 [12084.290318] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12084.293900] mdc_close+0x53b/0xd70 [mdc] [12084.295679] lmv_close+0x18d/0x490 [lmv] [12084.299641] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [12084.302817] ll_close_inode_openhandle+0x493/0xea0 [lustre] [12084.305718] ll_md_real_close+0xa5/0x2a0 [lustre] [12084.308835] ll_md_close+0x274/0x800 [lustre] [12084.310778] ll_file_release+0x128/0xc50 [lustre] [12084.313227] __fput+0x10d/0x440 [12084.314294] ____fput+0x12/0x20 [12084.315722] task_work_run+0xc6/0x110 [12084.321491] exit_to_usermode_loop+0x1dd/0x1f0 [12084.327148] do_syscall_64+0x3d6/0x440 [12084.330925] entry_SYSCALL_64_after_hwframe+0x49/0xae [12084.338244] RIP: 0033:0x7ff54af45cd8 [12084.343443] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [12084.356102] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [12084.358960] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [12084.363497] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [12084.370673] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [12084.377442] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [12084.383597] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12088.055163] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [12088.059410] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12088.109513] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12088.122591] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12088.130523] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [12088.136545] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [12088.152117] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12088.156304] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [12088.158362] RDX: 00000000000068f1 RSI: 0000000000000003 RDI: ffff96848811cae0 [12088.162568] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [12088.166534] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [12088.169645] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12088.180839] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [12088.190480] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12088.197324] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [12088.202924] Call Trace: [12088.205208] [12088.207280] ? show_regs.cold.9+0x22/0x2f [12088.208489] ? watchdog_timer_fn.cold.6+0x62/0x10a [12088.210127] ? __hrtimer_run_queues+0x1c2/0x4d0 [12088.217090] ? watchdog+0x40/0x40 [12088.219551] ? hrtimer_interrupt+0x128/0x2c0 [12088.225673] ? smp_apic_timer_interrupt+0xa6/0x270 [12088.226861] ? apic_timer_interrupt+0xf/0x20 [12088.229376] [12088.230212] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [12088.232757] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12088.237619] do_raw_spin_lock+0x177/0x190 [12088.244460] _raw_spin_lock+0x21/0x30 [12088.248211] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12088.252184] obd_mod_free+0x46/0x140 [ptlrpc] [12088.255288] mdc_commit_open+0x91/0xb0 [mdc] [12088.259105] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12088.263772] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12088.265617] after_reply+0xe2c/0x1780 [ptlrpc] [12088.268574] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12088.275368] ptlrpcd+0xa82/0xe40 [ptlrpc] [12088.282636] ? do_wait_intr+0xf0/0xf0 [12088.283908] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12088.285821] kthread+0x1d1/0x200 [12088.292618] ? set_kthread_struct+0x70/0x70 [12088.294638] ret_from_fork+0x1f/0x30 [12093.407048] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10134s! [12093.410112] Showing busy workqueues and worker pools: [12093.411838] workqueue events: flags=0x0 [12093.413472] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12093.416203] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [12093.419563] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [12093.422070] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [12093.425860] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12093.428271] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [12093.430779] workqueue events_long: flags=0x0 [12093.432350] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12093.434757] pending: lu_objects_flush [obdclass] [12093.436810] workqueue events_unbound: flags=0x2 [12093.438377] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [12093.440532] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [12093.444436] workqueue events_power_efficient: flags=0x80 [12093.446348] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12093.448822] pending: neigh_periodic_work [12093.450486] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12093.452896] pending: do_cache_clean, check_lifetime [12093.454617] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12093.457459] pending: neigh_periodic_work [12093.459232] workqueue writeback: flags=0x4a [12093.460775] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [12093.463045] pending: wb_workfn, wb_workfn [12093.464475] workqueue xprtiod: flags=0xa [12093.465933] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [12093.468263] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [12093.471224] workqueue ipv6_addrconf: flags=0x40008 [12093.472937] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12093.474462] pending: addrconf_verify_work [12093.475861] workqueue knbd0-recv: flags=0x1a [12093.476909] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12093.478604] in-flight: 657:recv_work [12093.479840] workqueue ptlrpc_pinger: flags=0x2 [12093.481007] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12093.483087] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [12093.485232] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10134s workers=4 idle: 20831 12642 28859 [12093.487918] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12083s workers=2 idle: 659 [12094.123024] rcu: INFO: rcu_sched self-detected stall on CPU [12094.125814] rcu: 0-...!: (5390354 ticks this GP) idle=98a/1/0x4000000000000002 softirq=125154/125154 fqs=959 [12094.130891] (t=10140283 jiffies g=253585 q=6133) [12094.133235] rcu: rcu_sched kthread starved for 10135243 jiffies! g253585 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [12094.138919] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [12094.142793] rcu: RCU grace-period kthread stack dump: [12094.145225] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [12094.149355] Call Trace: [12094.150524] __schedule+0x351/0xcb0 [12094.152197] schedule+0xc0/0x180 [12094.153593] schedule_timeout+0xb4/0x190 [12094.155341] ? do_raw_spin_unlock+0x75/0x190 [12094.157196] ? __next_timer_interrupt+0x160/0x160 [12094.159274] ? prepare_to_swait_event+0xf3/0x140 [12094.161510] rcu_gp_kthread+0x8dd/0x15b0 [12094.163327] ? force_qs_rnp+0x2c0/0x2c0 [12094.165222] kthread+0x1d1/0x200 [12094.166674] ? set_kthread_struct+0x70/0x70 [12094.168517] ret_from_fork+0x1f/0x30 [12094.170200] rcu: Stack dump where RCU GP kthread last ran: [12094.171780] Sending NMI from CPU 0 to CPUs 1: [12094.180669] NMI backtrace for cpu 1 [12094.180676] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12094.180678] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12094.180680] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12094.180777] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12094.180779] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 [12094.180782] RAX: 0000000000004a35 RBX: ffff96848811cae0 RCX: 0000000000000001 [12094.180783] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [12094.180785] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [12094.180786] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [12094.180788] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [12094.180789] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [12094.180790] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12094.180792] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [12094.180793] Call Trace: [12094.180794] [12094.180795] ? show_regs.cold.9+0x22/0x2f [12094.180796] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [12094.180797] ? nmi_cpu_backtrace_handler+0x15/0x40 [12094.180848] ? nmi_handle+0xbd/0x210 [12094.180850] ? default_do_nmi+0x4b/0x1b0 [12094.180850] ? do_nmi+0x280/0x380 [12094.180851] ? end_repeat_nmi+0x16/0x1f [12094.180853] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12094.180854] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12094.180855] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12094.180856] [12094.180857] do_raw_spin_lock+0x177/0x190 [12094.180858] _raw_spin_lock+0x21/0x30 [12094.180859] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12094.180860] ? do_raw_spin_unlock+0x75/0x190 [12094.180861] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12094.180863] ? do_wait_intr+0xf0/0xf0 [12094.180863] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12094.180864] mdc_close+0x53b/0xd70 [mdc] [12094.180866] lmv_close+0x18d/0x490 [lmv] [12094.180867] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [12094.180868] ll_close_inode_openhandle+0x493/0xea0 [lustre] [12094.180869] ll_md_real_close+0xa5/0x2a0 [lustre] [12094.180870] ll_md_close+0x274/0x800 [lustre] [12094.180872] ll_file_release+0x128/0xc50 [lustre] [12094.180873] __fput+0x10d/0x440 [12094.180874] ____fput+0x12/0x20 [12094.180875] task_work_run+0xc6/0x110 [12094.180876] exit_to_usermode_loop+0x1dd/0x1f0 [12094.180877] do_syscall_64+0x3d6/0x440 [12094.180878] entry_SYSCALL_64_after_hwframe+0x49/0xae [12094.180879] RIP: 0033:0x7ff54af45cd8 [12094.180954] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [12094.180958] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [12094.181022] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [12094.181024] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [12094.181025] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [12094.181027] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [12094.181028] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12094.181105] NMI backtrace for cpu 0 [12094.323159] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12094.327307] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12094.331244] Call Trace: [12094.332310] [12094.333197] ? dump_stack+0xbb/0x10e [12094.334634] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [12094.336752] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [12094.338729] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [12094.340807] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [12094.343297] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [12094.346559] ? rcu_dump_cpu_stacks+0x11b/0x171 [12094.348501] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [12094.350837] ? update_process_times+0x6d/0xc0 [12094.352702] ? tick_sched_handle.isra.18+0x27/0x90 [12094.355049] ? tick_sched_timer+0x53/0xb0 [12094.357054] ? __hrtimer_run_queues+0x1c2/0x4d0 [12094.359240] ? tick_sched_handle.isra.18+0x90/0x90 [12094.361435] ? hrtimer_interrupt+0x128/0x2c0 [12094.363384] ? smp_apic_timer_interrupt+0xa6/0x270 [12094.365617] ? apic_timer_interrupt+0xf/0x20 [12094.367501] [12094.368462] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [12094.371162] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12094.373584] ? do_raw_spin_lock+0x177/0x190 [12094.375675] ? _raw_spin_lock+0x21/0x30 [12094.377380] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12094.380357] ? obd_mod_free+0x46/0x140 [ptlrpc] [12094.382907] ? mdc_commit_open+0x91/0xb0 [mdc] [12094.385184] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12094.387663] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12094.390371] ? after_reply+0xe2c/0x1780 [ptlrpc] [12094.393051] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12094.395807] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [12094.398129] ? do_wait_intr+0xf0/0xf0 [12094.399814] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12094.402371] ? kthread+0x1d1/0x200 [12094.403811] ? set_kthread_struct+0x70/0x70 [12094.405459] ? ret_from_fork+0x1f/0x30 [12100.059026] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [12100.064102] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12100.098459] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12100.104427] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12100.108996] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12100.115704] RIP: 0010:native_safe_halt+0x2e/0x50 [12100.124353] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12100.136917] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12100.141710] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [12100.144211] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c2134154 [12100.147227] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [12100.150777] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [12100.156379] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [12100.159772] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [12100.164949] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12100.168513] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [12100.172065] Call Trace: [12100.173224] [12100.174353] ? show_regs.cold.9+0x22/0x2f [12100.178256] ? watchdog_timer_fn.cold.6+0x62/0x10a [12100.182978] ? __hrtimer_run_queues+0x1c2/0x4d0 [12100.185733] ? watchdog+0x40/0x40 [12100.187429] ? hrtimer_interrupt+0x128/0x2c0 [12100.189351] ? smp_apic_timer_interrupt+0xa6/0x270 [12100.192206] ? apic_timer_interrupt+0xf/0x20 [12100.193999] [12100.194941] ? native_safe_halt+0x2e/0x50 [12100.196646] kvm_wait+0xcd/0xe0 [12100.200088] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12100.202621] do_raw_spin_lock+0x177/0x190 [12100.205347] _raw_spin_lock+0x21/0x30 [12100.207219] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12100.210302] process_one_work+0x2c8/0x700 [12100.211720] worker_thread+0x296/0x6e0 [12100.213657] ? rescuer_thread+0x570/0x570 [12100.215281] kthread+0x1d1/0x200 [12100.215923] ? set_kthread_struct+0x70/0x70 [12100.216804] ret_from_fork+0x1f/0x30 [12108.060588] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [12108.083105] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12108.120802] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12108.132652] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12108.136182] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12108.140261] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12108.156594] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12108.165578] RAX: 0000000000007f88 RBX: ffff96848811cae0 RCX: 0000000000000001 [12108.172681] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [12108.176188] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [12108.180718] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [12108.183249] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [12108.189744] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [12108.196336] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12108.200730] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [12108.206609] Call Trace: [12108.207410] [12108.208139] ? show_regs.cold.9+0x22/0x2f [12108.212953] ? watchdog_timer_fn.cold.6+0x62/0x10a [12108.216939] ? __hrtimer_run_queues+0x1c2/0x4d0 [12108.220150] ? watchdog+0x40/0x40 [12108.221360] ? hrtimer_interrupt+0x128/0x2c0 [12108.223091] ? smp_apic_timer_interrupt+0xa6/0x270 [12108.224764] ? apic_timer_interrupt+0xf/0x20 [12108.226453] [12108.227255] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12108.228960] do_raw_spin_lock+0x177/0x190 [12108.230580] _raw_spin_lock+0x21/0x30 [12108.232540] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12108.235201] ? do_raw_spin_unlock+0x75/0x190 [12108.236359] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12108.240056] ? do_wait_intr+0xf0/0xf0 [12108.242014] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12108.246184] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [12108.249362] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12108.253249] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12108.257474] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12108.259537] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [12108.263202] mdc_intent_lock+0x353/0x7e0 [mdc] [12108.265462] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12108.271514] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12108.279451] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12108.283950] lmv_intent_open+0x338/0x1150 [lmv] [12108.286463] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12108.289543] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12108.294164] lmv_intent_lock+0x1dd/0x4c0 [lmv] [12108.296379] ll_intent_lock+0xc5/0xaa0 [lustre] [12108.300795] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12108.304273] ll_lookup_it+0xe0e/0x2270 [lustre] [12108.307748] ? do_raw_spin_unlock+0x75/0x190 [12108.310429] ? _raw_spin_unlock+0x12/0x30 [12108.313112] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [12108.316169] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [12108.320082] ll_atomic_open+0x36a/0xd70 [lustre] [12108.326305] lookup_open+0x356/0xb40 [12108.333198] do_last+0x313/0xfc0 [12108.336175] ? nd_jump_root+0xe5/0x160 [12108.341492] ? path_init+0x437/0x520 [12108.343502] path_openat+0xf7/0x500 [12108.344780] ? do_raw_spin_unlock+0x75/0x190 [12108.352509] do_filp_open+0x99/0x140 [12108.355783] ? getname_flags+0x6e/0x330 [12108.362569] ? __check_object_size+0xff/0x256 [12108.366954] ? do_raw_spin_unlock+0x75/0x190 [12108.369489] ? _raw_spin_unlock+0x12/0x30 [12108.372072] do_sys_openat2+0x2b4/0x410 [12108.374742] do_sys_open+0x73/0xa0 [12108.378124] __x64_sys_openat+0x24/0x30 [12108.383141] do_syscall_64+0xc1/0x440 [12108.389051] entry_SYSCALL_64_after_hwframe+0x49/0xae [12108.391579] RIP: 0033:0x7ff54af45332 [12108.393533] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [12108.406556] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [12108.411678] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [12108.416142] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [12108.426708] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [12108.435569] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [12108.442217] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [12120.056749] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [12120.058693] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [12120.064114] Modules linked in: lustre(O) [12120.067741] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12120.067813] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12120.067820] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12120.067823] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12120.067836] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12120.067840] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12120.067844] RAX: 0000000000006417 RBX: ffff96848811cae0 RCX: 0000000000000001 [12120.067847] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [12120.067848] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [12120.067850] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [12120.067852] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [12120.067854] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [12120.067870] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12120.067873] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [12120.067879] Call Trace: [12120.067885] [12120.067890] ? show_regs.cold.9+0x22/0x2f [12120.067898] ? watchdog_timer_fn.cold.6+0x62/0x10a [12120.067905] ? __hrtimer_run_queues+0x1c2/0x4d0 [12120.067909] ? watchdog+0x40/0x40 [12120.067913] ? hrtimer_interrupt+0x128/0x2c0 [12120.067917] ? smp_apic_timer_interrupt+0xa6/0x270 [12120.067924] ? apic_timer_interrupt+0xf/0x20 [12120.067929] [12120.067932] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12120.067938] do_raw_spin_lock+0x177/0x190 [12120.067943] _raw_spin_lock+0x21/0x30 [12120.067947] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12120.068231] ? do_raw_spin_unlock+0x75/0x190 [12120.068237] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12120.068441] ? do_wait_intr+0xf0/0xf0 [12120.068450] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12120.068753] mdc_close+0x53b/0xd70 [mdc] [12120.068802] lmv_close+0x18d/0x490 [lmv] [12120.068830] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [12120.069001] ll_close_inode_openhandle+0x493/0xea0 [lustre] [12120.069114] ll_md_real_close+0xa5/0x2a0 [lustre] [12120.069203] ll_md_close+0x274/0x800 [lustre] [12120.069289] ll_file_release+0x128/0xc50 [lustre] [12120.069378] __fput+0x10d/0x440 [12120.069387] ____fput+0x12/0x20 [12120.069391] task_work_run+0xc6/0x110 [12120.069398] exit_to_usermode_loop+0x1dd/0x1f0 [12120.069406] do_syscall_64+0x3d6/0x440 [12120.069412] entry_SYSCALL_64_after_hwframe+0x49/0xae [12120.069419] RIP: 0033:0x7ff54af45cd8 [12120.069426] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [12120.069430] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [12120.069435] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [12120.069437] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [12120.069440] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [12120.069441] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [12120.069444] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12120.311308] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12120.331136] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12120.334196] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12120.337842] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [12120.340713] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [12120.349209] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12120.352516] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [12120.355841] RDX: 0000000000002ff0 RSI: 0000000000000003 RDI: ffff96848811cae0 [12120.358970] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [12120.362350] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [12120.365544] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12120.368518] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [12120.371399] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12120.373594] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [12120.376474] Call Trace: [12120.377485] [12120.378338] ? show_regs.cold.9+0x22/0x2f [12120.380582] ? watchdog_timer_fn.cold.6+0x62/0x10a [12120.382522] ? __hrtimer_run_queues+0x1c2/0x4d0 [12120.384341] ? watchdog+0x40/0x40 [12120.385663] ? hrtimer_interrupt+0x128/0x2c0 [12120.387431] ? smp_apic_timer_interrupt+0xa6/0x270 [12120.389327] ? apic_timer_interrupt+0xf/0x20 [12120.391039] [12120.391799] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [12120.394309] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12120.396555] do_raw_spin_lock+0x177/0x190 [12120.398505] _raw_spin_lock+0x21/0x30 [12120.400069] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12120.402210] obd_mod_free+0x46/0x140 [ptlrpc] [12120.404074] mdc_commit_open+0x91/0xb0 [mdc] [12120.405561] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12120.408016] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12120.410448] after_reply+0xe2c/0x1780 [ptlrpc] [12120.413133] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12120.415514] ptlrpcd+0xa82/0xe40 [ptlrpc] [12120.417736] ? do_wait_intr+0xf0/0xf0 [12120.419218] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12120.421523] kthread+0x1d1/0x200 [12120.422935] ? set_kthread_struct+0x70/0x70 [12120.425647] ret_from_fork+0x1f/0x30 [12123.615044] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10164s! [12123.618908] Showing busy workqueues and worker pools: [12123.621342] workqueue events: flags=0x0 [12123.623288] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12123.626356] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [12123.631255] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [12123.634786] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [12123.639778] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12123.642744] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [12123.646114] workqueue events_long: flags=0x0 [12123.648029] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12123.650683] pending: lu_objects_flush [obdclass] [12123.653182] workqueue events_unbound: flags=0x2 [12123.655280] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [12123.657844] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [12123.662355] workqueue events_power_efficient: flags=0x80 [12123.664848] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12123.667836] pending: neigh_periodic_work [12123.669991] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12123.673153] pending: do_cache_clean, check_lifetime [12123.675583] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12123.678526] pending: neigh_periodic_work [12123.680526] workqueue writeback: flags=0x4a [12123.682498] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [12123.685461] pending: wb_workfn, wb_workfn [12123.687502] workqueue xprtiod: flags=0xa [12123.689267] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [12123.692149] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [12123.695517] workqueue ipv6_addrconf: flags=0x40008 [12123.697706] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12123.700430] pending: addrconf_verify_work [12123.702954] workqueue knbd0-recv: flags=0x1a [12123.704841] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12123.707781] in-flight: 657:recv_work [12123.709514] workqueue ptlrpc_pinger: flags=0x2 [12123.711515] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12123.714439] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [12123.717682] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10164s workers=4 idle: 20831 12642 28859 [12123.722808] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12113s workers=2 idle: 659 [12128.061428] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [12128.066039] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12128.097382] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12128.103030] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12128.109540] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12128.113089] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12128.117584] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12128.146092] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12128.151140] RAX: 00000000000028f4 RBX: ffff96848811cae0 RCX: 0000000000000001 [12128.159541] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [12128.161488] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [12128.172105] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [12128.177874] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [12128.186325] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [12128.191984] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12128.195933] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [12128.204606] Call Trace: [12128.210049] [12128.213377] ? show_regs.cold.9+0x22/0x2f [12128.217131] ? watchdog_timer_fn.cold.6+0x62/0x10a [12128.220225] ? __hrtimer_run_queues+0x1c2/0x4d0 [12128.221378] ? watchdog+0x40/0x40 [12128.223229] ? hrtimer_interrupt+0x128/0x2c0 [12128.225929] ? smp_apic_timer_interrupt+0xa6/0x270 [12128.230563] ? apic_timer_interrupt+0xf/0x20 [12128.232653] [12128.233314] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12128.238029] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12128.239732] do_raw_spin_lock+0x177/0x190 [12128.250357] _raw_spin_lock+0x21/0x30 [12128.251495] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12128.260318] process_one_work+0x2c8/0x700 [12128.262087] worker_thread+0x296/0x6e0 [12128.266728] ? rescuer_thread+0x570/0x570 [12128.271329] kthread+0x1d1/0x200 [12128.276770] ? set_kthread_struct+0x70/0x70 [12128.281087] ret_from_fork+0x1f/0x30 [12136.061853] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [12136.066462] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12136.109123] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12136.123831] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12136.137945] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12136.142869] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12136.167609] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12136.175601] RAX: 0000000000001186 RBX: ffff96848811cae0 RCX: 0000000000000001 [12136.180774] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [12136.191514] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [12136.203665] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [12136.217547] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [12136.223361] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [12136.233505] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12136.245974] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [12136.256571] Call Trace: [12136.259280] [12136.260543] ? show_regs.cold.9+0x22/0x2f [12136.271522] ? watchdog_timer_fn.cold.6+0x62/0x10a [12136.278491] ? __hrtimer_run_queues+0x1c2/0x4d0 [12136.285260] ? watchdog+0x40/0x40 [12136.289426] ? hrtimer_interrupt+0x128/0x2c0 [12136.297422] ? smp_apic_timer_interrupt+0xa6/0x270 [12136.302114] ? apic_timer_interrupt+0xf/0x20 [12136.307125] [12136.308871] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12136.311131] do_raw_spin_lock+0x177/0x190 [12136.317419] _raw_spin_lock+0x21/0x30 [12136.323064] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12136.329598] ? do_raw_spin_unlock+0x75/0x190 [12136.333834] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12136.341332] ? do_wait_intr+0xf0/0xf0 [12136.344731] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12136.354483] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [12136.367458] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12136.375287] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12136.380354] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12136.388002] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [12136.397823] mdc_intent_lock+0x353/0x7e0 [mdc] [12136.402392] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12136.416343] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12136.432006] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12136.440600] lmv_intent_open+0x338/0x1150 [lmv] [12136.456634] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12136.458648] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12136.470696] lmv_intent_lock+0x1dd/0x4c0 [lmv] [12136.482301] ll_intent_lock+0xc5/0xaa0 [lustre] [12136.483657] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12136.493076] ll_lookup_it+0xe0e/0x2270 [lustre] [12136.498978] ? do_raw_spin_unlock+0x75/0x190 [12136.500972] ? _raw_spin_unlock+0x12/0x30 [12136.503888] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [12136.507310] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [12136.509891] ll_atomic_open+0x36a/0xd70 [lustre] [12136.511493] lookup_open+0x356/0xb40 [12136.514423] do_last+0x313/0xfc0 [12136.520976] ? nd_jump_root+0xe5/0x160 [12136.522436] ? path_init+0x437/0x520 [12136.523739] path_openat+0xf7/0x500 [12136.525975] ? do_raw_spin_unlock+0x75/0x190 [12136.528438] do_filp_open+0x99/0x140 [12136.532111] ? getname_flags+0x6e/0x330 [12136.537218] ? __check_object_size+0xff/0x256 [12136.541397] ? do_raw_spin_unlock+0x75/0x190 [12136.546190] ? _raw_spin_unlock+0x12/0x30 [12136.553339] do_sys_openat2+0x2b4/0x410 [12136.558132] do_sys_open+0x73/0xa0 [12136.559202] __x64_sys_openat+0x24/0x30 [12136.561749] do_syscall_64+0xc1/0x440 [12136.567929] entry_SYSCALL_64_after_hwframe+0x49/0xae [12136.575023] RIP: 0033:0x7ff54af45332 [12136.588737] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [12136.624634] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [12136.633954] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [12136.645950] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [12136.656420] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [12136.678052] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [12136.680472] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [12148.055456] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [12148.060438] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [12148.065258] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12148.065400] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12148.065408] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12148.065411] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [12148.065426] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [12148.065430] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12148.065434] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [12148.065437] RDX: 0000000000005c1c RSI: 0000000000000003 RDI: ffff96848811cae0 [12148.065439] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [12148.065441] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [12148.065443] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12148.065447] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [12148.065450] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12148.065452] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [12148.065460] Call Trace: [12148.065465] [12148.065471] ? show_regs.cold.9+0x22/0x2f [12148.065480] ? watchdog_timer_fn.cold.6+0x62/0x10a [12148.065487] ? __hrtimer_run_queues+0x1c2/0x4d0 [12148.073084] Modules linked in: [12148.115815] ? watchdog+0x40/0x40 [12148.115835] ? hrtimer_interrupt+0x128/0x2c0 [12148.115841] ? smp_apic_timer_interrupt+0xa6/0x270 [12148.115850] ? apic_timer_interrupt+0xf/0x20 [12148.115854] [12148.115858] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [12148.115865] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12148.115869] do_raw_spin_lock+0x177/0x190 [12148.115875] _raw_spin_lock+0x21/0x30 [12148.115882] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12148.116163] obd_mod_free+0x46/0x140 [ptlrpc] [12148.116354] mdc_commit_open+0x91/0xb0 [mdc] [12148.116392] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12148.116584] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12148.151016] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12148.151095] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12148.151103] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12148.151107] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12148.151123] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12148.151127] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12148.151132] RAX: 0000000000007912 RBX: ffff96848811cae0 RCX: 0000000000000001 [12148.151134] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [12148.151137] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [12148.151139] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [12148.151142] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [12148.151146] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [12148.151149] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12148.151151] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [12148.151159] Call Trace: [12148.151165] [12148.151170] ? show_regs.cold.9+0x22/0x2f [12148.151180] ? watchdog_timer_fn.cold.6+0x62/0x10a [12148.151186] ? __hrtimer_run_queues+0x1c2/0x4d0 [12148.151191] ? watchdog+0x40/0x40 [12148.151354] ? hrtimer_interrupt+0x128/0x2c0 [12148.151359] ? smp_apic_timer_interrupt+0xa6/0x270 [12148.151431] ? apic_timer_interrupt+0xf/0x20 [12148.151436] [12148.151439] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12148.151447] do_raw_spin_lock+0x177/0x190 [12148.151452] _raw_spin_lock+0x21/0x30 [12148.151457] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12148.153654] ? do_raw_spin_unlock+0x75/0x190 [12148.153663] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12148.171561] after_reply+0xe2c/0x1780 [ptlrpc] [12148.172217] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12148.172439] ptlrpcd+0xa82/0xe40 [ptlrpc] [12148.172661] ? do_wait_intr+0xf0/0xf0 [12148.172673] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12148.173095] kthread+0x1d1/0x200 [12148.173103] ? set_kthread_struct+0x70/0x70 [12148.173107] ret_from_fork+0x1f/0x30 [12148.644604] ? do_wait_intr+0xf0/0xf0 [12148.647479] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12148.653723] mdc_close+0x53b/0xd70 [mdc] [12148.655493] lmv_close+0x18d/0x490 [lmv] [12148.659608] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [12148.663472] ll_close_inode_openhandle+0x493/0xea0 [lustre] [12148.667794] ll_md_real_close+0xa5/0x2a0 [lustre] [12148.674241] ll_md_close+0x274/0x800 [lustre] [12148.676337] ll_file_release+0x128/0xc50 [lustre] [12148.681431] __fput+0x10d/0x440 [12148.685938] ____fput+0x12/0x20 [12148.688100] task_work_run+0xc6/0x110 [12148.693509] exit_to_usermode_loop+0x1dd/0x1f0 [12148.697349] do_syscall_64+0x3d6/0x440 [12148.700836] entry_SYSCALL_64_after_hwframe+0x49/0xae [12148.704617] RIP: 0033:0x7ff54af45cd8 [12148.707407] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [12148.717419] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [12148.721648] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [12148.725563] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [12148.729648] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [12148.736235] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [12148.747376] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12153.833866] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10194s! [12153.848436] Showing busy workqueues and worker pools: [12153.852599] workqueue events: flags=0x0 [12153.868458] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12153.886519] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [12153.909326] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [12153.925838] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [12153.934195] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12153.937306] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [12153.942327] workqueue events_long: flags=0x0 [12153.948422] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12153.962694] pending: lu_objects_flush [obdclass] [12153.977187] workqueue events_unbound: flags=0x2 [12153.994628] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [12154.002717] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [12154.014149] workqueue events_power_efficient: flags=0x80 [12154.017413] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12154.034097] pending: neigh_periodic_work [12154.044835] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12154.057308] pending: do_cache_clean, check_lifetime [12154.064027] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12154.071148] pending: neigh_periodic_work [12154.079739] workqueue writeback: flags=0x4a [12154.085335] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [12154.099732] pending: wb_workfn, wb_workfn [12154.105988] workqueue xprtiod: flags=0xa [12154.114026] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [12154.122179] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [12154.144112] workqueue ipv6_addrconf: flags=0x40008 [12154.145946] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12154.163655] pending: addrconf_verify_work [12154.172311] workqueue knbd0-recv: flags=0x1a [12154.174826] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12154.180676] in-flight: 657:recv_work [12154.185619] workqueue ptlrpc_pinger: flags=0x2 [12154.188942] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12154.193253] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [12154.200112] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10195s workers=4 idle: 20831 12642 28859 [12154.215379] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12144s workers=2 idle: 659 [12156.061607] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [12156.069193] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12156.110166] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12156.123929] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12156.132397] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12156.139386] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12156.145750] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12156.161007] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [12156.168167] RAX: 0000000000004b91 RBX: ffff96848811cae0 RCX: 0000000000000001 [12156.171259] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [12156.179296] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [12156.186251] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [12156.192161] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [12156.199689] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [12156.205490] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12156.210566] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [12156.214222] Call Trace: [12156.215434] [12156.216415] ? show_regs.cold.9+0x22/0x2f [12156.218444] ? watchdog_timer_fn.cold.6+0x62/0x10a [12156.220562] ? __hrtimer_run_queues+0x1c2/0x4d0 [12156.222775] ? watchdog+0x40/0x40 [12156.224429] ? hrtimer_interrupt+0x128/0x2c0 [12156.226402] ? smp_apic_timer_interrupt+0xa6/0x270 [12156.228676] ? apic_timer_interrupt+0xf/0x20 [12156.230481] [12156.231434] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12156.234075] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12156.236679] do_raw_spin_lock+0x177/0x190 [12156.238398] _raw_spin_lock+0x21/0x30 [12156.240075] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12156.242468] process_one_work+0x2c8/0x700 [12156.244092] worker_thread+0x296/0x6e0 [12156.246045] ? rescuer_thread+0x570/0x570 [12156.247760] kthread+0x1d1/0x200 [12156.249262] ? set_kthread_struct+0x70/0x70 [12156.251214] ret_from_fork+0x1f/0x30 [12164.062217] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [12164.068820] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12164.113251] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12164.120832] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12164.130692] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12164.133379] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12164.147060] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12164.152348] RAX: 0000000000004628 RBX: ffff96848811cae0 RCX: 0000000000000001 [12164.160500] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [12164.168729] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [12164.173135] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [12164.178709] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [12164.183697] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [12164.194844] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12164.199038] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [12164.203613] Call Trace: [12164.205085] [12164.206306] ? show_regs.cold.9+0x22/0x2f [12164.208441] ? watchdog_timer_fn.cold.6+0x62/0x10a [12164.211156] ? __hrtimer_run_queues+0x1c2/0x4d0 [12164.213664] ? watchdog+0x40/0x40 [12164.215771] ? hrtimer_interrupt+0x128/0x2c0 [12164.220869] ? smp_apic_timer_interrupt+0xa6/0x270 [12164.225764] ? apic_timer_interrupt+0xf/0x20 [12164.233408] [12164.236041] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12164.240133] do_raw_spin_lock+0x177/0x190 [12164.242758] _raw_spin_lock+0x21/0x30 [12164.245149] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12164.248719] ? do_raw_spin_unlock+0x75/0x190 [12164.251786] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12164.256817] ? do_wait_intr+0xf0/0xf0 [12164.259984] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12164.262928] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [12164.265636] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12164.268014] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12164.271348] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12164.273004] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [12164.274427] mdc_intent_lock+0x353/0x7e0 [mdc] [12164.277877] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12164.301929] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12164.315361] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12164.321353] lmv_intent_open+0x338/0x1150 [lmv] [12164.324226] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12164.326222] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12164.334032] lmv_intent_lock+0x1dd/0x4c0 [lmv] [12164.339491] ll_intent_lock+0xc5/0xaa0 [lustre] [12164.349783] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12164.351515] ll_lookup_it+0xe0e/0x2270 [lustre] [12164.359197] ? do_raw_spin_unlock+0x75/0x190 [12164.364061] ? _raw_spin_unlock+0x12/0x30 [12164.364958] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [12164.366098] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [12164.376682] ll_atomic_open+0x36a/0xd70 [lustre] [12164.379910] lookup_open+0x356/0xb40 [12164.382996] do_last+0x313/0xfc0 [12164.384985] ? nd_jump_root+0xe5/0x160 [12164.389004] ? path_init+0x437/0x520 [12164.393314] path_openat+0xf7/0x500 [12164.400018] ? do_raw_spin_unlock+0x75/0x190 [12164.413683] do_filp_open+0x99/0x140 [12164.418637] ? getname_flags+0x6e/0x330 [12164.429948] ? __check_object_size+0xff/0x256 [12164.434445] ? do_raw_spin_unlock+0x75/0x190 [12164.441480] ? _raw_spin_unlock+0x12/0x30 [12164.449498] do_sys_openat2+0x2b4/0x410 [12164.454785] do_sys_open+0x73/0xa0 [12164.456006] __x64_sys_openat+0x24/0x30 [12164.457402] do_syscall_64+0xc1/0x440 [12164.464165] entry_SYSCALL_64_after_hwframe+0x49/0xae [12164.468054] RIP: 0033:0x7ff54af45332 [12164.471237] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [12164.504392] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [12164.526333] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [12164.541683] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [12164.552525] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [12164.566985] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [12164.568934] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [12176.064335] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [12176.075990] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12176.111067] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12176.118071] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12176.130333] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12176.135036] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12176.155959] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12176.163934] RAX: 0000000000007689 RBX: ffff96848811cae0 RCX: 0000000000000001 [12176.173892] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [12176.185582] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [12176.193542] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [12176.198999] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [12176.207991] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [12176.213925] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12176.218327] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [12176.223736] Call Trace: [12176.226114] [12176.228680] ? show_regs.cold.9+0x22/0x2f [12176.232706] ? watchdog_timer_fn.cold.6+0x62/0x10a [12176.238545] ? __hrtimer_run_queues+0x1c2/0x4d0 [12176.241051] ? watchdog+0x40/0x40 [12176.246121] ? hrtimer_interrupt+0x128/0x2c0 [12176.254803] ? smp_apic_timer_interrupt+0xa6/0x270 [12176.262499] ? apic_timer_interrupt+0xf/0x20 [12176.265467] [12176.266333] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12176.269561] do_raw_spin_lock+0x177/0x190 [12176.272501] _raw_spin_lock+0x21/0x30 [12176.273830] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12176.288583] ? do_raw_spin_unlock+0x75/0x190 [12176.292723] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12176.295866] ? do_wait_intr+0xf0/0xf0 [12176.298867] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12176.305738] mdc_close+0x53b/0xd70 [mdc] [12176.310181] lmv_close+0x18d/0x490 [lmv] [12176.313780] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [12176.322473] ll_close_inode_openhandle+0x493/0xea0 [lustre] [12176.330686] ll_md_real_close+0xa5/0x2a0 [lustre] [12176.334154] ll_md_close+0x274/0x800 [lustre] [12176.337506] ll_file_release+0x128/0xc50 [lustre] [12176.341461] __fput+0x10d/0x440 [12176.343604] ____fput+0x12/0x20 [12176.346221] task_work_run+0xc6/0x110 [12176.348705] exit_to_usermode_loop+0x1dd/0x1f0 [12176.357584] do_syscall_64+0x3d6/0x440 [12176.362437] entry_SYSCALL_64_after_hwframe+0x49/0xae [12176.367857] RIP: 0033:0x7ff54af45cd8 [12176.374652] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [12176.406898] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [12176.419321] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [12176.432183] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [12176.442850] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [12176.447918] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [12176.466790] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12180.055018] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [12180.057233] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12180.080057] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12180.085945] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12180.090914] RIP: 0010:native_safe_halt+0x2e/0x50 [12180.093628] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12180.104603] RSP: 0000:ffffaf9fc1b97bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12180.108576] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [12180.112344] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff96848811cae0 [12180.116011] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [12180.119598] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [12180.123551] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12180.127605] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [12180.132133] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12180.134978] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [12180.138498] Call Trace: [12180.140187] [12180.141483] ? show_regs.cold.9+0x22/0x2f [12180.143937] ? watchdog_timer_fn.cold.6+0x62/0x10a [12180.146384] ? __hrtimer_run_queues+0x1c2/0x4d0 [12180.148943] ? watchdog+0x40/0x40 [12180.150664] ? hrtimer_interrupt+0x128/0x2c0 [12180.152936] ? smp_apic_timer_interrupt+0xa6/0x270 [12180.155555] ? apic_timer_interrupt+0xf/0x20 [12180.158287] [12180.159409] ? native_safe_halt+0x2e/0x50 [12180.162172] kvm_wait+0xcd/0xe0 [12180.165005] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12180.167954] do_raw_spin_lock+0x177/0x190 [12180.170325] _raw_spin_lock+0x21/0x30 [12180.172459] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12180.175268] obd_mod_free+0x46/0x140 [ptlrpc] [12180.177830] mdc_commit_open+0x91/0xb0 [mdc] [12180.179671] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12180.182698] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12180.185849] after_reply+0xe2c/0x1780 [ptlrpc] [12180.188105] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12180.191110] ptlrpcd+0xa82/0xe40 [ptlrpc] [12180.193784] ? do_wait_intr+0xf0/0xf0 [12180.195578] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12180.198341] kthread+0x1d1/0x200 [12180.200052] ? set_kthread_struct+0x70/0x70 [12180.202040] ret_from_fork+0x1f/0x30 [12184.059380] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [12184.067530] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12184.090764] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12184.098174] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12184.104383] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12184.109014] RIP: 0010:native_safe_halt+0x2e/0x50 [12184.111835] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12184.124699] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12184.130587] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [12184.139275] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c2134154 [12184.144371] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [12184.154454] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [12184.157463] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [12184.162031] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [12184.166744] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12184.177193] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [12184.187229] Call Trace: [12184.188518] [12184.190417] ? show_regs.cold.9+0x22/0x2f [12184.191931] ? watchdog_timer_fn.cold.6+0x62/0x10a [12184.195359] ? __hrtimer_run_queues+0x1c2/0x4d0 [12184.198766] ? watchdog+0x40/0x40 [12184.200951] ? hrtimer_interrupt+0x128/0x2c0 [12184.202885] ? smp_apic_timer_interrupt+0xa6/0x270 [12184.205438] ? apic_timer_interrupt+0xf/0x20 [12184.209046] [12184.209773] ? native_safe_halt+0x2e/0x50 [12184.211971] kvm_wait+0xcd/0xe0 [12184.215834] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12184.219559] do_raw_spin_lock+0x177/0x190 [12184.221393] _raw_spin_lock+0x21/0x30 [12184.223232] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12184.226852] process_one_work+0x2c8/0x700 [12184.230429] worker_thread+0x296/0x6e0 [12184.233925] ? rescuer_thread+0x570/0x570 [12184.237049] kthread+0x1d1/0x200 [12184.238075] ? set_kthread_struct+0x70/0x70 [12184.240805] ret_from_fork+0x1f/0x30 [12184.543615] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10225s! [12184.552782] Showing busy workqueues and worker pools: [12184.560747] workqueue events: flags=0x0 [12184.563672] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12184.568245] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [12184.574862] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [12184.583415] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [12184.590965] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12184.599865] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [12184.612044] workqueue events_long: flags=0x0 [12184.617202] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12184.624469] pending: lu_objects_flush [obdclass] [12184.631761] workqueue events_unbound: flags=0x2 [12184.635392] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [12184.643810] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [12184.649885] workqueue events_power_efficient: flags=0x80 [12184.652862] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12184.658270] pending: neigh_periodic_work [12184.660376] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12184.669014] pending: do_cache_clean, check_lifetime [12184.676838] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12184.681767] pending: neigh_periodic_work [12184.683616] workqueue writeback: flags=0x4a [12184.685501] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [12184.690107] pending: wb_workfn, wb_workfn [12184.695861] workqueue xprtiod: flags=0xa [12184.699391] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [12184.703251] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [12184.710220] workqueue ipv6_addrconf: flags=0x40008 [12184.718588] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12184.727044] pending: addrconf_verify_work [12184.729661] workqueue knbd0-recv: flags=0x1a [12184.730938] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12184.734069] in-flight: 657:recv_work [12184.735459] workqueue ptlrpc_pinger: flags=0x2 [12184.737366] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12184.742360] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [12184.751349] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10225s workers=4 idle: 20831 12642 28859 [12184.758616] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12174s workers=2 idle: 659 [12192.062251] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [12192.067243] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12192.094149] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12192.103799] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12192.109460] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12192.113397] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12192.123577] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [12192.127660] RAX: 0000000000007ad1 RBX: ffff96848811cae0 RCX: 0000000000000001 [12192.131308] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [12192.134272] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [12192.137341] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [12192.140525] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [12192.143392] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [12192.146475] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12192.148669] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [12192.151460] Call Trace: [12192.152433] [12192.153279] ? show_regs.cold.9+0x22/0x2f [12192.154733] ? watchdog_timer_fn.cold.6+0x62/0x10a [12192.159186] ? __hrtimer_run_queues+0x1c2/0x4d0 [12192.161565] ? watchdog+0x40/0x40 [12192.162665] ? hrtimer_interrupt+0x128/0x2c0 [12192.166972] ? smp_apic_timer_interrupt+0xa6/0x270 [12192.171314] ? apic_timer_interrupt+0xf/0x20 [12192.175233] [12192.180804] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12192.187585] do_raw_spin_lock+0x177/0x190 [12192.193042] _raw_spin_lock+0x21/0x30 [12192.195344] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12192.205282] ? do_raw_spin_unlock+0x75/0x190 [12192.213118] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12192.220248] ? do_wait_intr+0xf0/0xf0 [12192.224303] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12192.225979] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [12192.228958] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12192.231960] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12192.233850] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12192.238732] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [12192.244451] mdc_intent_lock+0x353/0x7e0 [mdc] [12192.247855] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12192.250630] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12192.253600] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12192.259574] lmv_intent_open+0x338/0x1150 [lmv] [12192.261441] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12192.264900] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12192.270458] lmv_intent_lock+0x1dd/0x4c0 [lmv] [12192.277757] ll_intent_lock+0xc5/0xaa0 [lustre] [12192.287810] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12192.293356] ll_lookup_it+0xe0e/0x2270 [lustre] [12192.295680] ? do_raw_spin_unlock+0x75/0x190 [12192.303368] ? _raw_spin_unlock+0x12/0x30 [12192.307610] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [12192.313860] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [12192.317303] ll_atomic_open+0x36a/0xd70 [lustre] [12192.321857] lookup_open+0x356/0xb40 [12192.326674] do_last+0x313/0xfc0 [12192.327751] ? nd_jump_root+0xe5/0x160 [12192.328844] ? path_init+0x437/0x520 [12192.335781] path_openat+0xf7/0x500 [12192.341196] ? do_raw_spin_unlock+0x75/0x190 [12192.347148] do_filp_open+0x99/0x140 [12192.351925] ? getname_flags+0x6e/0x330 [12192.354375] ? __check_object_size+0xff/0x256 [12192.356494] ? do_raw_spin_unlock+0x75/0x190 [12192.358531] ? _raw_spin_unlock+0x12/0x30 [12192.359966] do_sys_openat2+0x2b4/0x410 [12192.361427] do_sys_open+0x73/0xa0 [12192.363108] __x64_sys_openat+0x24/0x30 [12192.364457] do_syscall_64+0xc1/0x440 [12192.365457] entry_SYSCALL_64_after_hwframe+0x49/0xae [12192.367463] RIP: 0033:0x7ff54af45332 [12192.368961] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [12192.375347] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [12192.377492] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [12192.379636] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [12192.382196] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [12192.384918] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [12192.387452] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [12204.058017] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [12204.062042] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12204.094446] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12204.108427] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12204.116658] RIP: 0010:native_safe_halt+0x2e/0x50 [12204.121107] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12204.136994] RSP: 0018:ffffaf9fc1543ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12204.142097] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [12204.146576] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [12204.154151] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [12204.156715] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [12204.161891] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [12204.166675] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [12204.171520] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12204.176982] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [12204.182667] Call Trace: [12204.187444] [12204.192093] ? show_regs.cold.9+0x22/0x2f [12204.198894] ? watchdog_timer_fn.cold.6+0x62/0x10a [12204.202779] ? __hrtimer_run_queues+0x1c2/0x4d0 [12204.204254] ? watchdog+0x40/0x40 [12204.205033] ? hrtimer_interrupt+0x128/0x2c0 [12204.207300] ? smp_apic_timer_interrupt+0xa6/0x270 [12204.211578] ? apic_timer_interrupt+0xf/0x20 [12204.214509] [12204.215631] ? native_safe_halt+0x2e/0x50 [12204.217976] kvm_wait+0xcd/0xe0 [12204.219756] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12204.222643] do_raw_spin_lock+0x177/0x190 [12204.224987] _raw_spin_lock+0x21/0x30 [12204.227134] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12204.230379] ? do_raw_spin_unlock+0x75/0x190 [12204.232774] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12204.235835] ? do_wait_intr+0xf0/0xf0 [12204.238690] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12204.243118] mdc_close+0x53b/0xd70 [mdc] [12204.246408] lmv_close+0x18d/0x490 [lmv] [12204.249629] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [12204.255843] ll_close_inode_openhandle+0x493/0xea0 [lustre] [12204.261078] ll_md_real_close+0xa5/0x2a0 [lustre] [12204.265452] ll_md_close+0x274/0x800 [lustre] [12204.267073] ll_file_release+0x128/0xc50 [lustre] [12204.271539] __fput+0x10d/0x440 [12204.274243] ____fput+0x12/0x20 [12204.276707] task_work_run+0xc6/0x110 [12204.280078] exit_to_usermode_loop+0x1dd/0x1f0 [12204.282564] do_syscall_64+0x3d6/0x440 [12204.284746] entry_SYSCALL_64_after_hwframe+0x49/0xae [12204.288100] RIP: 0033:0x7ff54af45cd8 [12204.295143] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [12204.317915] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [12204.335284] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [12204.337876] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [12204.351981] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [12204.358760] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [12204.364160] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12212.058555] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [12212.059057] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [12212.066607] Modules linked in: lustre(O) [12212.069770] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12212.070024] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12212.070032] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12212.070038] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12212.070410] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12212.070422] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12212.070426] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12212.070431] RAX: 000000000000078a RBX: ffff96848811cae0 RCX: 0000000000000001 [12212.070433] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [12212.070436] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [12212.070438] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [12212.070440] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [12212.070443] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [12212.070445] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12212.070448] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [12212.070457] Call Trace: [12212.070463] [12212.070469] ? show_regs.cold.9+0x22/0x2f [12212.070480] ? watchdog_timer_fn.cold.6+0x62/0x10a [12212.070487] ? __hrtimer_run_queues+0x1c2/0x4d0 [12212.070492] ? watchdog+0x40/0x40 [12212.070497] ? hrtimer_interrupt+0x128/0x2c0 [12212.070501] ? smp_apic_timer_interrupt+0xa6/0x270 [12212.070510] ? apic_timer_interrupt+0xf/0x20 [12212.070515] [12212.070517] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12212.070523] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12212.070528] do_raw_spin_lock+0x177/0x190 [12212.070533] _raw_spin_lock+0x21/0x30 [12212.070538] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12212.070739] process_one_work+0x2c8/0x700 [12212.070746] worker_thread+0x296/0x6e0 [12212.070751] ? rescuer_thread+0x570/0x570 [12212.070754] kthread+0x1d1/0x200 [12212.070758] ? set_kthread_struct+0x70/0x70 [12212.070762] ret_from_fork+0x1f/0x30 [12212.261465] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12212.295787] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12212.312493] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12212.318751] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [12212.322644] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [12212.333412] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12212.337132] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [12212.340297] RDX: 0000000000007b41 RSI: 0000000000000003 RDI: ffff96848811cae0 [12212.343417] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [12212.346306] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [12212.349097] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12212.351590] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [12212.354899] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12212.357099] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [12212.360594] Call Trace: [12212.361695] [12212.362526] ? show_regs.cold.9+0x22/0x2f [12212.364200] ? watchdog_timer_fn.cold.6+0x62/0x10a [12212.366036] ? __hrtimer_run_queues+0x1c2/0x4d0 [12212.367931] ? watchdog+0x40/0x40 [12212.369210] ? hrtimer_interrupt+0x128/0x2c0 [12212.371088] ? smp_apic_timer_interrupt+0xa6/0x270 [12212.373282] ? apic_timer_interrupt+0xf/0x20 [12212.374917] [12212.375759] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [12212.377772] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12212.380045] do_raw_spin_lock+0x177/0x190 [12212.381475] _raw_spin_lock+0x21/0x30 [12212.383241] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12212.385474] obd_mod_free+0x46/0x140 [ptlrpc] [12212.387576] mdc_commit_open+0x91/0xb0 [mdc] [12212.389602] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12212.391880] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12212.394501] after_reply+0xe2c/0x1780 [ptlrpc] [12212.397316] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12212.399502] ptlrpcd+0xa82/0xe40 [ptlrpc] [12212.401405] ? do_wait_intr+0xf0/0xf0 [12212.402938] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12212.405305] kthread+0x1d1/0x200 [12212.406501] ? set_kthread_struct+0x70/0x70 [12212.408871] ret_from_fork+0x1f/0x30 [12215.263197] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10256s! [12215.273213] Showing busy workqueues and worker pools: [12215.279754] workqueue events: flags=0x0 [12215.284422] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12215.290316] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [12215.298864] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [12215.309425] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [12215.315492] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12215.321004] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [12215.328755] workqueue events_long: flags=0x0 [12215.332216] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12215.335928] pending: lu_objects_flush [obdclass] [12215.341507] workqueue events_unbound: flags=0x2 [12215.346981] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [12215.353716] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [12215.376913] workqueue events_power_efficient: flags=0x80 [12215.385885] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12215.393388] pending: neigh_periodic_work [12215.406968] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12215.418559] pending: do_cache_clean, check_lifetime [12215.427807] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12215.443769] pending: neigh_periodic_work [12215.446917] workqueue writeback: flags=0x4a [12215.455862] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [12215.471466] pending: wb_workfn, wb_workfn [12215.480563] workqueue xprtiod: flags=0xa [12215.486353] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [12215.499267] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [12215.506736] workqueue ipv6_addrconf: flags=0x40008 [12215.510470] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12215.514412] pending: addrconf_verify_work [12215.517585] workqueue knbd0-recv: flags=0x1a [12215.523611] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12215.533575] in-flight: 657:recv_work [12215.536079] workqueue ptlrpc_pinger: flags=0x2 [12215.550015] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12215.556660] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [12215.569072] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10256s workers=4 idle: 20831 12642 28859 [12215.582679] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12205s workers=2 idle: 659 [12220.060032] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [12220.063918] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12220.122554] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12220.135695] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12220.145041] RIP: 0010:native_safe_halt+0x2e/0x50 [12220.147716] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12220.171873] RSP: 0000:ffffaf9fc15234f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12220.174827] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [12220.184366] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c21b4154 [12220.194054] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [12220.205477] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [12220.213757] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [12220.221594] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [12220.226465] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12220.229899] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [12220.233346] Call Trace: [12220.234265] [12220.234952] ? show_regs.cold.9+0x22/0x2f [12220.237468] ? watchdog_timer_fn.cold.6+0x62/0x10a [12220.243083] ? __hrtimer_run_queues+0x1c2/0x4d0 [12220.248275] ? watchdog+0x40/0x40 [12220.250206] ? hrtimer_interrupt+0x128/0x2c0 [12220.253930] ? smp_apic_timer_interrupt+0xa6/0x270 [12220.255665] ? apic_timer_interrupt+0xf/0x20 [12220.264901] [12220.267172] ? native_safe_halt+0x2e/0x50 [12220.269581] kvm_wait+0xcd/0xe0 [12220.275277] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12220.277271] do_raw_spin_lock+0x177/0x190 [12220.278695] _raw_spin_lock+0x21/0x30 [12220.287198] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12220.295338] ? do_raw_spin_unlock+0x75/0x190 [12220.304965] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12220.312273] ? do_wait_intr+0xf0/0xf0 [12220.317303] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12220.329427] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [12220.344695] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12220.358701] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12220.373363] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12220.380006] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [12220.386625] mdc_intent_lock+0x353/0x7e0 [mdc] [12220.393530] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12220.398014] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12220.406041] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12220.411661] lmv_intent_open+0x338/0x1150 [lmv] [12220.418984] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12220.427255] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12220.431538] lmv_intent_lock+0x1dd/0x4c0 [lmv] [12220.442135] ll_intent_lock+0xc5/0xaa0 [lustre] [12220.457090] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12220.469632] ll_lookup_it+0xe0e/0x2270 [lustre] [12220.471411] ? do_raw_spin_unlock+0x75/0x190 [12220.476192] ? _raw_spin_unlock+0x12/0x30 [12220.480257] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [12220.484330] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [12220.490127] ll_atomic_open+0x36a/0xd70 [lustre] [12220.493351] lookup_open+0x356/0xb40 [12220.495623] do_last+0x313/0xfc0 [12220.500510] ? nd_jump_root+0xe5/0x160 [12220.508075] ? path_init+0x437/0x520 [12220.512012] path_openat+0xf7/0x500 [12220.517038] ? do_raw_spin_unlock+0x75/0x190 [12220.522299] do_filp_open+0x99/0x140 [12220.525245] ? getname_flags+0x6e/0x330 [12220.527993] ? __check_object_size+0xff/0x256 [12220.533320] ? do_raw_spin_unlock+0x75/0x190 [12220.539004] ? _raw_spin_unlock+0x12/0x30 [12220.550705] do_sys_openat2+0x2b4/0x410 [12220.559178] do_sys_open+0x73/0xa0 [12220.566460] __x64_sys_openat+0x24/0x30 [12220.581084] do_syscall_64+0xc1/0x440 [12220.591649] entry_SYSCALL_64_after_hwframe+0x49/0xae [12220.600829] RIP: 0033:0x7ff54af45332 [12220.604852] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [12220.629439] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [12220.638484] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [12220.645429] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [12220.649099] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [12220.658280] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [12220.664768] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [12232.059195] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [12232.068113] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12232.134535] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12232.149224] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12232.175556] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12232.178574] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12232.195441] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12232.205442] RAX: 0000000000003914 RBX: ffff96848811cae0 RCX: 0000000000000001 [12232.217759] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [12232.222401] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [12232.232605] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [12232.237666] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [12232.243294] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [12232.254373] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12232.257366] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [12232.261200] Call Trace: [12232.265026] [12232.266331] ? show_regs.cold.9+0x22/0x2f [12232.269608] ? watchdog_timer_fn.cold.6+0x62/0x10a [12232.272609] ? __hrtimer_run_queues+0x1c2/0x4d0 [12232.276038] ? watchdog+0x40/0x40 [12232.277424] ? hrtimer_interrupt+0x128/0x2c0 [12232.288976] ? smp_apic_timer_interrupt+0xa6/0x270 [12232.290620] ? apic_timer_interrupt+0xf/0x20 [12232.294228] [12232.295447] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12232.307496] do_raw_spin_lock+0x177/0x190 [12232.308936] _raw_spin_lock+0x21/0x30 [12232.318922] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12232.322423] ? do_raw_spin_unlock+0x75/0x190 [12232.324919] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12232.337066] ? do_wait_intr+0xf0/0xf0 [12232.341202] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12232.346971] mdc_close+0x53b/0xd70 [mdc] [12232.349334] lmv_close+0x18d/0x490 [lmv] [12232.352437] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [12232.356933] ll_close_inode_openhandle+0x493/0xea0 [lustre] [12232.364628] ll_md_real_close+0xa5/0x2a0 [lustre] [12232.367501] ll_md_close+0x274/0x800 [lustre] [12232.370282] ll_file_release+0x128/0xc50 [lustre] [12232.374582] __fput+0x10d/0x440 [12232.377573] ____fput+0x12/0x20 [12232.382507] task_work_run+0xc6/0x110 [12232.387832] exit_to_usermode_loop+0x1dd/0x1f0 [12232.397576] do_syscall_64+0x3d6/0x440 [12232.413313] entry_SYSCALL_64_after_hwframe+0x49/0xae [12232.417651] RIP: 0033:0x7ff54af45cd8 [12232.420722] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [12232.437425] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [12232.443032] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [12232.449038] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [12232.455570] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [12232.459488] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [12232.463090] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12240.055101] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [12240.059180] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [12240.063640] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12240.063731] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12240.063740] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12240.063743] RIP: 0010:native_safe_halt+0x2e/0x50 [12240.063756] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12240.063761] RSP: 0000:ffffaf9fc1b97bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12240.063765] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [12240.063768] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff96848811cae0 [12240.063770] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [12240.063772] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [12240.063774] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12240.063777] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [12240.063780] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12240.063783] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [12240.063791] Call Trace: [12240.063796] [12240.063802] ? show_regs.cold.9+0x22/0x2f [12240.063811] ? watchdog_timer_fn.cold.6+0x62/0x10a [12240.063818] ? __hrtimer_run_queues+0x1c2/0x4d0 [12240.063822] ? watchdog+0x40/0x40 [12240.063826] ? hrtimer_interrupt+0x128/0x2c0 [12240.063830] ? smp_apic_timer_interrupt+0xa6/0x270 [12240.063836] ? apic_timer_interrupt+0xf/0x20 [12240.063840] [12240.063842] ? native_safe_halt+0x2e/0x50 [12240.063846] kvm_wait+0xcd/0xe0 [12240.063853] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12240.063859] do_raw_spin_lock+0x177/0x190 [12240.063863] _raw_spin_lock+0x21/0x30 [12240.063867] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12240.070253] Modules linked in: lustre(O) [12240.123187] obd_mod_free+0x46/0x140 [ptlrpc] [12240.140185] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12240.140274] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12240.140281] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12240.140290] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12240.152412] mdc_commit_open+0x91/0xb0 [mdc] [12240.158457] [12240.158481] RIP: 0010:native_safe_halt+0x2e/0x50 [12240.158501] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12240.158505] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12240.158511] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [12240.158513] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c2134154 [12240.158516] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [12240.158677] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [12240.158680] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [12240.158684] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [12240.158687] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12240.158689] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [12240.158705] Call Trace: [12240.158711] [12240.158718] ? show_regs.cold.9+0x22/0x2f [12240.158729] ? watchdog_timer_fn.cold.6+0x62/0x10a [12240.158817] ? __hrtimer_run_queues+0x1c2/0x4d0 [12240.158822] ? watchdog+0x40/0x40 [12240.158827] ? hrtimer_interrupt+0x128/0x2c0 [12240.158830] ? smp_apic_timer_interrupt+0xa6/0x270 [12240.158838] ? apic_timer_interrupt+0xf/0x20 [12240.158842] [12240.158845] ? native_safe_halt+0x2e/0x50 [12240.158849] kvm_wait+0xcd/0xe0 [12240.158857] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12240.158868] do_raw_spin_lock+0x177/0x190 [12240.158872] _raw_spin_lock+0x21/0x30 [12240.158876] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12240.159530] process_one_work+0x2c8/0x700 [12240.159538] worker_thread+0x296/0x6e0 [12240.159542] ? rescuer_thread+0x570/0x570 [12240.159546] kthread+0x1d1/0x200 [12240.159549] ? set_kthread_struct+0x70/0x70 [12240.159559] ret_from_fork+0x1f/0x30 [12240.690853] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12240.700079] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12240.710237] after_reply+0xe2c/0x1780 [ptlrpc] [12240.714544] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12240.717979] ptlrpcd+0xa82/0xe40 [ptlrpc] [12240.726932] ? do_wait_intr+0xf0/0xf0 [12240.731427] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12240.739039] kthread+0x1d1/0x200 [12240.742134] ? set_kthread_struct+0x70/0x70 [12240.754385] ret_from_fork+0x1f/0x30 [12245.986219] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10287s! [12246.002818] Showing busy workqueues and worker pools: [12246.005824] workqueue events: flags=0x0 [12246.008838] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12246.019366] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [12246.030834] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [12246.037279] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [12246.043260] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12246.046275] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [12246.050559] workqueue events_long: flags=0x0 [12246.053732] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12246.056797] pending: lu_objects_flush [obdclass] [12246.059311] workqueue events_unbound: flags=0x2 [12246.062365] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [12246.066949] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [12246.077293] workqueue events_power_efficient: flags=0x80 [12246.080914] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12246.087396] pending: neigh_periodic_work [12246.093687] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12246.103004] pending: do_cache_clean, check_lifetime [12246.112345] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12246.114604] pending: neigh_periodic_work [12246.119860] workqueue writeback: flags=0x4a [12246.122689] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [12246.130324] pending: wb_workfn, wb_workfn [12246.136457] workqueue xprtiod: flags=0xa [12246.144240] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [12246.146648] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [12246.149869] workqueue ipv6_addrconf: flags=0x40008 [12246.151319] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12246.154297] pending: addrconf_verify_work [12246.155831] workqueue knbd0-recv: flags=0x1a [12246.157365] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12246.160289] in-flight: 657:recv_work [12246.162048] workqueue ptlrpc_pinger: flags=0x2 [12246.163434] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12246.165352] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [12246.167437] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10287s workers=4 idle: 20831 12642 28859 [12246.171257] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12236s workers=2 idle: 659 [12248.063931] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [12248.074094] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12248.129008] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12248.138573] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12248.141938] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12248.151047] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12248.160205] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [12248.164109] RAX: 0000000000002f08 RBX: ffff96848811cae0 RCX: 0000000000000001 [12248.171214] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [12248.176936] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [12248.180929] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [12248.189588] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [12248.193156] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [12248.197207] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12248.200067] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [12248.203918] Call Trace: [12248.204602] [12248.205404] ? show_regs.cold.9+0x22/0x2f [12248.206982] ? watchdog_timer_fn.cold.6+0x62/0x10a [12248.208905] ? __hrtimer_run_queues+0x1c2/0x4d0 [12248.210544] ? watchdog+0x40/0x40 [12248.212364] ? hrtimer_interrupt+0x128/0x2c0 [12248.216692] ? smp_apic_timer_interrupt+0xa6/0x270 [12248.219421] ? apic_timer_interrupt+0xf/0x20 [12248.224517] [12248.225278] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12248.227365] do_raw_spin_lock+0x177/0x190 [12248.229883] _raw_spin_lock+0x21/0x30 [12248.231397] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12248.233764] ? do_raw_spin_unlock+0x75/0x190 [12248.235748] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12248.238323] ? do_wait_intr+0xf0/0xf0 [12248.242355] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12248.245713] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [12248.248712] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12248.252462] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12248.255469] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12248.259145] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [12248.261559] mdc_intent_lock+0x353/0x7e0 [mdc] [12248.263650] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12248.266096] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12248.273356] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12248.278583] lmv_intent_open+0x338/0x1150 [lmv] [12248.279994] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12248.286161] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12248.297293] lmv_intent_lock+0x1dd/0x4c0 [lmv] [12248.303564] ll_intent_lock+0xc5/0xaa0 [lustre] [12248.306669] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12248.309673] ll_lookup_it+0xe0e/0x2270 [lustre] [12248.317199] ? do_raw_spin_unlock+0x75/0x190 [12248.321419] ? _raw_spin_unlock+0x12/0x30 [12248.325943] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [12248.331777] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [12248.337415] ll_atomic_open+0x36a/0xd70 [lustre] [12248.339118] lookup_open+0x356/0xb40 [12248.341350] do_last+0x313/0xfc0 [12248.342927] ? nd_jump_root+0xe5/0x160 [12248.344677] ? path_init+0x437/0x520 [12248.346381] path_openat+0xf7/0x500 [12248.348000] ? do_raw_spin_unlock+0x75/0x190 [12248.353139] do_filp_open+0x99/0x140 [12248.358050] ? getname_flags+0x6e/0x330 [12248.359771] ? __check_object_size+0xff/0x256 [12248.364773] ? do_raw_spin_unlock+0x75/0x190 [12248.367055] ? _raw_spin_unlock+0x12/0x30 [12248.369100] do_sys_openat2+0x2b4/0x410 [12248.370484] do_sys_open+0x73/0xa0 [12248.371489] __x64_sys_openat+0x24/0x30 [12248.373755] do_syscall_64+0xc1/0x440 [12248.375079] entry_SYSCALL_64_after_hwframe+0x49/0xae [12248.378096] RIP: 0033:0x7ff54af45332 [12248.379799] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [12248.389428] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [12248.392802] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [12248.396064] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [12248.398723] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [12248.401099] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [12248.406795] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [12260.059643] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [12260.069188] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12260.116166] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12260.127356] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12260.140707] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12260.147230] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12260.164030] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12260.170712] RAX: 00000000000067b3 RBX: ffff96848811cae0 RCX: 0000000000000001 [12260.178949] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [12260.187703] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [12260.199310] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [12260.217035] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [12260.227696] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [12260.235427] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12260.242563] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [12260.257080] Call Trace: [12260.262010] [12260.263100] ? show_regs.cold.9+0x22/0x2f [12260.266051] ? watchdog_timer_fn.cold.6+0x62/0x10a [12260.268489] ? __hrtimer_run_queues+0x1c2/0x4d0 [12260.271242] ? watchdog+0x40/0x40 [12260.279153] ? hrtimer_interrupt+0x128/0x2c0 [12260.286848] ? smp_apic_timer_interrupt+0xa6/0x270 [12260.288404] ? apic_timer_interrupt+0xf/0x20 [12260.294396] [12260.296276] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12260.301159] do_raw_spin_lock+0x177/0x190 [12260.305708] _raw_spin_lock+0x21/0x30 [12260.307110] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12260.314349] ? do_raw_spin_unlock+0x75/0x190 [12260.319994] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12260.321847] ? do_wait_intr+0xf0/0xf0 [12260.327409] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12260.336652] mdc_close+0x53b/0xd70 [mdc] [12260.341210] lmv_close+0x18d/0x490 [lmv] [12260.344118] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [12260.350351] ll_close_inode_openhandle+0x493/0xea0 [lustre] [12260.354458] ll_md_real_close+0xa5/0x2a0 [lustre] [12260.360031] ll_md_close+0x274/0x800 [lustre] [12260.361884] ll_file_release+0x128/0xc50 [lustre] [12260.363344] __fput+0x10d/0x440 [12260.366584] ____fput+0x12/0x20 [12260.369440] task_work_run+0xc6/0x110 [12260.373546] exit_to_usermode_loop+0x1dd/0x1f0 [12260.375210] do_syscall_64+0x3d6/0x440 [12260.380509] entry_SYSCALL_64_after_hwframe+0x49/0xae [12260.381605] RIP: 0033:0x7ff54af45cd8 [12260.382397] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [12260.391536] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [12260.395542] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [12260.399928] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [12260.402494] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [12260.404752] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [12260.407314] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12268.060827] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [12268.069576] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12268.125438] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12268.135576] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12268.141562] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12268.145350] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12268.147300] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12268.158776] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12268.163122] RAX: 000000000000792f RBX: ffff96848811cae0 RCX: 0000000000000001 [12268.166554] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [12268.170388] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [12268.174270] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [12268.178004] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [12268.182498] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [12268.186905] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12268.189561] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [12268.193222] Call Trace: [12268.194576] [12268.195720] ? show_regs.cold.9+0x22/0x2f [12268.197883] ? watchdog_timer_fn.cold.6+0x62/0x10a [12268.200381] ? __hrtimer_run_queues+0x1c2/0x4d0 [12268.202824] ? watchdog+0x40/0x40 [12268.204823] ? hrtimer_interrupt+0x128/0x2c0 [12268.207938] ? smp_apic_timer_interrupt+0xa6/0x270 [12268.210295] ? apic_timer_interrupt+0xf/0x20 [12268.212678] [12268.213768] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12268.216682] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12268.218961] do_raw_spin_lock+0x177/0x190 [12268.221045] _raw_spin_lock+0x21/0x30 [12268.222865] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12268.225039] process_one_work+0x2c8/0x700 [12268.227374] worker_thread+0x296/0x6e0 [12268.229791] ? rescuer_thread+0x570/0x570 [12268.231930] kthread+0x1d1/0x200 [12268.233352] ? set_kthread_struct+0x70/0x70 [12268.235115] ret_from_fork+0x1f/0x30 [12272.056786] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [12272.078804] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12272.147852] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12272.157714] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12272.169149] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [12272.179758] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [12272.202925] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12272.209032] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [12272.214306] RDX: 0000000000004950 RSI: 0000000000000003 RDI: ffff96848811cae0 [12272.221159] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [12272.231899] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [12272.236807] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12272.242855] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [12272.251430] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12272.254778] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [12272.260681] Call Trace: [12272.262127] [12272.263480] ? show_regs.cold.9+0x22/0x2f [12272.269711] ? watchdog_timer_fn.cold.6+0x62/0x10a [12272.272095] ? __hrtimer_run_queues+0x1c2/0x4d0 [12272.278028] ? watchdog+0x40/0x40 [12272.282134] ? hrtimer_interrupt+0x128/0x2c0 [12272.285859] ? smp_apic_timer_interrupt+0xa6/0x270 [12272.289577] ? apic_timer_interrupt+0xf/0x20 [12272.291872] [12272.293026] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [12272.296053] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12272.301009] do_raw_spin_lock+0x177/0x190 [12272.302084] _raw_spin_lock+0x21/0x30 [12272.303135] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12272.305911] obd_mod_free+0x46/0x140 [ptlrpc] [12272.308452] mdc_commit_open+0x91/0xb0 [mdc] [12272.310595] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12272.313773] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12272.317081] after_reply+0xe2c/0x1780 [ptlrpc] [12272.318965] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12272.325114] ptlrpcd+0xa82/0xe40 [ptlrpc] [12272.328512] ? do_wait_intr+0xf0/0xf0 [12272.332928] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12272.342591] kthread+0x1d1/0x200 [12272.347095] ? set_kthread_struct+0x70/0x70 [12272.352067] ret_from_fork+0x1f/0x30 [12274.126676] rcu: INFO: rcu_sched self-detected stall on CPU [12274.129607] rcu: 0-...!: (5485405 ticks this GP) idle=98a/1/0x4000000000000002 softirq=125154/125154 fqs=959 [12274.143209] (t=10320286 jiffies g=253585 q=6179) [12274.146169] rcu: rcu_sched kthread starved for 10315246 jiffies! g253585 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [12274.163742] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [12274.174845] rcu: RCU grace-period kthread stack dump: [12274.180417] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [12274.204002] Call Trace: [12274.208776] __schedule+0x351/0xcb0 [12274.220112] schedule+0xc0/0x180 [12274.226897] schedule_timeout+0xb4/0x190 [12274.231127] ? do_raw_spin_unlock+0x75/0x190 [12274.234862] ? __next_timer_interrupt+0x160/0x160 [12274.237115] ? prepare_to_swait_event+0xf3/0x140 [12274.239842] rcu_gp_kthread+0x8dd/0x15b0 [12274.242161] ? force_qs_rnp+0x2c0/0x2c0 [12274.244831] kthread+0x1d1/0x200 [12274.246372] ? set_kthread_struct+0x70/0x70 [12274.249462] ret_from_fork+0x1f/0x30 [12274.251349] rcu: Stack dump where RCU GP kthread last ran: [12274.254493] Sending NMI from CPU 0 to CPUs 1: [12274.261317] NMI backtrace for cpu 1 skipped: idling at native_safe_halt+0x2e/0x50 [12274.264208] NMI backtrace for cpu 0 [12274.272379] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12274.294670] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12274.308661] Call Trace: [12274.310953] [12274.314521] ? dump_stack+0xbb/0x10e [12274.318443] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [12274.325381] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [12274.330429] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [12274.334185] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [12274.337189] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [12274.352529] ? rcu_dump_cpu_stacks+0x11b/0x171 [12274.361409] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [12274.367268] ? update_process_times+0x6d/0xc0 [12274.371410] ? tick_sched_handle.isra.18+0x27/0x90 [12274.377194] ? tick_sched_timer+0x53/0xb0 [12274.379542] ? __hrtimer_run_queues+0x1c2/0x4d0 [12274.385919] ? tick_sched_handle.isra.18+0x90/0x90 [12274.389241] ? hrtimer_interrupt+0x128/0x2c0 [12274.399501] ? smp_apic_timer_interrupt+0xa6/0x270 [12274.409664] ? apic_timer_interrupt+0xf/0x20 [12274.417332] [12274.425692] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [12274.437101] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12274.440980] ? do_raw_spin_lock+0x177/0x190 [12274.453945] ? _raw_spin_lock+0x21/0x30 [12274.457170] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12274.460963] ? obd_mod_free+0x46/0x140 [ptlrpc] [12274.465134] ? mdc_commit_open+0x91/0xb0 [mdc] [12274.466348] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12274.477535] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12274.481656] ? after_reply+0xe2c/0x1780 [ptlrpc] [12274.487006] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12274.495149] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [12274.497494] ? do_wait_intr+0xf0/0xf0 [12274.500535] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12274.504439] ? kthread+0x1d1/0x200 [12274.506653] ? set_kthread_struct+0x70/0x70 [12274.509551] ? ret_from_fork+0x1f/0x30 [12276.060068] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [12276.066770] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12276.119035] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12276.131807] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12276.139050] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12276.145656] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12276.158172] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12276.163675] RAX: 000000000000721e RBX: ffff96848811cae0 RCX: 0000000000000001 [12276.168152] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [12276.171226] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [12276.174901] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [12276.178647] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [12276.182792] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [12276.186836] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12276.189624] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [12276.191884] BUG: workqueue lockup - pool [12276.192851] Call Trace: [12276.192863] [12276.192868] ? show_regs.cold.9+0x22/0x2f [12276.192884] ? watchdog_timer_fn.cold.6+0x62/0x10a [12276.192891] ? __hrtimer_run_queues+0x1c2/0x4d0 [12276.192895] ? watchdog+0x40/0x40 [12276.192899] ? hrtimer_interrupt+0x128/0x2c0 [12276.192903] ? smp_apic_timer_interrupt+0xa6/0x270 [12276.193030] ? apic_timer_interrupt+0xf/0x20 [12276.193035] [12276.193039] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12276.193049] do_raw_spin_lock+0x177/0x190 [12276.193055] _raw_spin_lock+0x21/0x30 [12276.193060] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12276.193562] ? do_raw_spin_unlock+0x75/0x190 [12276.193570] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12276.193792] ? do_wait_intr+0xf0/0xf0 [12276.193801] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12276.194089] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [12276.194452] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12276.194764] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12276.194895] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12276.195072] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [12276.195102] mdc_intent_lock+0x353/0x7e0 [mdc] [12276.195132] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12276.195225] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12276.195473] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12276.195635] lmv_intent_open+0x338/0x1150 [lmv] [12276.195670] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12276.195767] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12276.195860] lmv_intent_lock+0x1dd/0x4c0 [lmv] [12276.195885] ll_intent_lock+0xc5/0xaa0 [lustre] [12276.195983] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12276.196066] ll_lookup_it+0xe0e/0x2270 [lustre] [12276.196149] ? do_raw_spin_unlock+0x75/0x190 [12276.196157] ? _raw_spin_unlock+0x12/0x30 [12276.196164] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [12276.196405] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [12276.196425] ll_atomic_open+0x36a/0xd70 [lustre] [12276.196522] lookup_open+0x356/0xb40 [12276.196532] do_last+0x313/0xfc0 [12276.196537] ? nd_jump_root+0xe5/0x160 [12276.196542] ? path_init+0x437/0x520 [12276.196546] path_openat+0xf7/0x500 [12276.196551] ? do_raw_spin_unlock+0x75/0x190 [12276.196556] do_filp_open+0x99/0x140 [12276.196564] ? getname_flags+0x6e/0x330 [12276.196568] ? __check_object_size+0xff/0x256 [12276.196576] ? do_raw_spin_unlock+0x75/0x190 [12276.196581] ? _raw_spin_unlock+0x12/0x30 [12276.196586] do_sys_openat2+0x2b4/0x410 [12276.196591] do_sys_open+0x73/0xa0 [12276.196596] __x64_sys_openat+0x24/0x30 [12276.196599] do_syscall_64+0xc1/0x440 [12276.199672] cpus=0-3 [12276.200662] entry_SYSCALL_64_after_hwframe+0x49/0xae [12276.200685] RIP: 0033:0x7ff54af45332 [12276.200697] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [12276.200701] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [12276.200706] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [12276.200709] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [12276.200711] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [12276.200714] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [12276.200716] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [12276.527728] flags=0x4 nice=0 stuck for 10317s! [12276.539484] Showing busy workqueues and worker pools: [12276.541850] workqueue events: flags=0x0 [12276.543272] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12276.555054] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [12276.576622] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [12276.587369] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [12276.604468] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12276.614452] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [12276.620736] workqueue events_long: flags=0x0 [12276.626370] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12276.638684] pending: lu_objects_flush [obdclass] [12276.644070] workqueue events_unbound: flags=0x2 [12276.647569] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [12276.651044] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [12276.663280] workqueue events_power_efficient: flags=0x80 [12276.669955] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12276.676556] pending: neigh_periodic_work [12276.678209] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12276.691096] pending: do_cache_clean, check_lifetime [12276.696925] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12276.703096] pending: neigh_periodic_work [12276.713375] workqueue writeback: flags=0x4a [12276.724375] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [12276.734985] pending: wb_workfn, wb_workfn [12276.739447] workqueue xprtiod: flags=0xa [12276.746695] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [12276.760278] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [12276.781415] workqueue ipv6_addrconf: flags=0x40008 [12276.792320] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12276.796055] pending: addrconf_verify_work [12276.800938] workqueue knbd0-recv: flags=0x1a [12276.806068] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12276.815471] in-flight: 657:recv_work [12276.819681] workqueue ptlrpc_pinger: flags=0x2 [12276.821505] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12276.830349] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [12276.835089] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10317s workers=4 idle: 20831 12642 28859 [12276.838979] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12267s workers=2 idle: 659 [12296.059025] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [12296.062237] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12296.113617] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12296.120242] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12296.129086] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12296.136901] RIP: 0010:native_safe_halt+0x2e/0x50 [12296.138144] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12296.165131] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12296.167722] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [12296.184895] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c2134154 [12296.193462] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [12296.204657] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [12296.209840] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [12296.224016] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [12296.242317] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12296.253406] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [12296.261335] Call Trace: [12296.267363] [12296.270427] ? show_regs.cold.9+0x22/0x2f [12296.274967] ? watchdog_timer_fn.cold.6+0x62/0x10a [12296.276740] ? __hrtimer_run_queues+0x1c2/0x4d0 [12296.288960] ? watchdog+0x40/0x40 [12296.292509] ? hrtimer_interrupt+0x128/0x2c0 [12296.299978] ? smp_apic_timer_interrupt+0xa6/0x270 [12296.306498] ? apic_timer_interrupt+0xf/0x20 [12296.308025] [12296.308779] ? native_safe_halt+0x2e/0x50 [12296.322440] kvm_wait+0xcd/0xe0 [12296.327485] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12296.333123] do_raw_spin_lock+0x177/0x190 [12296.334566] _raw_spin_lock+0x21/0x30 [12296.345255] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12296.356454] process_one_work+0x2c8/0x700 [12296.362464] worker_thread+0x296/0x6e0 [12296.370513] ? rescuer_thread+0x570/0x570 [12296.376572] kthread+0x1d1/0x200 [12296.384520] ? set_kthread_struct+0x70/0x70 [12296.394451] ret_from_fork+0x1f/0x30 [12300.058040] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [12300.061722] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12300.089721] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12300.094767] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12300.099444] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12300.102472] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12300.125611] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12300.137739] RAX: 0000000000007277 RBX: ffff96848811cae0 RCX: 0000000000000001 [12300.139633] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [12300.144345] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [12300.150436] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [12300.153418] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [12300.156598] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [12300.161383] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12300.165888] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [12300.170444] Call Trace: [12300.172284] [12300.173283] ? show_regs.cold.9+0x22/0x2f [12300.175747] ? watchdog_timer_fn.cold.6+0x62/0x10a [12300.179181] ? __hrtimer_run_queues+0x1c2/0x4d0 [12300.185154] ? watchdog+0x40/0x40 [12300.187514] ? hrtimer_interrupt+0x128/0x2c0 [12300.195168] ? smp_apic_timer_interrupt+0xa6/0x270 [12300.196946] ? apic_timer_interrupt+0xf/0x20 [12300.202275] [12300.203718] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12300.207068] do_raw_spin_lock+0x177/0x190 [12300.208132] _raw_spin_lock+0x21/0x30 [12300.213310] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12300.217517] ? do_raw_spin_unlock+0x75/0x190 [12300.220634] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12300.224446] ? do_wait_intr+0xf0/0xf0 [12300.226538] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12300.229844] mdc_close+0x53b/0xd70 [mdc] [12300.231740] lmv_close+0x18d/0x490 [lmv] [12300.234462] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [12300.237934] ll_close_inode_openhandle+0x493/0xea0 [lustre] [12300.241486] ll_md_real_close+0xa5/0x2a0 [lustre] [12300.244866] ll_md_close+0x274/0x800 [lustre] [12300.248325] ll_file_release+0x128/0xc50 [lustre] [12300.251482] __fput+0x10d/0x440 [12300.253549] ____fput+0x12/0x20 [12300.255808] task_work_run+0xc6/0x110 [12300.258205] exit_to_usermode_loop+0x1dd/0x1f0 [12300.260864] do_syscall_64+0x3d6/0x440 [12300.263717] entry_SYSCALL_64_after_hwframe+0x49/0xae [12300.266741] RIP: 0033:0x7ff54af45cd8 [12300.268673] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [12300.284321] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [12300.290825] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [12300.296744] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [12300.301402] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [12300.306808] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [12300.310834] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12304.056091] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [12304.060563] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [12304.070581] Modules linked in: [12304.072794] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12304.072876] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12304.072884] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12304.072887] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12304.072900] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12304.072904] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12304.072910] RAX: 0000000000007eb0 RBX: ffff96848811cae0 RCX: 0000000000000001 [12304.072912] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [12304.072915] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [12304.072917] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [12304.072919] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [12304.072923] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [12304.072926] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12304.072928] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [12304.072937] Call Trace: [12304.072943] [12304.072949] ? show_regs.cold.9+0x22/0x2f [12304.072958] ? watchdog_timer_fn.cold.6+0x62/0x10a [12304.072964] ? __hrtimer_run_queues+0x1c2/0x4d0 [12304.072968] ? watchdog+0x40/0x40 [12304.072973] ? hrtimer_interrupt+0x128/0x2c0 [12304.072977] ? smp_apic_timer_interrupt+0xa6/0x270 [12304.072987] ? apic_timer_interrupt+0xf/0x20 [12304.072991] [12304.072994] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12304.073000] do_raw_spin_lock+0x177/0x190 [12304.073005] _raw_spin_lock+0x21/0x30 [12304.073009] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12304.073353] ? do_raw_spin_unlock+0x75/0x190 [12304.073359] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12304.073576] ? do_wait_intr+0xf0/0xf0 [12304.073585] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12304.073802] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [12304.074027] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12304.074278] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12304.074397] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12304.074441] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [12304.074475] mdc_intent_lock+0x353/0x7e0 [mdc] [12304.074505] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12304.074595] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12304.074834] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12304.074867] lmv_intent_open+0x338/0x1150 [lmv] [12304.074900] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12304.074995] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12304.075095] lmv_intent_lock+0x1dd/0x4c0 [lmv] [12304.075118] ll_intent_lock+0xc5/0xaa0 [lustre] [12304.075213] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12304.075308] ll_lookup_it+0xe0e/0x2270 [lustre] [12304.075400] ? do_raw_spin_unlock+0x75/0x190 [12304.075407] ? _raw_spin_unlock+0x12/0x30 [12304.075414] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [12304.075443] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [12304.075466] ll_atomic_open+0x36a/0xd70 [lustre] [12304.075559] lookup_open+0x356/0xb40 [12304.075569] do_last+0x313/0xfc0 [12304.075574] ? nd_jump_root+0xe5/0x160 [12304.075579] ? path_init+0x437/0x520 [12304.075584] path_openat+0xf7/0x500 [12304.075589] ? do_raw_spin_unlock+0x75/0x190 [12304.075594] do_filp_open+0x99/0x140 [12304.075600] ? getname_flags+0x6e/0x330 [12304.075605] ? __check_object_size+0xff/0x256 [12304.075613] ? do_raw_spin_unlock+0x75/0x190 [12304.075617] ? _raw_spin_unlock+0x12/0x30 [12304.075622] do_sys_openat2+0x2b4/0x410 [12304.075627] do_sys_open+0x73/0xa0 [12304.075631] __x64_sys_openat+0x24/0x30 [12304.075635] do_syscall_64+0xc1/0x440 [12304.075641] entry_SYSCALL_64_after_hwframe+0x49/0xae [12304.075647] RIP: 0033:0x7ff54af45332 [12304.075654] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [12304.075658] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [12304.075663] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [12304.075666] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [12304.075669] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [12304.075671] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [12304.075673] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [12304.558675] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12304.620836] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12304.632489] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12304.644569] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [12304.656551] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [12304.678493] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12304.690968] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [12304.701986] RDX: 000000000000232d RSI: 0000000000000003 RDI: ffff96848811cae0 [12304.709160] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [12304.715768] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [12304.724079] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12304.728670] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [12304.733923] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12304.737924] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [12304.742664] Call Trace: [12304.745645] [12304.748897] ? show_regs.cold.9+0x22/0x2f [12304.758773] ? watchdog_timer_fn.cold.6+0x62/0x10a [12304.768256] ? __hrtimer_run_queues+0x1c2/0x4d0 [12304.776417] ? watchdog+0x40/0x40 [12304.778405] ? hrtimer_interrupt+0x128/0x2c0 [12304.785391] ? smp_apic_timer_interrupt+0xa6/0x270 [12304.787695] ? apic_timer_interrupt+0xf/0x20 [12304.793533] [12304.794404] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [12304.797859] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12304.806445] do_raw_spin_lock+0x177/0x190 [12304.809492] _raw_spin_lock+0x21/0x30 [12304.815784] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12304.822522] obd_mod_free+0x46/0x140 [ptlrpc] [12304.826197] mdc_commit_open+0x91/0xb0 [mdc] [12304.832324] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12304.833984] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12304.837109] after_reply+0xe2c/0x1780 [ptlrpc] [12304.839543] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12304.842837] ptlrpcd+0xa82/0xe40 [ptlrpc] [12304.847697] ? do_wait_intr+0xf0/0xf0 [12304.851323] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12304.860873] kthread+0x1d1/0x200 [12304.865343] ? set_kthread_struct+0x70/0x70 [12304.869525] ret_from_fork+0x1f/0x30 [12306.911188] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10348s! [12306.914459] Showing busy workqueues and worker pools: [12306.916617] workqueue events: flags=0x0 [12306.918324] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12306.920620] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [12306.924398] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [12306.926608] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [12306.932296] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12306.935075] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [12306.938638] workqueue events_long: flags=0x0 [12306.940535] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12306.943647] pending: lu_objects_flush [obdclass] [12306.946931] workqueue events_unbound: flags=0x2 [12306.949300] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [12306.951841] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [12306.956306] workqueue events_power_efficient: flags=0x80 [12306.958561] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12306.961373] pending: neigh_periodic_work [12306.963307] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12306.966245] pending: do_cache_clean, check_lifetime [12306.968633] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12306.970689] pending: neigh_periodic_work [12306.972526] workqueue writeback: flags=0x4a [12306.976088] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [12306.980746] pending: wb_workfn, wb_workfn [12306.983414] workqueue xprtiod: flags=0xa [12306.984770] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [12306.989440] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [12306.995684] workqueue ipv6_addrconf: flags=0x40008 [12306.997510] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12307.001528] pending: addrconf_verify_work [12307.004424] workqueue knbd0-recv: flags=0x1a [12307.006519] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12307.009904] in-flight: 657:recv_work [12307.012625] workqueue ptlrpc_pinger: flags=0x2 [12307.014980] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12307.017560] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [12307.020398] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10348s workers=4 idle: 20831 12642 28859 [12307.023751] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12297s workers=2 idle: 659 [12324.059018] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [12324.061690] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12324.115176] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12324.128314] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12324.134581] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12324.138772] RIP: 0010:native_safe_halt+0x2e/0x50 [12324.141552] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12324.155582] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12324.158784] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [12324.161509] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [12324.164458] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [12324.167434] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [12324.170433] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [12324.173437] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [12324.177065] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12324.179582] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [12324.182508] Call Trace: [12324.183534] [12324.184298] ? show_regs.cold.9+0x22/0x2f [12324.186143] ? watchdog_timer_fn.cold.6+0x62/0x10a [12324.188165] ? __hrtimer_run_queues+0x1c2/0x4d0 [12324.190075] ? watchdog+0x40/0x40 [12324.191535] ? hrtimer_interrupt+0x128/0x2c0 [12324.193302] ? smp_apic_timer_interrupt+0xa6/0x270 [12324.195628] ? apic_timer_interrupt+0xf/0x20 [12324.197461] [12324.198317] ? native_safe_halt+0x2e/0x50 [12324.200047] kvm_wait+0xcd/0xe0 [12324.201342] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12324.206325] do_raw_spin_lock+0x177/0x190 [12324.207892] _raw_spin_lock+0x21/0x30 [12324.219037] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12324.222921] process_one_work+0x2c8/0x700 [12324.230514] worker_thread+0x296/0x6e0 [12324.233050] ? rescuer_thread+0x570/0x570 [12324.236782] kthread+0x1d1/0x200 [12324.238543] ? set_kthread_struct+0x70/0x70 [12324.241523] ret_from_fork+0x1f/0x30 [12328.061068] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [12328.072882] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12328.153166] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12328.167535] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12328.188537] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12328.190471] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12328.232963] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12328.255283] RAX: 000000000000460d RBX: ffff96848811cae0 RCX: 0000000000000001 [12328.271151] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [12328.275474] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [12328.296188] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [12328.305804] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [12328.307564] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [12328.334804] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12328.338296] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [12328.352151] Call Trace: [12328.359586] [12328.360295] ? show_regs.cold.9+0x22/0x2f [12328.361754] ? watchdog_timer_fn.cold.6+0x62/0x10a [12328.375900] ? __hrtimer_run_queues+0x1c2/0x4d0 [12328.377422] ? watchdog+0x40/0x40 [12328.384076] ? hrtimer_interrupt+0x128/0x2c0 [12328.393488] ? smp_apic_timer_interrupt+0xa6/0x270 [12328.404400] ? apic_timer_interrupt+0xf/0x20 [12328.410348] [12328.412933] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12328.414823] do_raw_spin_lock+0x177/0x190 [12328.423698] _raw_spin_lock+0x21/0x30 [12328.426637] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12328.435135] ? do_raw_spin_unlock+0x75/0x190 [12328.436300] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12328.444935] ? do_wait_intr+0xf0/0xf0 [12328.446177] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12328.451179] mdc_close+0x53b/0xd70 [mdc] [12328.458975] lmv_close+0x18d/0x490 [lmv] [12328.466658] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [12328.474991] ll_close_inode_openhandle+0x493/0xea0 [lustre] [12328.479140] ll_md_real_close+0xa5/0x2a0 [lustre] [12328.482355] ll_md_close+0x274/0x800 [lustre] [12328.487084] ll_file_release+0x128/0xc50 [lustre] [12328.489713] __fput+0x10d/0x440 [12328.494288] ____fput+0x12/0x20 [12328.498343] task_work_run+0xc6/0x110 [12328.501323] exit_to_usermode_loop+0x1dd/0x1f0 [12328.502832] do_syscall_64+0x3d6/0x440 [12328.515366] entry_SYSCALL_64_after_hwframe+0x49/0xae [12328.519614] RIP: 0033:0x7ff54af45cd8 [12328.526296] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [12328.553548] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [12328.558314] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [12328.564051] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [12328.571825] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [12328.582803] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [12328.590614] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12332.055063] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [12332.060141] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [12332.060493] Modules linked in: [12332.065262] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) [12332.067776] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12332.067859] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12332.067866] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12332.067869] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [12332.067883] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [12332.067887] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12332.067892] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [12332.067894] RDX: 000000000000278c RSI: 0000000000000003 RDI: ffff96848811cae0 [12332.067897] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [12332.067899] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [12332.067902] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12332.067905] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [12332.067908] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12332.067910] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [12332.067918] Call Trace: [12332.067924] [12332.067929] ? show_regs.cold.9+0x22/0x2f [12332.067939] ? watchdog_timer_fn.cold.6+0x62/0x10a [12332.067946] ? __hrtimer_run_queues+0x1c2/0x4d0 [12332.067950] ? watchdog+0x40/0x40 [12332.067955] ? hrtimer_interrupt+0x128/0x2c0 [12332.067959] ? smp_apic_timer_interrupt+0xa6/0x270 [12332.068060] ? apic_timer_interrupt+0xf/0x20 [12332.068065] [12332.068068] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [12332.068072] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12332.068078] do_raw_spin_lock+0x177/0x190 [12332.068082] _raw_spin_lock+0x21/0x30 [12332.068088] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12332.068357] obd_mod_free+0x46/0x140 [ptlrpc] [12332.068642] mdc_commit_open+0x91/0xb0 [mdc] [12332.068678] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12332.068890] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12332.082025] fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12332.082080] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12332.082088] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12332.082090] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12332.082106] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12332.082111] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12332.082116] RAX: 000000000000490a RBX: ffff96848811cae0 RCX: 0000000000000001 [12332.082119] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [12332.082121] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [12332.082123] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [12332.082125] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [12332.082128] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [12332.082130] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12332.082132] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [12332.082142] Call Trace: [12332.082148] [12332.082154] ? show_regs.cold.9+0x22/0x2f [12332.082165] ? watchdog_timer_fn.cold.6+0x62/0x10a [12332.082172] ? __hrtimer_run_queues+0x1c2/0x4d0 [12332.082176] ? watchdog+0x40/0x40 [12332.082180] ? hrtimer_interrupt+0x128/0x2c0 [12332.082185] ? smp_apic_timer_interrupt+0xa6/0x270 [12332.082194] ? apic_timer_interrupt+0xf/0x20 [12332.082200] [12332.082203] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12332.082209] do_raw_spin_lock+0x177/0x190 [12332.082214] _raw_spin_lock+0x21/0x30 [12332.082219] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12332.085308] ? do_raw_spin_unlock+0x75/0x190 [12332.085322] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12332.137443] after_reply+0xe2c/0x1780 [ptlrpc] [12332.137741] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12332.141281] ptlrpcd+0xa82/0xe40 [ptlrpc] [12332.141497] ? do_wait_intr+0xf0/0xf0 [12332.141508] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12332.144616] kthread+0x1d1/0x200 [12332.144634] ? set_kthread_struct+0x70/0x70 [12332.144639] ret_from_fork+0x1f/0x30 [12332.520032] ? do_wait_intr+0xf0/0xf0 [12332.521490] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12332.529231] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [12332.532141] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12332.534873] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12332.537225] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12332.539570] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [12332.541607] mdc_intent_lock+0x353/0x7e0 [mdc] [12332.543457] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12332.545994] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12332.548601] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12332.551053] lmv_intent_open+0x338/0x1150 [lmv] [12332.552954] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12332.555222] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12332.557485] lmv_intent_lock+0x1dd/0x4c0 [lmv] [12332.559440] ll_intent_lock+0xc5/0xaa0 [lustre] [12332.561573] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12332.564005] ll_lookup_it+0xe0e/0x2270 [lustre] [12332.566065] ? do_raw_spin_unlock+0x75/0x190 [12332.567952] ? _raw_spin_unlock+0x12/0x30 [12332.569637] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [12332.571829] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [12332.573993] ll_atomic_open+0x36a/0xd70 [lustre] [12332.576041] lookup_open+0x356/0xb40 [12332.577476] do_last+0x313/0xfc0 [12332.578903] ? nd_jump_root+0xe5/0x160 [12332.580462] ? path_init+0x437/0x520 [12332.582158] path_openat+0xf7/0x500 [12332.583607] ? do_raw_spin_unlock+0x75/0x190 [12332.585366] do_filp_open+0x99/0x140 [12332.586868] ? getname_flags+0x6e/0x330 [12332.588456] ? __check_object_size+0xff/0x256 [12332.590370] ? do_raw_spin_unlock+0x75/0x190 [12332.592229] ? _raw_spin_unlock+0x12/0x30 [12332.593918] do_sys_openat2+0x2b4/0x410 [12332.595617] do_sys_open+0x73/0xa0 [12332.597039] __x64_sys_openat+0x24/0x30 [12332.598545] do_syscall_64+0xc1/0x440 [12332.600122] entry_SYSCALL_64_after_hwframe+0x49/0xae [12332.602209] RIP: 0033:0x7ff54af45332 [12332.603698] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [12332.611377] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [12332.614401] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [12332.617439] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [12332.620411] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [12332.623441] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [12332.626625] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [12337.120345] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10378s! [12337.124381] Showing busy workqueues and worker pools: [12337.136184] workqueue events: flags=0x0 [12337.143483] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12337.155353] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [12337.169914] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [12337.183167] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [12337.225061] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12337.229993] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [12337.238564] workqueue events_long: flags=0x0 [12337.240629] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12337.245081] pending: lu_objects_flush [obdclass] [12337.247717] workqueue events_unbound: flags=0x2 [12337.249118] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [12337.251339] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [12337.254704] workqueue events_power_efficient: flags=0x80 [12337.257145] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12337.260354] pending: neigh_periodic_work [12337.262763] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12337.265618] pending: do_cache_clean, check_lifetime [12337.269212] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12337.279016] pending: neigh_periodic_work [12337.287194] workqueue writeback: flags=0x4a [12337.294075] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [12337.299990] pending: wb_workfn, wb_workfn [12337.306377] workqueue xprtiod: flags=0xa [12337.308285] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [12337.314267] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [12337.318879] workqueue ipv6_addrconf: flags=0x40008 [12337.322192] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12337.331247] pending: addrconf_verify_work [12337.334768] workqueue knbd0-recv: flags=0x1a [12337.338517] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12337.345914] in-flight: 657:recv_work [12337.350109] workqueue ptlrpc_pinger: flags=0x2 [12337.359135] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12337.362749] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [12337.373346] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10378s workers=4 idle: 20831 12642 28859 [12337.385059] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12327s workers=2 idle: 659 [12352.062114] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [12352.068064] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12352.095756] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12352.101325] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12352.105932] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12352.109518] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12352.112214] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12352.121103] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12352.124168] RAX: 0000000000006d16 RBX: ffff96848811cae0 RCX: 0000000000000001 [12352.127598] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [12352.130638] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [12352.133811] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [12352.137168] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [12352.140584] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [12352.144018] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12352.146737] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [12352.149640] Call Trace: [12352.150634] [12352.151410] ? show_regs.cold.9+0x22/0x2f [12352.152971] ? watchdog_timer_fn.cold.6+0x62/0x10a [12352.154988] ? __hrtimer_run_queues+0x1c2/0x4d0 [12352.156891] ? watchdog+0x40/0x40 [12352.158618] ? hrtimer_interrupt+0x128/0x2c0 [12352.160364] ? smp_apic_timer_interrupt+0xa6/0x270 [12352.162180] ? apic_timer_interrupt+0xf/0x20 [12352.164164] [12352.164981] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12352.167548] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12352.169806] do_raw_spin_lock+0x177/0x190 [12352.171521] _raw_spin_lock+0x21/0x30 [12352.172992] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12352.175583] process_one_work+0x2c8/0x700 [12352.177204] worker_thread+0x296/0x6e0 [12352.181410] ? rescuer_thread+0x570/0x570 [12352.186676] kthread+0x1d1/0x200 [12352.187999] ? set_kthread_struct+0x70/0x70 [12352.197946] ret_from_fork+0x1f/0x30 [12356.059576] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [12356.069803] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12356.136155] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12356.153528] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12356.162586] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12356.169477] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12356.194305] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12356.213234] RAX: 0000000000007093 RBX: ffff96848811cae0 RCX: 0000000000000001 [12356.226738] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [12356.238601] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [12356.252845] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [12356.265677] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [12356.273507] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [12356.280091] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12356.283651] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [12356.287276] Call Trace: [12356.293246] [12356.296637] ? show_regs.cold.9+0x22/0x2f [12356.304757] ? watchdog_timer_fn.cold.6+0x62/0x10a [12356.316675] ? __hrtimer_run_queues+0x1c2/0x4d0 [12356.322556] ? watchdog+0x40/0x40 [12356.325795] ? hrtimer_interrupt+0x128/0x2c0 [12356.329100] ? smp_apic_timer_interrupt+0xa6/0x270 [12356.333557] ? apic_timer_interrupt+0xf/0x20 [12356.345843] [12356.353380] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12356.362529] do_raw_spin_lock+0x177/0x190 [12356.364875] _raw_spin_lock+0x21/0x30 [12356.374174] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12356.376510] ? do_raw_spin_unlock+0x75/0x190 [12356.386354] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12356.391321] ? do_wait_intr+0xf0/0xf0 [12356.395328] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12356.399455] mdc_close+0x53b/0xd70 [mdc] [12356.402594] lmv_close+0x18d/0x490 [lmv] [12356.405317] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [12356.409373] ll_close_inode_openhandle+0x493/0xea0 [lustre] [12356.414048] ll_md_real_close+0xa5/0x2a0 [lustre] [12356.417542] ll_md_close+0x274/0x800 [lustre] [12356.420547] ll_file_release+0x128/0xc50 [lustre] [12356.423590] __fput+0x10d/0x440 [12356.426634] ____fput+0x12/0x20 [12356.428888] task_work_run+0xc6/0x110 [12356.431283] exit_to_usermode_loop+0x1dd/0x1f0 [12356.434267] do_syscall_64+0x3d6/0x440 [12356.436852] entry_SYSCALL_64_after_hwframe+0x49/0xae [12356.439956] RIP: 0033:0x7ff54af45cd8 [12356.442897] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [12356.453707] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [12356.458134] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [12356.461584] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [12356.465875] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [12356.469293] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [12356.473041] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12360.060019] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [12360.063733] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12360.088011] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12360.092623] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12360.096574] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12360.098288] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12360.110809] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12360.117275] RAX: 0000000000006dab RBX: ffff96848811cae0 RCX: 0000000000000001 [12360.122094] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [12360.125719] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [12360.131963] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [12360.135763] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [12360.139587] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [12360.143133] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12360.146569] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [12360.150474] Call Trace: [12360.151661] [12360.152665] ? show_regs.cold.9+0x22/0x2f [12360.154623] ? watchdog_timer_fn.cold.6+0x62/0x10a [12360.156788] ? __hrtimer_run_queues+0x1c2/0x4d0 [12360.158808] ? watchdog+0x40/0x40 [12360.160143] ? hrtimer_interrupt+0x128/0x2c0 [12360.162198] ? smp_apic_timer_interrupt+0xa6/0x270 [12360.164865] ? apic_timer_interrupt+0xf/0x20 [12360.166612] [12360.167808] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12360.170048] do_raw_spin_lock+0x177/0x190 [12360.171975] _raw_spin_lock+0x21/0x30 [12360.173778] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12360.175939] ? do_raw_spin_unlock+0x75/0x190 [12360.177975] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12360.180782] ? do_wait_intr+0xf0/0xf0 [12360.182462] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12360.184924] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [12360.187442] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12360.190020] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12360.192330] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12360.194974] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [12360.197649] mdc_intent_lock+0x353/0x7e0 [mdc] [12360.199681] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12360.202305] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12360.205261] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12360.207950] lmv_intent_open+0x338/0x1150 [lmv] [12360.210043] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12360.213077] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12360.215963] lmv_intent_lock+0x1dd/0x4c0 [lmv] [12360.218141] ll_intent_lock+0xc5/0xaa0 [lustre] [12360.220788] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12360.223072] ll_lookup_it+0xe0e/0x2270 [lustre] [12360.225261] ? do_raw_spin_unlock+0x75/0x190 [12360.227537] ? _raw_spin_unlock+0x12/0x30 [12360.230050] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [12360.238245] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [12360.242825] ll_atomic_open+0x36a/0xd70 [lustre] [12360.245220] lookup_open+0x356/0xb40 [12360.248109] do_last+0x313/0xfc0 [12360.249313] ? nd_jump_root+0xe5/0x160 [12360.252476] ? path_init+0x437/0x520 [12360.254624] path_openat+0xf7/0x500 [12360.256736] ? do_raw_spin_unlock+0x75/0x190 [12360.259008] do_filp_open+0x99/0x140 [12360.260909] ? getname_flags+0x6e/0x330 [12360.262819] ? __check_object_size+0xff/0x256 [12360.265054] ? do_raw_spin_unlock+0x75/0x190 [12360.267819] ? _raw_spin_unlock+0x12/0x30 [12360.270705] do_sys_openat2+0x2b4/0x410 [12360.272771] do_sys_open+0x73/0xa0 [12360.274184] __x64_sys_openat+0x24/0x30 [12360.276762] do_syscall_64+0xc1/0x440 [12360.278500] entry_SYSCALL_64_after_hwframe+0x49/0xae [12360.281139] RIP: 0033:0x7ff54af45332 [12360.282463] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [12360.290071] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [12360.293448] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [12360.296511] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [12360.299443] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [12360.302127] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [12360.305113] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [12364.056305] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [12364.069231] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12364.140049] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12364.154354] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12364.162923] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [12364.167569] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [12364.181821] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12364.191530] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [12364.199653] RDX: 00000000000017fc RSI: 0000000000000003 RDI: ffff96848811cae0 [12364.205807] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [12364.214874] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [12364.220105] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12364.227559] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [12364.241843] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12364.251644] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [12364.258776] Call Trace: [12364.261269] [12364.262722] ? show_regs.cold.9+0x22/0x2f [12364.265531] ? watchdog_timer_fn.cold.6+0x62/0x10a [12364.269575] ? __hrtimer_run_queues+0x1c2/0x4d0 [12364.273684] ? watchdog+0x40/0x40 [12364.279208] ? hrtimer_interrupt+0x128/0x2c0 [12364.283750] ? smp_apic_timer_interrupt+0xa6/0x270 [12364.290259] ? apic_timer_interrupt+0xf/0x20 [12364.294951] [12364.303263] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [12364.307823] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12364.319206] do_raw_spin_lock+0x177/0x190 [12364.323003] _raw_spin_lock+0x21/0x30 [12364.325777] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12364.335202] obd_mod_free+0x46/0x140 [ptlrpc] [12364.344309] mdc_commit_open+0x91/0xb0 [mdc] [12364.347229] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12364.355045] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12364.357064] after_reply+0xe2c/0x1780 [ptlrpc] [12364.365622] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12364.374196] ptlrpcd+0xa82/0xe40 [ptlrpc] [12364.376847] ? do_wait_intr+0xf0/0xf0 [12364.378900] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12364.392709] kthread+0x1d1/0x200 [12364.396841] ? set_kthread_struct+0x70/0x70 [12364.411675] ret_from_fork+0x1f/0x30 [12367.839049] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10408s! [12367.843759] Showing busy workqueues and worker pools: [12367.846297] workqueue events: flags=0x0 [12367.847675] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12367.857527] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [12367.866122] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [12367.876568] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [12367.887750] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12367.890758] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [12367.899097] workqueue events_long: flags=0x0 [12367.908926] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12367.916265] pending: lu_objects_flush [obdclass] [12367.919633] workqueue events_unbound: flags=0x2 [12367.922598] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [12367.932597] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [12367.939811] workqueue events_power_efficient: flags=0x80 [12367.947928] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12367.959805] pending: neigh_periodic_work [12367.963014] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12367.970664] pending: do_cache_clean, check_lifetime [12367.977737] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12367.982013] pending: neigh_periodic_work [12367.986237] workqueue writeback: flags=0x4a [12367.991217] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [12367.995590] pending: wb_workfn, wb_workfn [12368.006585] workqueue xprtiod: flags=0xa [12368.007743] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [12368.009194] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [12368.017169] workqueue ipv6_addrconf: flags=0x40008 [12368.019713] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12368.023363] pending: addrconf_verify_work [12368.024933] workqueue knbd0-recv: flags=0x1a [12368.029244] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12368.039656] in-flight: 657:recv_work [12368.048794] workqueue ptlrpc_pinger: flags=0x2 [12368.052151] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12368.055974] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [12368.061092] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10409s workers=4 idle: 20831 12642 28859 [12368.066730] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12358s workers=2 idle: 659 [12380.068945] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [12380.074927] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12380.128576] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12380.137667] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12380.143660] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12380.147946] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12380.156615] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12380.174985] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12380.180650] RAX: 0000000000007f07 RBX: ffff96848811cae0 RCX: 0000000000000001 [12380.189164] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [12380.190844] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [12380.197061] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [12380.205719] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [12380.214493] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [12380.219877] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12380.221931] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [12380.233400] Call Trace: [12380.235483] [12380.236785] ? show_regs.cold.9+0x22/0x2f [12380.238449] ? watchdog_timer_fn.cold.6+0x62/0x10a [12380.241001] ? __hrtimer_run_queues+0x1c2/0x4d0 [12380.243853] ? watchdog+0x40/0x40 [12380.247146] ? hrtimer_interrupt+0x128/0x2c0 [12380.250021] ? smp_apic_timer_interrupt+0xa6/0x270 [12380.253034] ? apic_timer_interrupt+0xf/0x20 [12380.255360] [12380.256539] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12380.258946] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12380.261784] do_raw_spin_lock+0x177/0x190 [12380.263856] _raw_spin_lock+0x21/0x30 [12380.265591] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12380.268596] process_one_work+0x2c8/0x700 [12380.270930] worker_thread+0x296/0x6e0 [12380.272810] ? rescuer_thread+0x570/0x570 [12380.274807] kthread+0x1d1/0x200 [12380.276485] ? set_kthread_struct+0x70/0x70 [12380.278811] ret_from_fork+0x1f/0x30 [12384.059513] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [12384.067566] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12384.089854] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12384.094672] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12384.097654] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12384.099111] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12384.106252] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12384.109376] RAX: 0000000000004386 RBX: ffff96848811cae0 RCX: 0000000000000001 [12384.112520] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [12384.115719] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [12384.119446] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [12384.126030] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [12384.128642] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [12384.132604] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12384.135452] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [12384.138077] Call Trace: [12384.139232] [12384.140007] ? show_regs.cold.9+0x22/0x2f [12384.141689] ? watchdog_timer_fn.cold.6+0x62/0x10a [12384.143528] ? __hrtimer_run_queues+0x1c2/0x4d0 [12384.145309] ? watchdog+0x40/0x40 [12384.146289] ? hrtimer_interrupt+0x128/0x2c0 [12384.150364] ? smp_apic_timer_interrupt+0xa6/0x270 [12384.152789] ? apic_timer_interrupt+0xf/0x20 [12384.154185] [12384.159923] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12384.163629] do_raw_spin_lock+0x177/0x190 [12384.165046] _raw_spin_lock+0x21/0x30 [12384.167778] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12384.170820] ? do_raw_spin_unlock+0x75/0x190 [12384.177830] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12384.181950] ? do_wait_intr+0xf0/0xf0 [12384.183587] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12384.186010] mdc_close+0x53b/0xd70 [mdc] [12384.187481] lmv_close+0x18d/0x490 [lmv] [12384.188619] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [12384.190647] ll_close_inode_openhandle+0x493/0xea0 [lustre] [12384.192944] ll_md_real_close+0xa5/0x2a0 [lustre] [12384.195234] ll_md_close+0x274/0x800 [lustre] [12384.197100] ll_file_release+0x128/0xc50 [lustre] [12384.199534] __fput+0x10d/0x440 [12384.201003] ____fput+0x12/0x20 [12384.202363] task_work_run+0xc6/0x110 [12384.203894] exit_to_usermode_loop+0x1dd/0x1f0 [12384.205676] do_syscall_64+0x3d6/0x440 [12384.206869] entry_SYSCALL_64_after_hwframe+0x49/0xae [12384.208533] RIP: 0033:0x7ff54af45cd8 [12384.210197] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [12384.216894] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [12384.219707] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [12384.222925] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [12384.226551] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [12384.229651] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [12384.233291] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12388.064584] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [12388.073677] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12388.149441] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12388.155215] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12388.158712] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12388.164360] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12388.180825] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12388.184952] RAX: 0000000000001311 RBX: ffff96848811cae0 RCX: 0000000000000001 [12388.189015] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [12388.195798] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [12388.210052] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [12388.219056] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [12388.224566] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [12388.232309] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12388.238855] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [12388.250571] Call Trace: [12388.252671] [12388.257664] ? show_regs.cold.9+0x22/0x2f [12388.262448] ? watchdog_timer_fn.cold.6+0x62/0x10a [12388.264173] ? __hrtimer_run_queues+0x1c2/0x4d0 [12388.271392] ? watchdog+0x40/0x40 [12388.275702] ? hrtimer_interrupt+0x128/0x2c0 [12388.279768] ? smp_apic_timer_interrupt+0xa6/0x270 [12388.282960] ? apic_timer_interrupt+0xf/0x20 [12388.285922] [12388.289970] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12388.298044] do_raw_spin_lock+0x177/0x190 [12388.300045] _raw_spin_lock+0x21/0x30 [12388.302117] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12388.304013] ? do_raw_spin_unlock+0x75/0x190 [12388.318608] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12388.331013] ? do_wait_intr+0xf0/0xf0 [12388.333904] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12388.339780] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [12388.351023] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12388.367506] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12388.386887] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12388.392058] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [12388.402302] mdc_intent_lock+0x353/0x7e0 [mdc] [12388.410100] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12388.415993] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12388.422650] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12388.426058] lmv_intent_open+0x338/0x1150 [lmv] [12388.431723] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12388.433582] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12388.438029] lmv_intent_lock+0x1dd/0x4c0 [lmv] [12388.441294] ll_intent_lock+0xc5/0xaa0 [lustre] [12388.443680] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12388.446452] ll_lookup_it+0xe0e/0x2270 [lustre] [12388.448986] ? do_raw_spin_unlock+0x75/0x190 [12388.450740] ? _raw_spin_unlock+0x12/0x30 [12388.455067] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [12388.457417] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [12388.459300] ll_atomic_open+0x36a/0xd70 [lustre] [12388.464730] lookup_open+0x356/0xb40 [12388.467144] do_last+0x313/0xfc0 [12388.470350] ? nd_jump_root+0xe5/0x160 [12388.473992] ? path_init+0x437/0x520 [12388.476759] path_openat+0xf7/0x500 [12388.478825] ? do_raw_spin_unlock+0x75/0x190 [12388.482378] do_filp_open+0x99/0x140 [12388.489919] ? getname_flags+0x6e/0x330 [12388.494745] ? __check_object_size+0xff/0x256 [12388.505624] ? do_raw_spin_unlock+0x75/0x190 [12388.509818] ? _raw_spin_unlock+0x12/0x30 [12388.514615] do_sys_openat2+0x2b4/0x410 [12388.517582] do_sys_open+0x73/0xa0 [12388.520420] __x64_sys_openat+0x24/0x30 [12388.523002] do_syscall_64+0xc1/0x440 [12388.527514] entry_SYSCALL_64_after_hwframe+0x49/0xae [12388.530751] RIP: 0033:0x7ff54af45332 [12388.535814] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [12388.545149] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [12388.549679] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [12388.555784] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [12388.557612] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [12388.562982] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [12388.565551] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [12396.055019] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [12396.060872] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12396.096709] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12396.106626] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12396.111407] RIP: 0010:native_safe_halt+0x2e/0x50 [12396.115888] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12396.145756] RSP: 0000:ffffaf9fc1b97bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12396.159035] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [12396.171773] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff96848811cae0 [12396.184855] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [12396.197437] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [12396.208434] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12396.214278] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [12396.218668] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12396.222063] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [12396.227779] Call Trace: [12396.229974] [12396.232617] ? show_regs.cold.9+0x22/0x2f [12396.234920] ? watchdog_timer_fn.cold.6+0x62/0x10a [12396.238291] ? __hrtimer_run_queues+0x1c2/0x4d0 [12396.241231] ? watchdog+0x40/0x40 [12396.244876] ? hrtimer_interrupt+0x128/0x2c0 [12396.247004] ? smp_apic_timer_interrupt+0xa6/0x270 [12396.251497] ? apic_timer_interrupt+0xf/0x20 [12396.254715] [12396.256769] ? native_safe_halt+0x2e/0x50 [12396.259770] kvm_wait+0xcd/0xe0 [12396.263671] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12396.269041] do_raw_spin_lock+0x177/0x190 [12396.271725] _raw_spin_lock+0x21/0x30 [12396.278095] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12396.283995] obd_mod_free+0x46/0x140 [ptlrpc] [12396.289375] mdc_commit_open+0x91/0xb0 [mdc] [12396.295355] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12396.297394] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12396.307627] after_reply+0xe2c/0x1780 [ptlrpc] [12396.311855] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12396.317371] ptlrpcd+0xa82/0xe40 [ptlrpc] [12396.320311] ? do_wait_intr+0xf0/0xf0 [12396.324614] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12396.332719] kthread+0x1d1/0x200 [12396.337573] ? set_kthread_struct+0x70/0x70 [12396.345695] ret_from_fork+0x1f/0x30 [12398.559025] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10439s! [12398.571372] Showing busy workqueues and worker pools: [12398.580254] workqueue events: flags=0x0 [12398.582580] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12398.591115] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [12398.603743] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [12398.613662] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [12398.621500] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12398.625261] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [12398.631243] workqueue events_long: flags=0x0 [12398.633724] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12398.635923] pending: lu_objects_flush [obdclass] [12398.651148] workqueue events_unbound: flags=0x2 [12398.657951] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [12398.661170] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [12398.669386] workqueue events_power_efficient: flags=0x80 [12398.672183] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12398.675548] pending: neigh_periodic_work [12398.678186] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12398.681460] pending: do_cache_clean, check_lifetime [12398.687456] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12398.694565] pending: neigh_periodic_work [12398.699140] workqueue writeback: flags=0x4a [12398.703392] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [12398.710722] pending: wb_workfn, wb_workfn [12398.712292] workqueue xprtiod: flags=0xa [12398.716953] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [12398.728187] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [12398.736053] workqueue ipv6_addrconf: flags=0x40008 [12398.740985] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12398.745304] pending: addrconf_verify_work [12398.748507] workqueue knbd0-recv: flags=0x1a [12398.753881] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12398.759613] in-flight: 657:recv_work [12398.763096] workqueue ptlrpc_pinger: flags=0x2 [12398.766061] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12398.770038] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [12398.774144] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10439s workers=4 idle: 20831 12642 28859 [12398.778731] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12388s workers=2 idle: 659 [12408.059747] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [12408.068301] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12408.121848] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12408.138704] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12408.146564] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12408.151357] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12408.154205] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12408.169710] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12408.179660] RAX: 0000000000007613 RBX: ffff96848811cae0 RCX: 0000000000000001 [12408.183012] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [12408.192324] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [12408.201421] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [12408.207705] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [12408.212327] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [12408.216538] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12408.220435] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [12408.231060] Call Trace: [12408.232456] [12408.233776] ? show_regs.cold.9+0x22/0x2f [12408.239724] ? watchdog_timer_fn.cold.6+0x62/0x10a [12408.245341] ? __hrtimer_run_queues+0x1c2/0x4d0 [12408.250635] ? watchdog+0x40/0x40 [12408.253581] ? hrtimer_interrupt+0x128/0x2c0 [12408.255815] ? smp_apic_timer_interrupt+0xa6/0x270 [12408.258995] ? apic_timer_interrupt+0xf/0x20 [12408.261415] [12408.263972] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12408.267341] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12408.274465] do_raw_spin_lock+0x177/0x190 [12408.277456] _raw_spin_lock+0x21/0x30 [12408.282847] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12408.286870] process_one_work+0x2c8/0x700 [12408.291833] worker_thread+0x296/0x6e0 [12408.296016] ? rescuer_thread+0x570/0x570 [12408.299325] kthread+0x1d1/0x200 [12408.301276] ? set_kthread_struct+0x70/0x70 [12408.305390] ret_from_fork+0x1f/0x30 [12412.058023] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [12412.060757] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12412.080927] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12412.091364] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12412.096354] RIP: 0010:native_safe_halt+0x2e/0x50 [12412.102457] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12412.121810] RSP: 0018:ffffaf9fc1543ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12412.127767] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [12412.137405] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c20b4154 [12412.139647] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [12412.147441] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [12412.153839] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [12412.158239] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [12412.162690] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12412.168592] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [12412.174518] Call Trace: [12412.178695] [12412.180203] ? show_regs.cold.9+0x22/0x2f [12412.185186] ? watchdog_timer_fn.cold.6+0x62/0x10a [12412.188149] ? __hrtimer_run_queues+0x1c2/0x4d0 [12412.192670] ? watchdog+0x40/0x40 [12412.195463] ? hrtimer_interrupt+0x128/0x2c0 [12412.199622] ? smp_apic_timer_interrupt+0xa6/0x270 [12412.204251] ? apic_timer_interrupt+0xf/0x20 [12412.207541] [12412.208450] ? native_safe_halt+0x2e/0x50 [12412.214896] kvm_wait+0xcd/0xe0 [12412.219515] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12412.234164] do_raw_spin_lock+0x177/0x190 [12412.235330] _raw_spin_lock+0x21/0x30 [12412.239781] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12412.247714] ? do_raw_spin_unlock+0x75/0x190 [12412.253894] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12412.263431] ? do_wait_intr+0xf0/0xf0 [12412.264573] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12412.269621] mdc_close+0x53b/0xd70 [mdc] [12412.275471] lmv_close+0x18d/0x490 [lmv] [12412.280743] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [12412.286217] ll_close_inode_openhandle+0x493/0xea0 [lustre] [12412.290140] ll_md_real_close+0xa5/0x2a0 [lustre] [12412.291737] ll_md_close+0x274/0x800 [lustre] [12412.294572] ll_file_release+0x128/0xc50 [lustre] [12412.301459] __fput+0x10d/0x440 [12412.305307] ____fput+0x12/0x20 [12412.308259] task_work_run+0xc6/0x110 [12412.311616] exit_to_usermode_loop+0x1dd/0x1f0 [12412.314918] do_syscall_64+0x3d6/0x440 [12412.317387] entry_SYSCALL_64_after_hwframe+0x49/0xae [12412.321278] RIP: 0033:0x7ff54af45cd8 [12412.322807] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [12412.337584] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [12412.343640] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [12412.349115] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [12412.353209] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [12412.356595] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [12412.359180] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12416.060025] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [12416.063260] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12416.080322] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12416.084189] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12416.087143] RIP: 0010:native_safe_halt+0x2e/0x50 [12416.088779] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12416.095327] RSP: 0000:ffffaf9fc15234f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12416.098775] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [12416.101871] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c21b4154 [12416.105786] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [12416.111362] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [12416.115712] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [12416.122977] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [12416.125513] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12416.131529] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [12416.136507] Call Trace: [12416.137625] [12416.138938] ? show_regs.cold.9+0x22/0x2f [12416.140390] ? watchdog_timer_fn.cold.6+0x62/0x10a [12416.142507] ? __hrtimer_run_queues+0x1c2/0x4d0 [12416.143996] ? watchdog+0x40/0x40 [12416.145595] ? hrtimer_interrupt+0x128/0x2c0 [12416.147531] ? smp_apic_timer_interrupt+0xa6/0x270 [12416.149510] ? apic_timer_interrupt+0xf/0x20 [12416.154974] [12416.155943] ? native_safe_halt+0x2e/0x50 [12416.157618] kvm_wait+0xcd/0xe0 [12416.159076] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12416.161206] do_raw_spin_lock+0x177/0x190 [12416.162994] _raw_spin_lock+0x21/0x30 [12416.164392] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12416.166186] ? do_raw_spin_unlock+0x75/0x190 [12416.168802] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12416.171352] ? do_wait_intr+0xf0/0xf0 [12416.174325] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12416.177530] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [12416.180112] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12416.183000] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12416.185050] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12416.187268] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [12416.189077] mdc_intent_lock+0x353/0x7e0 [mdc] [12416.191231] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12416.197396] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12416.201033] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12416.203436] lmv_intent_open+0x338/0x1150 [lmv] [12416.206089] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12416.208238] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12416.210670] lmv_intent_lock+0x1dd/0x4c0 [lmv] [12416.212361] ll_intent_lock+0xc5/0xaa0 [lustre] [12416.214771] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12416.217556] ll_lookup_it+0xe0e/0x2270 [lustre] [12416.220136] ? do_raw_spin_unlock+0x75/0x190 [12416.223024] ? _raw_spin_unlock+0x12/0x30 [12416.225125] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [12416.227562] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [12416.230030] ll_atomic_open+0x36a/0xd70 [lustre] [12416.231919] lookup_open+0x356/0xb40 [12416.233449] do_last+0x313/0xfc0 [12416.234612] ? nd_jump_root+0xe5/0x160 [12416.236124] ? path_init+0x437/0x520 [12416.237631] path_openat+0xf7/0x500 [12416.238996] ? do_raw_spin_unlock+0x75/0x190 [12416.240375] do_filp_open+0x99/0x140 [12416.242298] ? getname_flags+0x6e/0x330 [12416.244281] ? __check_object_size+0xff/0x256 [12416.246299] ? do_raw_spin_unlock+0x75/0x190 [12416.248138] ? _raw_spin_unlock+0x12/0x30 [12416.249668] do_sys_openat2+0x2b4/0x410 [12416.252154] do_sys_open+0x73/0xa0 [12416.253442] __x64_sys_openat+0x24/0x30 [12416.255097] do_syscall_64+0xc1/0x440 [12416.256675] entry_SYSCALL_64_after_hwframe+0x49/0xae [12416.258812] RIP: 0033:0x7ff54af45332 [12416.260316] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [12416.267243] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [12416.270297] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [12416.273506] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [12416.276926] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [12416.279588] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [12416.282175] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [12424.055030] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [12424.060191] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12424.076750] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12424.079616] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12424.082126] RIP: 0010:native_safe_halt+0x2e/0x50 [12424.083413] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12424.090557] RSP: 0000:ffffaf9fc1b97bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12424.092516] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [12424.094252] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff96848811cae0 [12424.096216] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [12424.097630] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [12424.099196] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12424.100575] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [12424.102240] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12424.103733] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [12424.105625] Call Trace: [12424.106240] [12424.106561] ? show_regs.cold.9+0x22/0x2f [12424.107827] ? watchdog_timer_fn.cold.6+0x62/0x10a [12424.109429] ? __hrtimer_run_queues+0x1c2/0x4d0 [12424.110965] ? watchdog+0x40/0x40 [12424.111613] ? hrtimer_interrupt+0x128/0x2c0 [12424.113049] ? smp_apic_timer_interrupt+0xa6/0x270 [12424.114600] ? apic_timer_interrupt+0xf/0x20 [12424.115697] [12424.116591] ? native_safe_halt+0x2e/0x50 [12424.117521] kvm_wait+0xcd/0xe0 [12424.118241] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12424.119373] do_raw_spin_lock+0x177/0x190 [12424.120256] _raw_spin_lock+0x21/0x30 [12424.120955] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12424.122612] obd_mod_free+0x46/0x140 [ptlrpc] [12424.124526] mdc_commit_open+0x91/0xb0 [mdc] [12424.126281] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12424.128571] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12424.136359] after_reply+0xe2c/0x1780 [ptlrpc] [12424.140480] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12424.144177] ptlrpcd+0xa82/0xe40 [ptlrpc] [12424.146561] ? do_wait_intr+0xf0/0xf0 [12424.148506] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12424.151425] kthread+0x1d1/0x200 [12424.153147] ? set_kthread_struct+0x70/0x70 [12424.154513] ret_from_fork+0x1f/0x30 [12429.280538] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10470s! [12429.286574] Showing busy workqueues and worker pools: [12429.290760] workqueue events: flags=0x0 [12429.291791] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12429.294874] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [12429.297577] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [12429.302145] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [12429.307939] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12429.311377] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [12429.318658] workqueue events_long: flags=0x0 [12429.321476] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12429.325381] pending: lu_objects_flush [obdclass] [12429.327639] workqueue events_unbound: flags=0x2 [12429.332593] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [12429.335324] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [12429.340503] workqueue events_power_efficient: flags=0x80 [12429.343152] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12429.346296] pending: neigh_periodic_work [12429.347453] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12429.349565] pending: do_cache_clean, check_lifetime [12429.351217] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12429.353243] pending: neigh_periodic_work [12429.355344] workqueue writeback: flags=0x4a [12429.356630] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [12429.358834] pending: wb_workfn, wb_workfn [12429.360180] workqueue xprtiod: flags=0xa [12429.361321] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [12429.363424] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [12429.365563] workqueue ipv6_addrconf: flags=0x40008 [12429.367543] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12429.369570] pending: addrconf_verify_work [12429.370969] workqueue knbd0-recv: flags=0x1a [12429.373373] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12429.375526] in-flight: 657:recv_work [12429.377180] workqueue ptlrpc_pinger: flags=0x2 [12429.379369] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12429.382738] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [12429.384651] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10470s workers=4 idle: 20831 12642 28859 [12429.387805] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12419s workers=2 idle: 659 [12436.059655] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [12436.063677] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12436.090763] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12436.098847] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12436.108847] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12436.115153] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12436.119807] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12436.135369] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12436.140957] RAX: 00000000000048d3 RBX: ffff96848811cae0 RCX: 0000000000000001 [12436.148515] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [12436.153747] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [12436.162908] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [12436.171488] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [12436.180484] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [12436.188788] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12436.190988] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [12436.196153] Call Trace: [12436.198155] [12436.201219] ? show_regs.cold.9+0x22/0x2f [12436.209225] ? watchdog_timer_fn.cold.6+0x62/0x10a [12436.211997] ? __hrtimer_run_queues+0x1c2/0x4d0 [12436.213470] ? watchdog+0x40/0x40 [12436.215356] ? hrtimer_interrupt+0x128/0x2c0 [12436.218799] ? smp_apic_timer_interrupt+0xa6/0x270 [12436.222960] ? apic_timer_interrupt+0xf/0x20 [12436.224351] [12436.225010] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12436.231874] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12436.235713] do_raw_spin_lock+0x177/0x190 [12436.237123] _raw_spin_lock+0x21/0x30 [12436.238315] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12436.242426] process_one_work+0x2c8/0x700 [12436.245614] worker_thread+0x296/0x6e0 [12436.247565] ? rescuer_thread+0x570/0x570 [12436.250504] kthread+0x1d1/0x200 [12436.251868] ? set_kthread_struct+0x70/0x70 [12436.254450] ret_from_fork+0x1f/0x30 [12440.058046] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [12440.066674] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12440.102461] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12440.109948] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12440.116395] RIP: 0010:native_safe_halt+0x2e/0x50 [12440.118752] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12440.132607] RSP: 0018:ffffaf9fc1543ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12440.136081] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [12440.139723] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c20b4154 [12440.142785] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [12440.153206] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [12440.158081] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [12440.161426] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [12440.165450] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12440.168196] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [12440.176937] Call Trace: [12440.180387] [12440.181176] ? show_regs.cold.9+0x22/0x2f [12440.186010] ? watchdog_timer_fn.cold.6+0x62/0x10a [12440.188188] ? __hrtimer_run_queues+0x1c2/0x4d0 [12440.190427] ? watchdog+0x40/0x40 [12440.192741] ? hrtimer_interrupt+0x128/0x2c0 [12440.194950] ? smp_apic_timer_interrupt+0xa6/0x270 [12440.200307] ? apic_timer_interrupt+0xf/0x20 [12440.205180] [12440.206851] ? native_safe_halt+0x2e/0x50 [12440.208983] kvm_wait+0xcd/0xe0 [12440.212575] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12440.219927] do_raw_spin_lock+0x177/0x190 [12440.223681] _raw_spin_lock+0x21/0x30 [12440.224941] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12440.229463] ? do_raw_spin_unlock+0x75/0x190 [12440.232795] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12440.235840] ? do_wait_intr+0xf0/0xf0 [12440.238802] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12440.243002] mdc_close+0x53b/0xd70 [mdc] [12440.244928] lmv_close+0x18d/0x490 [lmv] [12440.246277] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [12440.252713] ll_close_inode_openhandle+0x493/0xea0 [lustre] [12440.255456] ll_md_real_close+0xa5/0x2a0 [lustre] [12440.258238] ll_md_close+0x274/0x800 [lustre] [12440.261661] ll_file_release+0x128/0xc50 [lustre] [12440.265208] __fput+0x10d/0x440 [12440.266310] ____fput+0x12/0x20 [12440.267459] task_work_run+0xc6/0x110 [12440.272458] exit_to_usermode_loop+0x1dd/0x1f0 [12440.274811] do_syscall_64+0x3d6/0x440 [12440.276446] entry_SYSCALL_64_after_hwframe+0x49/0xae [12440.280393] RIP: 0033:0x7ff54af45cd8 [12440.283668] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [12440.296386] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [12440.301393] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [12440.305197] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [12440.311107] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [12440.317174] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [12440.322637] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12444.060023] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [12444.063235] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12444.084253] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12444.089250] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12444.110213] RIP: 0010:native_safe_halt+0x2e/0x50 [12444.117120] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12444.132392] RSP: 0000:ffffaf9fc15234f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12444.137556] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [12444.141064] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [12444.144710] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [12444.148283] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [12444.152036] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [12444.154501] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [12444.158094] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12444.160462] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [12444.164654] Call Trace: [12444.165647] [12444.166704] ? show_regs.cold.9+0x22/0x2f [12444.168338] ? watchdog_timer_fn.cold.6+0x62/0x10a [12444.169857] ? __hrtimer_run_queues+0x1c2/0x4d0 [12444.171678] ? watchdog+0x40/0x40 [12444.172804] ? hrtimer_interrupt+0x128/0x2c0 [12444.174291] ? smp_apic_timer_interrupt+0xa6/0x270 [12444.177056] ? apic_timer_interrupt+0xf/0x20 [12444.179362] [12444.180327] ? native_safe_halt+0x2e/0x50 [12444.183348] kvm_wait+0xcd/0xe0 [12444.184845] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12444.187250] do_raw_spin_lock+0x177/0x190 [12444.189298] _raw_spin_lock+0x21/0x30 [12444.191010] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12444.193524] ? do_raw_spin_unlock+0x75/0x190 [12444.197836] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12444.200305] ? do_wait_intr+0xf0/0xf0 [12444.202052] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12444.205320] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [12444.208337] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12444.211334] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12444.214222] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12444.216485] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [12444.219189] mdc_intent_lock+0x353/0x7e0 [mdc] [12444.221138] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12444.224075] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12444.227748] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12444.230799] lmv_intent_open+0x338/0x1150 [lmv] [12444.233780] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12444.236484] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12444.239522] lmv_intent_lock+0x1dd/0x4c0 [lmv] [12444.241511] ll_intent_lock+0xc5/0xaa0 [lustre] [12444.244165] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12444.251106] ll_lookup_it+0xe0e/0x2270 [lustre] [12444.254105] ? do_raw_spin_unlock+0x75/0x190 [12444.256632] ? _raw_spin_unlock+0x12/0x30 [12444.258433] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [12444.261466] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [12444.263553] ll_atomic_open+0x36a/0xd70 [lustre] [12444.266297] lookup_open+0x356/0xb40 [12444.267846] do_last+0x313/0xfc0 [12444.269377] ? nd_jump_root+0xe5/0x160 [12444.271049] ? path_init+0x437/0x520 [12444.272427] path_openat+0xf7/0x500 [12444.274244] ? do_raw_spin_unlock+0x75/0x190 [12444.276227] do_filp_open+0x99/0x140 [12444.278764] ? getname_flags+0x6e/0x330 [12444.280813] ? __check_object_size+0xff/0x256 [12444.282444] ? do_raw_spin_unlock+0x75/0x190 [12444.284008] ? _raw_spin_unlock+0x12/0x30 [12444.286311] do_sys_openat2+0x2b4/0x410 [12444.288086] do_sys_open+0x73/0xa0 [12444.290204] __x64_sys_openat+0x24/0x30 [12444.292112] do_syscall_64+0xc1/0x440 [12444.293945] entry_SYSCALL_64_after_hwframe+0x49/0xae [12444.295806] RIP: 0033:0x7ff54af45332 [12444.297016] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [12444.304514] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [12444.309249] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [12444.313205] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [12444.316325] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [12444.319565] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [12444.322934] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [12454.129315] rcu: INFO: rcu_sched self-detected stall on CPU [12454.135519] rcu: 0-...!: (5586502 ticks this GP) idle=98a/1/0x4000000000000002 softirq=125154/125154 fqs=959 [12454.144951] (t=10500289 jiffies g=253585 q=6225) [12454.148388] rcu: rcu_sched kthread starved for 10495249 jiffies! g253585 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [12454.157099] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [12454.161777] rcu: RCU grace-period kthread stack dump: [12454.165428] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [12454.176917] Call Trace: [12454.178094] __schedule+0x351/0xcb0 [12454.179545] schedule+0xc0/0x180 [12454.180588] schedule_timeout+0xb4/0x190 [12454.188810] ? do_raw_spin_unlock+0x75/0x190 [12454.194987] ? __next_timer_interrupt+0x160/0x160 [12454.201976] ? prepare_to_swait_event+0xf3/0x140 [12454.207679] rcu_gp_kthread+0x8dd/0x15b0 [12454.213331] ? force_qs_rnp+0x2c0/0x2c0 [12454.216794] kthread+0x1d1/0x200 [12454.222172] ? set_kthread_struct+0x70/0x70 [12454.225323] ret_from_fork+0x1f/0x30 [12454.231706] rcu: Stack dump where RCU GP kthread last ran: [12454.239391] Sending NMI from CPU 0 to CPUs 1: [12454.252022] NMI backtrace for cpu 1 [12454.252029] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12454.252032] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12454.252033] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12454.254890] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12454.254901] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 [12454.254905] RAX: 0000000000007e50 RBX: ffff96848811cae0 RCX: 0000000000000001 [12454.254907] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [12454.254908] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [12454.254910] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [12454.254912] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [12454.254913] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [12454.254915] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12454.254917] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [12454.254918] Call Trace: [12454.254919] [12454.254920] ? show_regs.cold.9+0x22/0x2f [12454.254921] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [12454.254922] ? nmi_cpu_backtrace_handler+0x15/0x40 [12454.254923] ? nmi_handle+0xbd/0x210 [12454.254924] ? default_do_nmi+0x4b/0x1b0 [12454.254925] ? do_nmi+0x280/0x380 [12454.254926] ? end_repeat_nmi+0x16/0x1f [12454.254927] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12454.254928] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12454.254929] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12454.254930] [12454.254931] do_raw_spin_lock+0x177/0x190 [12454.254932] _raw_spin_lock+0x21/0x30 [12454.254933] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12454.254934] ? do_raw_spin_unlock+0x75/0x190 [12454.254935] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12454.254936] ? do_wait_intr+0xf0/0xf0 [12454.254937] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12454.254938] mdc_close+0x53b/0xd70 [mdc] [12454.254939] lmv_close+0x18d/0x490 [lmv] [12454.254940] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [12454.254941] ll_close_inode_openhandle+0x493/0xea0 [lustre] [12454.254942] ll_md_real_close+0xa5/0x2a0 [lustre] [12454.254943] ll_md_close+0x274/0x800 [lustre] [12454.254945] ll_file_release+0x128/0xc50 [lustre] [12454.254946] __fput+0x10d/0x440 [12454.254946] ____fput+0x12/0x20 [12454.254947] task_work_run+0xc6/0x110 [12454.254948] exit_to_usermode_loop+0x1dd/0x1f0 [12454.254949] do_syscall_64+0x3d6/0x440 [12454.254950] entry_SYSCALL_64_after_hwframe+0x49/0xae [12454.254951] RIP: 0033:0x7ff54af45cd8 [12454.254952] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [12454.254954] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [12454.254956] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [12454.254957] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [12454.254958] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [12454.254960] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [12454.254961] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12454.256806] NMI backtrace for cpu 0 [12454.499754] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12454.509845] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12454.523316] Call Trace: [12454.524928] [12454.528948] ? dump_stack+0xbb/0x10e [12454.530255] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [12454.532241] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [12454.535940] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [12454.545856] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [12454.553740] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [12454.559972] ? rcu_dump_cpu_stacks+0x11b/0x171 [12454.564688] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [12454.567552] ? update_process_times+0x6d/0xc0 [12454.569916] ? tick_sched_handle.isra.18+0x27/0x90 [12454.572978] ? tick_sched_timer+0x53/0xb0 [12454.575033] ? __hrtimer_run_queues+0x1c2/0x4d0 [12454.579489] ? tick_sched_handle.isra.18+0x90/0x90 [12454.584560] ? hrtimer_interrupt+0x128/0x2c0 [12454.587025] ? smp_apic_timer_interrupt+0xa6/0x270 [12454.591046] ? apic_timer_interrupt+0xf/0x20 [12454.593408] [12454.596094] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [12454.600744] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12454.604734] ? do_raw_spin_lock+0x177/0x190 [12454.609903] ? _raw_spin_lock+0x21/0x30 [12454.619787] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12454.629567] ? obd_mod_free+0x46/0x140 [ptlrpc] [12454.634403] ? mdc_commit_open+0x91/0xb0 [mdc] [12454.638496] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12454.644725] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12454.651109] ? after_reply+0xe2c/0x1780 [ptlrpc] [12454.654769] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12454.661009] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [12454.664710] ? do_wait_intr+0xf0/0xf0 [12454.665542] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12454.672805] ? kthread+0x1d1/0x200 [12454.678299] ? set_kthread_struct+0x70/0x70 [12454.680093] ? ret_from_fork+0x1f/0x30 [12459.490167] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10500s! [12459.509830] Showing busy workqueues and worker pools: [12459.520690] workqueue events: flags=0x0 [12459.528390] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12459.543840] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [12459.561328] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [12459.567270] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [12459.577158] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12459.590805] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [12459.605225] workqueue events_long: flags=0x0 [12459.609212] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12459.619895] pending: lu_objects_flush [obdclass] [12459.624956] workqueue events_unbound: flags=0x2 [12459.626837] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [12459.637231] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [12459.651342] workqueue events_power_efficient: flags=0x80 [12459.665054] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12459.667048] pending: neigh_periodic_work [12459.676510] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12459.686277] pending: do_cache_clean, check_lifetime [12459.693452] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12459.696022] pending: neigh_periodic_work [12459.697807] workqueue writeback: flags=0x4a [12459.699550] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [12459.702008] pending: wb_workfn, wb_workfn [12459.703749] workqueue xprtiod: flags=0xa [12459.709537] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [12459.717777] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [12459.724499] workqueue ipv6_addrconf: flags=0x40008 [12459.728358] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12459.732937] pending: addrconf_verify_work [12459.738070] workqueue knbd0-recv: flags=0x1a [12459.739858] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12459.741963] in-flight: 657:recv_work [12459.743703] workqueue ptlrpc_pinger: flags=0x2 [12459.745335] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12459.749687] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [12459.752968] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10500s workers=4 idle: 20831 12642 28859 [12459.757311] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12449s workers=2 idle: 659 [12464.059600] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [12464.063798] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12464.091816] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12464.096646] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12464.100962] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12464.104463] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12464.109257] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12464.119965] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12464.125962] RAX: 000000000000105d RBX: ffff96848811cae0 RCX: 0000000000000001 [12464.132985] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [12464.138268] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [12464.141326] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [12464.146031] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [12464.152001] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [12464.157727] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12464.161677] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [12464.164565] Call Trace: [12464.165716] [12464.166611] ? show_regs.cold.9+0x22/0x2f [12464.168267] ? watchdog_timer_fn.cold.6+0x62/0x10a [12464.170734] ? __hrtimer_run_queues+0x1c2/0x4d0 [12464.174344] ? watchdog+0x40/0x40 [12464.175690] ? hrtimer_interrupt+0x128/0x2c0 [12464.177223] ? smp_apic_timer_interrupt+0xa6/0x270 [12464.179470] ? apic_timer_interrupt+0xf/0x20 [12464.181156] [12464.182053] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12464.184328] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12464.186419] do_raw_spin_lock+0x177/0x190 [12464.189438] _raw_spin_lock+0x21/0x30 [12464.192120] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12464.203153] process_one_work+0x2c8/0x700 [12464.206712] worker_thread+0x296/0x6e0 [12464.209714] ? rescuer_thread+0x570/0x570 [12464.212811] kthread+0x1d1/0x200 [12464.214979] ? set_kthread_struct+0x70/0x70 [12464.217150] ret_from_fork+0x1f/0x30 [12472.060286] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [12472.064442] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12472.090560] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12472.093407] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12472.097913] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12472.100825] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12472.108908] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12472.112876] RAX: 000000000000460a RBX: ffff96848811cae0 RCX: 0000000000000001 [12472.116136] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [12472.120047] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [12472.123665] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [12472.126873] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [12472.131505] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [12472.134889] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12472.137894] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [12472.140640] Call Trace: [12472.142267] [12472.143350] ? show_regs.cold.9+0x22/0x2f [12472.145072] ? watchdog_timer_fn.cold.6+0x62/0x10a [12472.147117] ? __hrtimer_run_queues+0x1c2/0x4d0 [12472.148790] ? watchdog+0x40/0x40 [12472.149897] ? hrtimer_interrupt+0x128/0x2c0 [12472.151551] ? smp_apic_timer_interrupt+0xa6/0x270 [12472.153198] ? apic_timer_interrupt+0xf/0x20 [12472.154839] [12472.155720] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12472.157980] do_raw_spin_lock+0x177/0x190 [12472.159806] _raw_spin_lock+0x21/0x30 [12472.161465] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12472.163604] ? do_raw_spin_unlock+0x75/0x190 [12472.165281] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12472.167775] ? do_wait_intr+0xf0/0xf0 [12472.169223] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12472.171506] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [12472.173813] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12472.176943] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12472.180700] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12472.184540] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [12472.187334] mdc_intent_lock+0x353/0x7e0 [mdc] [12472.190007] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12472.194959] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12472.200007] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12472.203089] lmv_intent_open+0x338/0x1150 [lmv] [12472.204398] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12472.206975] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12472.209794] lmv_intent_lock+0x1dd/0x4c0 [lmv] [12472.213007] ll_intent_lock+0xc5/0xaa0 [lustre] [12472.216125] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12472.222253] ll_lookup_it+0xe0e/0x2270 [lustre] [12472.229531] ? do_raw_spin_unlock+0x75/0x190 [12472.232739] ? _raw_spin_unlock+0x12/0x30 [12472.233957] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [12472.237854] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [12472.243517] ll_atomic_open+0x36a/0xd70 [lustre] [12472.255991] lookup_open+0x356/0xb40 [12472.260040] do_last+0x313/0xfc0 [12472.261213] ? nd_jump_root+0xe5/0x160 [12472.262381] ? path_init+0x437/0x520 [12472.263778] path_openat+0xf7/0x500 [12472.265486] ? do_raw_spin_unlock+0x75/0x190 [12472.267569] do_filp_open+0x99/0x140 [12472.269959] ? getname_flags+0x6e/0x330 [12472.274643] ? __check_object_size+0xff/0x256 [12472.278567] ? do_raw_spin_unlock+0x75/0x190 [12472.282080] ? _raw_spin_unlock+0x12/0x30 [12472.285832] do_sys_openat2+0x2b4/0x410 [12472.287123] do_sys_open+0x73/0xa0 [12472.288160] __x64_sys_openat+0x24/0x30 [12472.290160] do_syscall_64+0xc1/0x440 [12472.291852] entry_SYSCALL_64_after_hwframe+0x49/0xae [12472.293648] RIP: 0033:0x7ff54af45332 [12472.294558] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [12472.306550] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [12472.314047] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [12472.317474] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [12472.324035] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [12472.328562] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [12472.334153] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [12480.058027] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [12480.062972] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12480.118410] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12480.131717] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12480.139032] RIP: 0010:native_safe_halt+0x2e/0x50 [12480.141980] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12480.152998] RSP: 0018:ffffaf9fc1543ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12480.156013] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [12480.159960] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c20b4154 [12480.167423] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [12480.171942] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [12480.176256] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [12480.179964] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [12480.186055] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12480.190676] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [12480.197797] Call Trace: [12480.199076] [12480.200543] ? show_regs.cold.9+0x22/0x2f [12480.202704] ? watchdog_timer_fn.cold.6+0x62/0x10a [12480.205728] ? __hrtimer_run_queues+0x1c2/0x4d0 [12480.208097] ? watchdog+0x40/0x40 [12480.209860] ? hrtimer_interrupt+0x128/0x2c0 [12480.211451] ? smp_apic_timer_interrupt+0xa6/0x270 [12480.218951] ? apic_timer_interrupt+0xf/0x20 [12480.224833] [12480.228323] ? native_safe_halt+0x2e/0x50 [12480.232079] kvm_wait+0xcd/0xe0 [12480.234111] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12480.237373] do_raw_spin_lock+0x177/0x190 [12480.240858] _raw_spin_lock+0x21/0x30 [12480.248446] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12480.250713] ? do_raw_spin_unlock+0x75/0x190 [12480.257173] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12480.260528] ? do_wait_intr+0xf0/0xf0 [12480.262988] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12480.266070] mdc_close+0x53b/0xd70 [mdc] [12480.268799] lmv_close+0x18d/0x490 [lmv] [12480.271404] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [12480.274331] ll_close_inode_openhandle+0x493/0xea0 [lustre] [12480.277025] ll_md_real_close+0xa5/0x2a0 [lustre] [12480.279853] ll_md_close+0x274/0x800 [lustre] [12480.281953] ll_file_release+0x128/0xc50 [lustre] [12480.284329] __fput+0x10d/0x440 [12480.285868] ____fput+0x12/0x20 [12480.287692] task_work_run+0xc6/0x110 [12480.290077] exit_to_usermode_loop+0x1dd/0x1f0 [12480.292337] do_syscall_64+0x3d6/0x440 [12480.294699] entry_SYSCALL_64_after_hwframe+0x49/0xae [12480.297485] RIP: 0033:0x7ff54af45cd8 [12480.299115] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [12480.309831] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [12480.313332] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [12480.316662] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [12480.320112] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [12480.323864] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [12480.327154] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12484.055127] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [12484.058613] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12484.080462] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12484.085748] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12484.089918] RIP: 0010:native_safe_halt+0x2e/0x50 [12484.092417] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12484.101018] RSP: 0000:ffffaf9fc1b97bb8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [12484.104729] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [12484.110130] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff96848811cae0 [12484.113536] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [12484.117270] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [12484.121293] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12484.125387] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [12484.129687] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12484.131932] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [12484.134567] Call Trace: [12484.135983] [12484.136965] ? show_regs.cold.9+0x22/0x2f [12484.139195] ? watchdog_timer_fn.cold.6+0x62/0x10a [12484.141561] ? __hrtimer_run_queues+0x1c2/0x4d0 [12484.143878] ? watchdog+0x40/0x40 [12484.145598] ? hrtimer_interrupt+0x128/0x2c0 [12484.147669] ? smp_apic_timer_interrupt+0xa6/0x270 [12484.149874] ? apic_timer_interrupt+0xf/0x20 [12484.151925] [12484.152833] ? native_safe_halt+0x2e/0x50 [12484.154961] kvm_wait+0xcd/0xe0 [12484.156137] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12484.158752] do_raw_spin_lock+0x177/0x190 [12484.160593] _raw_spin_lock+0x21/0x30 [12484.162643] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12484.165366] obd_mod_free+0x46/0x140 [ptlrpc] [12484.167658] mdc_commit_open+0x91/0xb0 [mdc] [12484.169742] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12484.172627] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12484.175593] after_reply+0xe2c/0x1780 [ptlrpc] [12484.177954] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12484.180522] ptlrpcd+0xa82/0xe40 [ptlrpc] [12484.182841] ? do_wait_intr+0xf0/0xf0 [12484.184692] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12484.186870] kthread+0x1d1/0x200 [12484.187842] ? set_kthread_struct+0x70/0x70 [12484.189375] ret_from_fork+0x1f/0x30 [12490.214249] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10531s! [12490.226425] Showing busy workqueues and worker pools: [12490.228327] workqueue events: flags=0x0 [12490.232496] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12490.244373] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [12490.249534] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [12490.256381] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [12490.266111] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12490.269777] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [12490.273014] workqueue events_long: flags=0x0 [12490.275219] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12490.278386] pending: lu_objects_flush [obdclass] [12490.280717] workqueue events_unbound: flags=0x2 [12490.282894] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [12490.285910] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [12490.290828] workqueue events_power_efficient: flags=0x80 [12490.293734] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12490.296909] pending: neigh_periodic_work [12490.298814] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12490.302780] pending: do_cache_clean, check_lifetime [12490.305103] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12490.308224] pending: neigh_periodic_work [12490.310082] workqueue writeback: flags=0x4a [12490.312074] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [12490.314928] pending: wb_workfn, wb_workfn [12490.317259] workqueue xprtiod: flags=0xa [12490.319190] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [12490.322077] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [12490.325780] workqueue ipv6_addrconf: flags=0x40008 [12490.327959] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12490.330460] pending: addrconf_verify_work [12490.332561] workqueue knbd0-recv: flags=0x1a [12490.334454] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12490.337025] in-flight: 657:recv_work [12490.338690] workqueue ptlrpc_pinger: flags=0x2 [12490.340608] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12490.343091] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [12490.345463] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10531s workers=4 idle: 20831 12642 28859 [12490.348951] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12480s workers=2 idle: 659 [12492.059041] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [12492.064233] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12492.081748] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12492.085895] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12492.088957] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12492.091720] RIP: 0010:native_safe_halt+0x2e/0x50 [12492.093500] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12492.100870] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12492.104038] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [12492.107073] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [12492.110308] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [12492.113039] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [12492.115733] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [12492.118644] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [12492.121982] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12492.124313] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [12492.127002] Call Trace: [12492.128001] [12492.128893] ? show_regs.cold.9+0x22/0x2f [12492.130636] ? watchdog_timer_fn.cold.6+0x62/0x10a [12492.132610] ? __hrtimer_run_queues+0x1c2/0x4d0 [12492.134631] ? watchdog+0x40/0x40 [12492.138818] ? hrtimer_interrupt+0x128/0x2c0 [12492.141722] ? smp_apic_timer_interrupt+0xa6/0x270 [12492.144309] ? apic_timer_interrupt+0xf/0x20 [12492.147836] [12492.149848] ? native_safe_halt+0x2e/0x50 [12492.151366] kvm_wait+0xcd/0xe0 [12492.153397] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12492.156587] do_raw_spin_lock+0x177/0x190 [12492.158576] _raw_spin_lock+0x21/0x30 [12492.160809] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12492.164181] process_one_work+0x2c8/0x700 [12492.165794] worker_thread+0x296/0x6e0 [12492.168417] ? rescuer_thread+0x570/0x570 [12492.170415] kthread+0x1d1/0x200 [12492.173196] ? set_kthread_struct+0x70/0x70 [12492.177216] ret_from_fork+0x1f/0x30 [12500.067120] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [12500.075056] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12500.128369] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12500.138962] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12500.144905] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12500.150690] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12500.164415] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12500.178622] RAX: 00000000000023d7 RBX: ffff96848811cae0 RCX: 0000000000000001 [12500.185709] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [12500.197436] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [12500.207956] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [12500.212695] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [12500.223132] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [12500.230325] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12500.234454] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [12500.240468] Call Trace: [12500.241409] [12500.246043] ? show_regs.cold.9+0x22/0x2f [12500.247792] ? watchdog_timer_fn.cold.6+0x62/0x10a [12500.256223] ? __hrtimer_run_queues+0x1c2/0x4d0 [12500.258471] ? watchdog+0x40/0x40 [12500.259819] ? hrtimer_interrupt+0x128/0x2c0 [12500.261667] ? smp_apic_timer_interrupt+0xa6/0x270 [12500.264673] ? apic_timer_interrupt+0xf/0x20 [12500.267338] [12500.268409] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12500.272770] do_raw_spin_lock+0x177/0x190 [12500.276240] _raw_spin_lock+0x21/0x30 [12500.278631] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12500.281418] ? do_raw_spin_unlock+0x75/0x190 [12500.284338] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12500.292109] ? do_wait_intr+0xf0/0xf0 [12500.297276] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12500.305141] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [12500.309259] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12500.312789] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12500.319207] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12500.324414] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [12500.328479] mdc_intent_lock+0x353/0x7e0 [mdc] [12500.331769] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12500.334634] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12500.338327] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12500.341252] lmv_intent_open+0x338/0x1150 [lmv] [12500.344560] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12500.348708] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12500.352132] lmv_intent_lock+0x1dd/0x4c0 [lmv] [12500.356372] ll_intent_lock+0xc5/0xaa0 [lustre] [12500.358641] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12500.361207] ll_lookup_it+0xe0e/0x2270 [lustre] [12500.365191] ? do_raw_spin_unlock+0x75/0x190 [12500.367844] ? _raw_spin_unlock+0x12/0x30 [12500.371031] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [12500.375092] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [12500.379917] ll_atomic_open+0x36a/0xd70 [lustre] [12500.383457] lookup_open+0x356/0xb40 [12500.386243] do_last+0x313/0xfc0 [12500.388693] ? nd_jump_root+0xe5/0x160 [12500.391203] ? path_init+0x437/0x520 [12500.394136] path_openat+0xf7/0x500 [12500.396701] ? do_raw_spin_unlock+0x75/0x190 [12500.399543] do_filp_open+0x99/0x140 [12500.401089] ? getname_flags+0x6e/0x330 [12500.403789] ? __check_object_size+0xff/0x256 [12500.407438] ? do_raw_spin_unlock+0x75/0x190 [12500.410480] ? _raw_spin_unlock+0x12/0x30 [12500.414484] do_sys_openat2+0x2b4/0x410 [12500.417179] do_sys_open+0x73/0xa0 [12500.419834] __x64_sys_openat+0x24/0x30 [12500.423673] do_syscall_64+0xc1/0x440 [12500.427734] entry_SYSCALL_64_after_hwframe+0x49/0xae [12500.431505] RIP: 0033:0x7ff54af45332 [12500.443300] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [12500.463407] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [12500.469697] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [12500.474694] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [12500.481923] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [12500.489549] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [12500.494881] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [12508.058270] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [12508.068900] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12508.111830] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12508.119317] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12508.125685] RIP: 0010:__pv_queued_spin_lock_slowpath+0x328/0x6d0 [12508.130369] Code: d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 05 18 d4 dc 02 01 e0 48 83 05 06 d4 dc 02 01 48 83 05 be d5 dc 02 01 8b 45 08 85 [12508.145331] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12508.152672] RAX: 000000000000760a RBX: ffff96848811cae0 RCX: 0000000000000001 [12508.157341] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [12508.163317] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [12508.168906] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [12508.174601] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [12508.179652] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [12508.185679] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12508.190637] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [12508.196754] Call Trace: [12508.198435] [12508.200545] ? show_regs.cold.9+0x22/0x2f [12508.203879] ? watchdog_timer_fn.cold.6+0x62/0x10a [12508.206394] ? __hrtimer_run_queues+0x1c2/0x4d0 [12508.210771] ? watchdog+0x40/0x40 [12508.213748] ? hrtimer_interrupt+0x128/0x2c0 [12508.217635] ? smp_apic_timer_interrupt+0xa6/0x270 [12508.220818] ? apic_timer_interrupt+0xf/0x20 [12508.223717] [12508.225807] ? __pv_queued_spin_lock_slowpath+0x328/0x6d0 [12508.229821] do_raw_spin_lock+0x177/0x190 [12508.232689] _raw_spin_lock+0x21/0x30 [12508.235324] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12508.239932] ? do_raw_spin_unlock+0x75/0x190 [12508.243800] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12508.247794] ? do_wait_intr+0xf0/0xf0 [12508.251608] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12508.254673] mdc_close+0x53b/0xd70 [mdc] [12508.256915] lmv_close+0x18d/0x490 [lmv] [12508.259676] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [12508.263289] ll_close_inode_openhandle+0x493/0xea0 [lustre] [12508.267537] ll_md_real_close+0xa5/0x2a0 [lustre] [12508.270627] ll_md_close+0x274/0x800 [lustre] [12508.274178] ll_file_release+0x128/0xc50 [lustre] [12508.277516] __fput+0x10d/0x440 [12508.280585] ____fput+0x12/0x20 [12508.282686] task_work_run+0xc6/0x110 [12508.284984] exit_to_usermode_loop+0x1dd/0x1f0 [12508.287685] do_syscall_64+0x3d6/0x440 [12508.289747] entry_SYSCALL_64_after_hwframe+0x49/0xae [12508.293061] RIP: 0033:0x7ff54af45cd8 [12508.295313] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [12508.305896] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [12508.310788] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [12508.315418] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [12508.320088] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [12508.325637] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [12508.330263] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12516.056204] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [12516.065182] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12516.126988] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12516.133829] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12516.138774] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [12516.140802] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [12516.147465] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12516.149830] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [12516.153474] RDX: 0000000000003413 RSI: 0000000000000003 RDI: ffff96848811cae0 [12516.157817] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [12516.162442] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [12516.165488] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12516.170438] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [12516.174264] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12516.177079] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [12516.181167] Call Trace: [12516.182298] [12516.183266] ? show_regs.cold.9+0x22/0x2f [12516.185207] ? watchdog_timer_fn.cold.6+0x62/0x10a [12516.188166] ? __hrtimer_run_queues+0x1c2/0x4d0 [12516.190714] ? watchdog+0x40/0x40 [12516.192384] ? hrtimer_interrupt+0x128/0x2c0 [12516.194406] ? smp_apic_timer_interrupt+0xa6/0x270 [12516.197281] ? apic_timer_interrupt+0xf/0x20 [12516.199355] [12516.200325] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [12516.203229] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12516.205773] do_raw_spin_lock+0x177/0x190 [12516.207522] _raw_spin_lock+0x21/0x30 [12516.209703] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12516.212343] obd_mod_free+0x46/0x140 [ptlrpc] [12516.214658] mdc_commit_open+0x91/0xb0 [mdc] [12516.217250] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12516.219749] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12516.222506] after_reply+0xe2c/0x1780 [ptlrpc] [12516.225982] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12516.228963] ptlrpcd+0xa82/0xe40 [ptlrpc] [12516.231407] ? do_wait_intr+0xf0/0xf0 [12516.233233] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12516.236452] kthread+0x1d1/0x200 [12516.238127] ? set_kthread_struct+0x70/0x70 [12516.240721] ret_from_fork+0x1f/0x30 [12520.062310] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [12520.067845] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12520.099094] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12520.103257] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12520.108667] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12520.111622] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12520.115502] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12520.126050] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12520.131030] RAX: 0000000000007f86 RBX: ffff96848811cae0 RCX: 0000000000000001 [12520.133759] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [12520.139304] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [12520.144616] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [12520.152242] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [12520.157685] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [12520.162330] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12520.164564] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [12520.167939] Call Trace: [12520.169333] [12520.170223] ? show_regs.cold.9+0x22/0x2f [12520.171605] ? watchdog_timer_fn.cold.6+0x62/0x10a [12520.178189] ? __hrtimer_run_queues+0x1c2/0x4d0 [12520.181964] ? watchdog+0x40/0x40 [12520.183868] ? hrtimer_interrupt+0x128/0x2c0 [12520.185416] ? smp_apic_timer_interrupt+0xa6/0x270 [12520.187137] ? apic_timer_interrupt+0xf/0x20 [12520.188700] [12520.189526] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12520.191521] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12520.193445] do_raw_spin_lock+0x177/0x190 [12520.195192] _raw_spin_lock+0x21/0x30 [12520.197086] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12520.200130] process_one_work+0x2c8/0x700 [12520.202202] worker_thread+0x296/0x6e0 [12520.204158] ? rescuer_thread+0x570/0x570 [12520.205941] kthread+0x1d1/0x200 [12520.207436] ? set_kthread_struct+0x70/0x70 [12520.209319] ret_from_fork+0x1f/0x30 [12520.415045] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10561s! [12520.424118] Showing busy workqueues and worker pools: [12520.431061] workqueue events: flags=0x0 [12520.434134] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12520.440498] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [12520.451776] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [12520.460314] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [12520.477034] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12520.491764] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [12520.497925] workqueue events_long: flags=0x0 [12520.500039] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12520.503373] pending: lu_objects_flush [obdclass] [12520.507819] workqueue events_unbound: flags=0x2 [12520.509827] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [12520.512774] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [12520.516798] workqueue events_power_efficient: flags=0x80 [12520.522192] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12520.528836] pending: neigh_periodic_work [12520.531488] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12520.548662] pending: do_cache_clean, check_lifetime [12520.550505] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12520.555718] pending: neigh_periodic_work [12520.557639] workqueue writeback: flags=0x4a [12520.559523] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [12520.562268] pending: wb_workfn, wb_workfn [12520.564444] workqueue xprtiod: flags=0xa [12520.566288] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [12520.568960] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [12520.572897] workqueue ipv6_addrconf: flags=0x40008 [12520.575338] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12520.578724] pending: addrconf_verify_work [12520.588900] workqueue knbd0-recv: flags=0x1a [12520.591835] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12520.598776] in-flight: 657:recv_work [12520.602766] workqueue ptlrpc_pinger: flags=0x2 [12520.605628] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12520.616194] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [12520.624702] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10561s workers=4 idle: 20831 12642 28859 [12520.633853] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12510s workers=2 idle: 659 [12528.065755] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [12528.074644] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12528.115769] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12528.125667] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12528.131582] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12528.135670] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12528.147656] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [12528.154064] RAX: 0000000000007f92 RBX: ffff96848811cae0 RCX: 0000000000000001 [12528.158396] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [12528.162920] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [12528.167360] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [12528.174222] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [12528.178693] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [12528.183457] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12528.187711] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [12528.192711] Call Trace: [12528.194396] [12528.195619] ? show_regs.cold.9+0x22/0x2f [12528.199027] ? watchdog_timer_fn.cold.6+0x62/0x10a [12528.202661] ? __hrtimer_run_queues+0x1c2/0x4d0 [12528.206242] ? watchdog+0x40/0x40 [12528.208513] ? hrtimer_interrupt+0x128/0x2c0 [12528.211451] ? smp_apic_timer_interrupt+0xa6/0x270 [12528.214789] ? apic_timer_interrupt+0xf/0x20 [12528.217411] [12528.218659] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12528.221984] do_raw_spin_lock+0x177/0x190 [12528.223983] _raw_spin_lock+0x21/0x30 [12528.226505] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12528.229750] ? do_raw_spin_unlock+0x75/0x190 [12528.232451] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12528.236279] ? do_wait_intr+0xf0/0xf0 [12528.238472] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12528.242120] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [12528.245751] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12528.249928] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12528.252976] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12528.257409] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [12528.260271] mdc_intent_lock+0x353/0x7e0 [mdc] [12528.263105] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12528.266709] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12528.270442] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12528.274130] lmv_intent_open+0x338/0x1150 [lmv] [12528.276705] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12528.280083] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12528.283135] lmv_intent_lock+0x1dd/0x4c0 [lmv] [12528.286279] ll_intent_lock+0xc5/0xaa0 [lustre] [12528.288630] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12528.292297] ll_lookup_it+0xe0e/0x2270 [lustre] [12528.294586] ? do_raw_spin_unlock+0x75/0x190 [12528.296502] ? _raw_spin_unlock+0x12/0x30 [12528.298405] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [12528.300835] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [12528.303609] ll_atomic_open+0x36a/0xd70 [lustre] [12528.306437] lookup_open+0x356/0xb40 [12528.308130] do_last+0x313/0xfc0 [12528.309597] ? nd_jump_root+0xe5/0x160 [12528.311405] ? path_init+0x437/0x520 [12528.313247] path_openat+0xf7/0x500 [12528.315085] ? do_raw_spin_unlock+0x75/0x190 [12528.317246] do_filp_open+0x99/0x140 [12528.318912] ? getname_flags+0x6e/0x330 [12528.320697] ? __check_object_size+0xff/0x256 [12528.325518] ? do_raw_spin_unlock+0x75/0x190 [12528.331446] ? _raw_spin_unlock+0x12/0x30 [12528.336996] do_sys_openat2+0x2b4/0x410 [12528.343470] do_sys_open+0x73/0xa0 [12528.347014] __x64_sys_openat+0x24/0x30 [12528.353144] do_syscall_64+0xc1/0x440 [12528.354968] entry_SYSCALL_64_after_hwframe+0x49/0xae [12528.359461] RIP: 0033:0x7ff54af45332 [12528.362609] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [12528.383407] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [12528.394334] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [12528.400560] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [12528.417246] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [12528.427027] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [12528.431234] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [12536.058072] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [12536.060623] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12536.086723] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12536.091540] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12536.096196] RIP: 0010:native_safe_halt+0x2e/0x50 [12536.098275] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12536.106368] RSP: 0018:ffffaf9fc1543ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12536.109807] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [12536.113287] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c20b4154 [12536.117354] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [12536.120904] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [12536.143225] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [12536.146628] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [12536.152498] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12536.155160] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [12536.163313] Call Trace: [12536.164924] [12536.166217] ? show_regs.cold.9+0x22/0x2f [12536.168852] ? watchdog_timer_fn.cold.6+0x62/0x10a [12536.171282] ? __hrtimer_run_queues+0x1c2/0x4d0 [12536.173266] ? watchdog+0x40/0x40 [12536.174840] ? hrtimer_interrupt+0x128/0x2c0 [12536.184218] ? smp_apic_timer_interrupt+0xa6/0x270 [12536.186439] ? apic_timer_interrupt+0xf/0x20 [12536.188548] [12536.189419] ? native_safe_halt+0x2e/0x50 [12536.191212] kvm_wait+0xcd/0xe0 [12536.193434] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12536.203845] do_raw_spin_lock+0x177/0x190 [12536.206386] _raw_spin_lock+0x21/0x30 [12536.208553] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12536.218244] ? do_raw_spin_unlock+0x75/0x190 [12536.219657] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12536.222549] ? do_wait_intr+0xf0/0xf0 [12536.225907] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12536.227530] mdc_close+0x53b/0xd70 [mdc] [12536.229167] lmv_close+0x18d/0x490 [lmv] [12536.230529] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [12536.234974] ll_close_inode_openhandle+0x493/0xea0 [lustre] [12536.237716] ll_md_real_close+0xa5/0x2a0 [lustre] [12536.240030] ll_md_close+0x274/0x800 [lustre] [12536.245304] ll_file_release+0x128/0xc50 [lustre] [12536.250307] __fput+0x10d/0x440 [12536.254072] ____fput+0x12/0x20 [12536.257619] task_work_run+0xc6/0x110 [12536.261156] exit_to_usermode_loop+0x1dd/0x1f0 [12536.263112] do_syscall_64+0x3d6/0x440 [12536.265458] entry_SYSCALL_64_after_hwframe+0x49/0xae [12536.267744] RIP: 0033:0x7ff54af45cd8 [12536.269182] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [12536.280355] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [12536.291597] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [12536.298207] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [12536.302286] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [12536.316328] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [12536.323940] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12548.055292] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [12548.068092] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [12548.068936] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12548.069014] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12548.069022] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12548.069025] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [12548.069042] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [12548.069047] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12548.069052] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [12548.069054] RDX: 0000000000001491 RSI: 0000000000000003 RDI: ffff96848811cae0 [12548.069057] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [12548.069059] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [12548.069062] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12548.069065] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [12548.069068] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12548.069070] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [12548.069080] Call Trace: [12548.069086] [12548.069094] ? show_regs.cold.9+0x22/0x2f [12548.069104] ? watchdog_timer_fn.cold.6+0x62/0x10a [12548.069412] ? __hrtimer_run_queues+0x1c2/0x4d0 [12548.069418] ? watchdog+0x40/0x40 [12548.069422] ? hrtimer_interrupt+0x128/0x2c0 [12548.069426] ? smp_apic_timer_interrupt+0xa6/0x270 [12548.069434] ? apic_timer_interrupt+0xf/0x20 [12548.069448] [12548.069451] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [12548.069456] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12548.069461] do_raw_spin_lock+0x177/0x190 [12548.069465] _raw_spin_lock+0x21/0x30 [12548.069471] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12548.069904] obd_mod_free+0x46/0x140 [ptlrpc] [12548.070275] mdc_commit_open+0x91/0xb0 [mdc] [12548.070313] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12548.070572] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12548.071256] after_reply+0xe2c/0x1780 [ptlrpc] [12548.071543] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12548.074208] ptlrpcd+0xa82/0xe40 [ptlrpc] [12548.075646] ? do_wait_intr+0xf0/0xf0 [12548.075660] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12548.086941] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12548.087024] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12548.087031] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12548.087038] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12548.087335] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12548.087345] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12548.087349] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12548.087352] RAX: 0000000000007e86 RBX: ffff96848811cae0 RCX: 0000000000000001 [12548.087355] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [12548.087363] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [12548.087365] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [12548.087367] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [12548.087370] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [12548.087373] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12548.087375] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [12548.087382] Call Trace: [12548.087387] [12548.087393] ? show_regs.cold.9+0x22/0x2f [12548.087401] ? watchdog_timer_fn.cold.6+0x62/0x10a [12548.087407] ? __hrtimer_run_queues+0x1c2/0x4d0 [12548.087411] ? watchdog+0x40/0x40 [12548.087415] ? hrtimer_interrupt+0x128/0x2c0 [12548.087419] ? smp_apic_timer_interrupt+0xa6/0x270 [12548.087427] ? apic_timer_interrupt+0xf/0x20 [12548.087432] [12548.087434] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12548.087439] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12548.087455] do_raw_spin_lock+0x177/0x190 [12548.087459] _raw_spin_lock+0x21/0x30 [12548.087674] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12548.087904] process_one_work+0x2c8/0x700 [12548.087920] worker_thread+0x296/0x6e0 [12548.087925] ? rescuer_thread+0x570/0x570 [12548.087928] kthread+0x1d1/0x200 [12548.087932] ? set_kthread_struct+0x70/0x70 [12548.087936] ret_from_fork+0x1f/0x30 [12548.420894] kthread+0x1d1/0x200 [12548.423029] ? set_kthread_struct+0x70/0x70 [12548.427681] ret_from_fork+0x1f/0x30 [12551.135155] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10592s! [12551.142910] Showing busy workqueues and worker pools: [12551.149119] workqueue events: flags=0x0 [12551.150587] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12551.158158] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [12551.166965] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [12551.173347] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [12551.187468] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12551.199032] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [12551.209053] workqueue events_long: flags=0x0 [12551.218856] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12551.227101] pending: lu_objects_flush [obdclass] [12551.232351] workqueue events_unbound: flags=0x2 [12551.235385] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [12551.238618] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [12551.244989] workqueue events_power_efficient: flags=0x80 [12551.249025] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12551.252637] pending: neigh_periodic_work [12551.257048] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12551.275041] pending: do_cache_clean, check_lifetime [12551.282288] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12551.294794] pending: neigh_periodic_work [12551.303241] workqueue writeback: flags=0x4a [12551.307809] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [12551.309890] pending: wb_workfn, wb_workfn [12551.322540] workqueue xprtiod: flags=0xa [12551.326174] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [12551.336472] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [12551.347271] workqueue ipv6_addrconf: flags=0x40008 [12551.357741] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12551.362311] pending: addrconf_verify_work [12551.367095] workqueue knbd0-recv: flags=0x1a [12551.372762] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12551.380814] in-flight: 657:recv_work [12551.383410] workqueue ptlrpc_pinger: flags=0x2 [12551.390167] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12551.398355] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [12551.402854] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10592s workers=4 idle: 20831 12642 28859 [12551.407684] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12541s workers=2 idle: 659 [12556.060247] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [12556.072311] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12556.101988] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12556.108175] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12556.113042] RIP: 0010:load_balance+0x725/0x1090 [12556.115168] Code: 48 83 00 01 9c 58 0f 1f 44 00 00 48 83 05 7a 6d dd 02 01 48 83 05 82 b9 dd 02 01 f6 c4 02 0f 85 22 03 00 00 48 8b 3c 24 57 9d <0f> 1f 44 00 00 8b 84 24 98 00 00 00 a8 02 74 23 83 e0 fd 48 83 05 [12556.125464] RSP: 0000:ffffaf9fc0d8cde0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12556.131937] RAX: 0000000000000006 RBX: ffff96848330a870 RCX: ffff9684c21b3440 [12556.137340] RDX: 0000000000000008 RSI: ffff96848330a7c0 RDI: 0000000000000202 [12556.146337] RBP: ffffaf9fc0d8ce90 R08: ffff9684a0e24f80 R09: 0000000000000000 [12556.151049] R10: ffff96848330a858 R11: 00000000000093a6 R12: ffff96848330a7c0 [12556.154312] R13: ffff9684c2133e08 R14: 000000000000000a R15: ffff9684c2133380 [12556.157442] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [12556.162318] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12556.167247] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [12556.172385] Call Trace: [12556.174222] [12556.176438] ? show_regs.cold.9+0x22/0x2f [12556.178898] ? watchdog_timer_fn.cold.6+0x62/0x10a [12556.182961] ? __hrtimer_run_queues+0x1c2/0x4d0 [12556.186223] ? watchdog+0x40/0x40 [12556.188889] ? hrtimer_interrupt+0x128/0x2c0 [12556.190345] ? smp_apic_timer_interrupt+0xa6/0x270 [12556.194440] ? apic_timer_interrupt+0xf/0x20 [12556.199227] ? apic_timer_interrupt+0xa/0x20 [12556.202092] ? load_balance+0x725/0x1090 [12556.205902] rebalance_domains+0x348/0x580 [12556.208226] ? kvm_clock_get_cycles+0x2c/0x50 [12556.211021] run_rebalance_domains+0x68/0xa0 [12556.215291] __do_softirq+0x134/0x4e8 [12556.219242] irq_exit_rcu+0x1c6/0x1e0 [12556.220828] irq_exit+0x12/0x30 [12556.224608] smp_apic_timer_interrupt+0xe0/0x270 [12556.229658] apic_timer_interrupt+0xf/0x20 [12556.231598] [12556.232989] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12556.235933] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12556.246337] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12556.249920] RAX: 0000000000004d8e RBX: ffff96848811cae0 RCX: 0000000000000001 [12556.252669] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [12556.256425] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [12556.261952] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [12556.267938] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [12556.271108] do_raw_spin_lock+0x177/0x190 [12556.272475] _raw_spin_lock+0x21/0x30 [12556.274161] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12556.276830] ? do_raw_spin_unlock+0x75/0x190 [12556.278171] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12556.280069] ? do_wait_intr+0xf0/0xf0 [12556.281448] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12556.283810] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [12556.286066] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12556.288736] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12556.291005] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12556.297430] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [12556.299537] mdc_intent_lock+0x353/0x7e0 [mdc] [12556.302706] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12556.305840] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12556.309274] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12556.311880] lmv_intent_open+0x338/0x1150 [lmv] [12556.313991] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12556.315812] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12556.318556] lmv_intent_lock+0x1dd/0x4c0 [lmv] [12556.321187] ll_intent_lock+0xc5/0xaa0 [lustre] [12556.323288] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12556.325746] ll_lookup_it+0xe0e/0x2270 [lustre] [12556.328299] ? do_raw_spin_unlock+0x75/0x190 [12556.330194] ? _raw_spin_unlock+0x12/0x30 [12556.332051] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [12556.334778] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [12556.337054] ll_atomic_open+0x36a/0xd70 [lustre] [12556.339294] lookup_open+0x356/0xb40 [12556.340944] do_last+0x313/0xfc0 [12556.342674] ? nd_jump_root+0xe5/0x160 [12556.344152] ? path_init+0x437/0x520 [12556.345858] path_openat+0xf7/0x500 [12556.347447] ? do_raw_spin_unlock+0x75/0x190 [12556.349805] do_filp_open+0x99/0x140 [12556.351151] ? getname_flags+0x6e/0x330 [12556.352869] ? __check_object_size+0xff/0x256 [12556.354922] ? do_raw_spin_unlock+0x75/0x190 [12556.360905] ? _raw_spin_unlock+0x12/0x30 [12556.362973] do_sys_openat2+0x2b4/0x410 [12556.367887] do_sys_open+0x73/0xa0 [12556.370215] __x64_sys_openat+0x24/0x30 [12556.372413] do_syscall_64+0xc1/0x440 [12556.374346] entry_SYSCALL_64_after_hwframe+0x49/0xae [12556.377017] RIP: 0033:0x7ff54af45332 [12556.378992] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [12556.391396] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [12556.398314] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [12556.401898] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [12556.406806] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [12556.413444] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [12556.417761] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [12564.058315] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [12564.061750] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12564.096336] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12564.103058] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12564.108706] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12564.112367] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12564.129295] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12564.135579] RAX: 0000000000007086 RBX: ffff96848811cae0 RCX: 0000000000000001 [12564.140318] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [12564.144735] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [12564.148636] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [12564.152893] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [12564.157215] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [12564.162370] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12564.165638] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [12564.170311] Call Trace: [12564.171855] [12564.173271] ? show_regs.cold.9+0x22/0x2f [12564.175548] ? watchdog_timer_fn.cold.6+0x62/0x10a [12564.178169] ? __hrtimer_run_queues+0x1c2/0x4d0 [12564.181514] ? watchdog+0x40/0x40 [12564.182734] ? hrtimer_interrupt+0x128/0x2c0 [12564.184102] ? smp_apic_timer_interrupt+0xa6/0x270 [12564.187353] ? apic_timer_interrupt+0xf/0x20 [12564.190315] [12564.191518] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12564.194689] do_raw_spin_lock+0x177/0x190 [12564.196695] _raw_spin_lock+0x21/0x30 [12564.198770] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12564.201809] ? do_raw_spin_unlock+0x75/0x190 [12564.203832] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12564.207794] ? do_wait_intr+0xf0/0xf0 [12564.210338] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12564.213590] mdc_close+0x53b/0xd70 [mdc] [12564.216288] lmv_close+0x18d/0x490 [lmv] [12564.218342] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [12564.221856] ll_close_inode_openhandle+0x493/0xea0 [lustre] [12564.225358] ll_md_real_close+0xa5/0x2a0 [lustre] [12564.228352] ll_md_close+0x274/0x800 [lustre] [12564.231731] ll_file_release+0x128/0xc50 [lustre] [12564.234719] __fput+0x10d/0x440 [12564.236795] ____fput+0x12/0x20 [12564.238843] task_work_run+0xc6/0x110 [12564.240129] exit_to_usermode_loop+0x1dd/0x1f0 [12564.243327] do_syscall_64+0x3d6/0x440 [12564.245291] entry_SYSCALL_64_after_hwframe+0x49/0xae [12564.248585] RIP: 0033:0x7ff54af45cd8 [12564.250395] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [12564.260344] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [12564.264080] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [12564.267919] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [12564.272549] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [12564.276709] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [12564.280826] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12576.058597] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [12576.061005] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [12576.064107] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12576.064189] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12576.065147] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12576.074794] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12576.074873] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12576.074882] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12576.074889] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12576.075200] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12576.075211] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12576.075215] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12576.075220] RAX: 0000000000005507 RBX: ffff96848811cae0 RCX: 0000000000000001 [12576.075222] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [12576.075225] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [12576.075227] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [12576.075229] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [12576.075232] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [12576.075235] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12576.075238] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [12576.075247] Call Trace: [12576.075252] [12576.075258] ? show_regs.cold.9+0x22/0x2f [12576.075268] ? watchdog_timer_fn.cold.6+0x62/0x10a [12576.075276] ? __hrtimer_run_queues+0x1c2/0x4d0 [12576.075281] ? watchdog+0x40/0x40 [12576.075285] ? hrtimer_interrupt+0x128/0x2c0 [12576.075289] ? smp_apic_timer_interrupt+0xa6/0x270 [12576.075375] ? apic_timer_interrupt+0xf/0x20 [12576.075379] [12576.075382] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12576.075387] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12576.075393] do_raw_spin_lock+0x177/0x190 [12576.075398] _raw_spin_lock+0x21/0x30 [12576.075403] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12576.075762] process_one_work+0x2c8/0x700 [12576.075770] worker_thread+0x296/0x6e0 [12576.075775] ? rescuer_thread+0x570/0x570 [12576.075779] kthread+0x1d1/0x200 [12576.075786] ? set_kthread_struct+0x70/0x70 [12576.075790] ret_from_fork+0x1f/0x30 [12576.331453] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [12576.339154] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [12576.361669] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12576.368179] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [12576.374747] RDX: 0000000000000a66 RSI: 0000000000000003 RDI: ffff96848811cae0 [12576.380138] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [12576.384949] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [12576.391601] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12576.398618] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [12576.404086] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12576.407804] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [12576.412525] Call Trace: [12576.414281] [12576.415723] ? show_regs.cold.9+0x22/0x2f [12576.421421] ? watchdog_timer_fn.cold.6+0x62/0x10a [12576.424788] ? __hrtimer_run_queues+0x1c2/0x4d0 [12576.428180] ? watchdog+0x40/0x40 [12576.430911] ? hrtimer_interrupt+0x128/0x2c0 [12576.433021] ? smp_apic_timer_interrupt+0xa6/0x270 [12576.437121] ? apic_timer_interrupt+0xf/0x20 [12576.440544] [12576.441850] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [12576.446328] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12576.450478] do_raw_spin_lock+0x177/0x190 [12576.455025] _raw_spin_lock+0x21/0x30 [12576.458901] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12576.462542] obd_mod_free+0x46/0x140 [ptlrpc] [12576.465348] mdc_commit_open+0x91/0xb0 [mdc] [12576.467906] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12576.471473] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12576.474700] after_reply+0xe2c/0x1780 [ptlrpc] [12576.477556] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12576.480664] ptlrpcd+0xa82/0xe40 [ptlrpc] [12576.483326] ? do_wait_intr+0xf0/0xf0 [12576.485759] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12576.488887] kthread+0x1d1/0x200 [12576.490787] ? set_kthread_struct+0x70/0x70 [12576.493126] ret_from_fork+0x1f/0x30 [12581.855123] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10622s! [12581.862089] Showing busy workqueues and worker pools: [12581.871031] workqueue events: flags=0x0 [12581.876025] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12581.882617] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [12581.890628] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [12581.898597] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [12581.908867] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12581.916194] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [12581.928540] workqueue events_long: flags=0x0 [12581.934879] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12581.937096] pending: lu_objects_flush [obdclass] [12581.939702] workqueue events_unbound: flags=0x2 [12581.941805] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [12581.944953] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [12581.950623] workqueue events_power_efficient: flags=0x80 [12581.952860] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12581.955874] pending: neigh_periodic_work [12581.957723] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12581.960690] pending: do_cache_clean, check_lifetime [12581.963432] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12581.966113] pending: neigh_periodic_work [12581.972054] workqueue writeback: flags=0x4a [12581.977726] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [12581.980898] pending: wb_workfn, wb_workfn [12581.982865] workqueue xprtiod: flags=0xa [12581.987865] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [12581.991016] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [12581.997902] workqueue ipv6_addrconf: flags=0x40008 [12582.012929] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12582.023710] pending: addrconf_verify_work [12582.030201] workqueue knbd0-recv: flags=0x1a [12582.036285] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12582.044151] in-flight: 657:recv_work [12582.054777] workqueue ptlrpc_pinger: flags=0x2 [12582.059034] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12582.069612] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [12582.082142] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10623s workers=4 idle: 20831 12642 28859 [12582.098722] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12572s workers=2 idle: 659 [12584.062958] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [12584.066048] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12584.105346] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12584.114591] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12584.122948] RIP: 0010:native_safe_halt+0x2e/0x50 [12584.127618] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12584.149573] RSP: 0000:ffffaf9fc15234f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12584.153594] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [12584.157462] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c21b4154 [12584.161131] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [12584.165663] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [12584.171157] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [12584.175467] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [12584.180245] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12584.185667] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [12584.193966] Call Trace: [12584.195994] [12584.197986] ? show_regs.cold.9+0x22/0x2f [12584.199929] ? watchdog_timer_fn.cold.6+0x62/0x10a [12584.202271] ? __hrtimer_run_queues+0x1c2/0x4d0 [12584.204702] ? watchdog+0x40/0x40 [12584.206150] ? hrtimer_interrupt+0x128/0x2c0 [12584.212810] ? smp_apic_timer_interrupt+0xa6/0x270 [12584.214764] ? apic_timer_interrupt+0xf/0x20 [12584.217450] [12584.218263] ? native_safe_halt+0x2e/0x50 [12584.220090] kvm_wait+0xcd/0xe0 [12584.223312] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12584.227444] do_raw_spin_lock+0x177/0x190 [12584.231027] _raw_spin_lock+0x21/0x30 [12584.234738] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12584.239811] ? do_raw_spin_unlock+0x75/0x190 [12584.243133] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12584.248825] ? do_wait_intr+0xf0/0xf0 [12584.251950] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12584.256065] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [12584.260656] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12584.263291] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12584.267837] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12584.271084] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [12584.277251] mdc_intent_lock+0x353/0x7e0 [mdc] [12584.279519] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12584.281970] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12584.285732] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12584.288728] lmv_intent_open+0x338/0x1150 [lmv] [12584.293947] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12584.300807] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12584.305715] lmv_intent_lock+0x1dd/0x4c0 [lmv] [12584.310146] ll_intent_lock+0xc5/0xaa0 [lustre] [12584.313755] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12584.316167] ll_lookup_it+0xe0e/0x2270 [lustre] [12584.319061] ? do_raw_spin_unlock+0x75/0x190 [12584.321006] ? _raw_spin_unlock+0x12/0x30 [12584.325596] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [12584.332437] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [12584.334743] ll_atomic_open+0x36a/0xd70 [lustre] [12584.338819] lookup_open+0x356/0xb40 [12584.340977] do_last+0x313/0xfc0 [12584.342678] ? nd_jump_root+0xe5/0x160 [12584.345688] ? path_init+0x437/0x520 [12584.347469] path_openat+0xf7/0x500 [12584.349457] ? do_raw_spin_unlock+0x75/0x190 [12584.351836] do_filp_open+0x99/0x140 [12584.353953] ? getname_flags+0x6e/0x330 [12584.356779] ? __check_object_size+0xff/0x256 [12584.358716] ? do_raw_spin_unlock+0x75/0x190 [12584.360469] ? _raw_spin_unlock+0x12/0x30 [12584.362805] do_sys_openat2+0x2b4/0x410 [12584.364475] do_sys_open+0x73/0xa0 [12584.365835] __x64_sys_openat+0x24/0x30 [12584.369092] do_syscall_64+0xc1/0x440 [12584.373895] entry_SYSCALL_64_after_hwframe+0x49/0xae [12584.377722] RIP: 0033:0x7ff54af45332 [12584.380274] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [12584.395810] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [12584.403272] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [12584.408812] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [12584.413947] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [12584.418910] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [12584.423859] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [12592.058011] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [12592.062504] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12592.091812] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12592.098550] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12592.103022] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12592.105783] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12592.117868] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12592.123155] RAX: 0000000000000798 RBX: ffff96848811cae0 RCX: 0000000000000001 [12592.127026] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [12592.134410] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [12592.140953] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [12592.147400] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [12592.152307] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [12592.154972] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12592.163981] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [12592.168833] Call Trace: [12592.169759] [12592.170862] ? show_regs.cold.9+0x22/0x2f [12592.173429] ? watchdog_timer_fn.cold.6+0x62/0x10a [12592.176973] ? __hrtimer_run_queues+0x1c2/0x4d0 [12592.181184] ? watchdog+0x40/0x40 [12592.182833] ? hrtimer_interrupt+0x128/0x2c0 [12592.185344] ? smp_apic_timer_interrupt+0xa6/0x270 [12592.187791] ? apic_timer_interrupt+0xf/0x20 [12592.192010] [12592.194559] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12592.200770] do_raw_spin_lock+0x177/0x190 [12592.204679] _raw_spin_lock+0x21/0x30 [12592.209026] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12592.216309] ? do_raw_spin_unlock+0x75/0x190 [12592.220841] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12592.227345] ? do_wait_intr+0xf0/0xf0 [12592.228997] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12592.230965] mdc_close+0x53b/0xd70 [mdc] [12592.233070] lmv_close+0x18d/0x490 [lmv] [12592.234599] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [12592.236933] ll_close_inode_openhandle+0x493/0xea0 [lustre] [12592.239786] ll_md_real_close+0xa5/0x2a0 [lustre] [12592.241642] ll_md_close+0x274/0x800 [lustre] [12592.243358] ll_file_release+0x128/0xc50 [lustre] [12592.245496] __fput+0x10d/0x440 [12592.246535] ____fput+0x12/0x20 [12592.249039] task_work_run+0xc6/0x110 [12592.252676] exit_to_usermode_loop+0x1dd/0x1f0 [12592.255271] do_syscall_64+0x3d6/0x440 [12592.257668] entry_SYSCALL_64_after_hwframe+0x49/0xae [12592.261611] RIP: 0033:0x7ff54af45cd8 [12592.262647] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [12592.269949] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [12592.273294] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [12592.275605] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [12592.278389] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [12592.281577] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [12592.284357] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12604.059014] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [12604.061612] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12604.076580] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12604.079871] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12604.083156] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12604.085323] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12604.087226] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12604.093795] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12604.096470] RAX: 0000000000001a74 RBX: ffff96848811cae0 RCX: 0000000000000001 [12604.100448] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [12604.102134] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [12604.105293] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [12604.110057] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [12604.112666] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [12604.121556] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12604.126672] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [12604.135781] Call Trace: [12604.136787] [12604.137605] ? show_regs.cold.9+0x22/0x2f [12604.147859] ? watchdog_timer_fn.cold.6+0x62/0x10a [12604.155345] ? __hrtimer_run_queues+0x1c2/0x4d0 [12604.156777] ? watchdog+0x40/0x40 [12604.157757] ? hrtimer_interrupt+0x128/0x2c0 [12604.159135] ? smp_apic_timer_interrupt+0xa6/0x270 [12604.164708] ? apic_timer_interrupt+0xf/0x20 [12604.166703] [12604.167854] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12604.170854] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12604.177725] do_raw_spin_lock+0x177/0x190 [12604.181770] _raw_spin_lock+0x21/0x30 [12604.184526] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12604.187516] process_one_work+0x2c8/0x700 [12604.189659] worker_thread+0x296/0x6e0 [12604.191565] ? rescuer_thread+0x570/0x570 [12604.194090] kthread+0x1d1/0x200 [12604.195194] ? set_kthread_struct+0x70/0x70 [12604.196672] ret_from_fork+0x1f/0x30 [12608.055119] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [12608.057375] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12608.110611] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12608.122656] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12608.128889] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [12608.133166] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [12608.149244] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12608.151791] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [12608.169113] RDX: 0000000000005cf8 RSI: 0000000000000003 RDI: ffff96848811cae0 [12608.173492] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [12608.178944] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [12608.184532] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12608.188632] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [12608.193288] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12608.206178] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [12608.215575] Call Trace: [12608.217698] [12608.219663] ? show_regs.cold.9+0x22/0x2f [12608.222742] ? watchdog_timer_fn.cold.6+0x62/0x10a [12608.227876] ? __hrtimer_run_queues+0x1c2/0x4d0 [12608.231950] ? watchdog+0x40/0x40 [12608.235656] ? hrtimer_interrupt+0x128/0x2c0 [12608.239460] ? smp_apic_timer_interrupt+0xa6/0x270 [12608.244277] ? apic_timer_interrupt+0xf/0x20 [12608.247934] [12608.251213] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [12608.256735] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12608.257935] do_raw_spin_lock+0x177/0x190 [12608.258831] _raw_spin_lock+0x21/0x30 [12608.261782] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12608.268523] obd_mod_free+0x46/0x140 [ptlrpc] [12608.270795] mdc_commit_open+0x91/0xb0 [mdc] [12608.275460] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12608.281499] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12608.287936] after_reply+0xe2c/0x1780 [ptlrpc] [12608.293795] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12608.299795] ptlrpcd+0xa82/0xe40 [ptlrpc] [12608.304554] ? do_wait_intr+0xf0/0xf0 [12608.306558] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12608.311219] kthread+0x1d1/0x200 [12608.313625] ? set_kthread_struct+0x70/0x70 [12608.317871] ret_from_fork+0x1f/0x30 [12612.060972] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [12612.069878] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12612.119104] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12612.132217] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12612.137612] RIP: 0010:__pv_queued_spin_lock_slowpath+0x328/0x6d0 [12612.144506] Code: d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 05 18 d4 dc 02 01 e0 48 83 05 06 d4 dc 02 01 48 83 05 be d5 dc 02 01 8b 45 08 85 [12612.156948] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12612.161763] RAX: 000000000000542e RBX: ffff96848811cae0 RCX: 0000000000000001 [12612.166412] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [12612.168844] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [12612.177743] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [12612.183801] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [12612.196010] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [12612.199622] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12612.208765] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [12612.214521] Call Trace: [12612.216315] [12612.217057] ? show_regs.cold.9+0x22/0x2f [12612.219211] ? watchdog_timer_fn.cold.6+0x62/0x10a [12612.221601] ? __hrtimer_run_queues+0x1c2/0x4d0 [12612.224213] ? watchdog+0x40/0x40 [12612.226882] ? hrtimer_interrupt+0x128/0x2c0 [12612.228529] ? smp_apic_timer_interrupt+0xa6/0x270 [12612.233377] ? apic_timer_interrupt+0xf/0x20 [12612.236436] [12612.237162] ? __pv_queued_spin_lock_slowpath+0x328/0x6d0 [12612.238533] do_raw_spin_lock+0x177/0x190 [12612.240885] _raw_spin_lock+0x21/0x30 [12612.243697] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12612.248180] ? do_raw_spin_unlock+0x75/0x190 [12612.250241] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12612.254567] ? do_wait_intr+0xf0/0xf0 [12612.259135] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12612.265476] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [12612.270112] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12612.275776] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12612.277453] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12612.278982] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [12612.284870] mdc_intent_lock+0x353/0x7e0 [mdc] [12612.290595] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12612.296071] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12612.298822] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12612.300175] lmv_intent_open+0x338/0x1150 [lmv] [12612.305981] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12612.312901] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12612.316496] lmv_intent_lock+0x1dd/0x4c0 [lmv] [12612.318632] ll_intent_lock+0xc5/0xaa0 [lustre] [12612.320982] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12612.324069] ll_lookup_it+0xe0e/0x2270 [lustre] [12612.326783] ? do_raw_spin_unlock+0x75/0x190 [12612.328552] ? _raw_spin_unlock+0x12/0x30 [12612.330526] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [12612.333052] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [12612.336567] ll_atomic_open+0x36a/0xd70 [lustre] [12612.339869] lookup_open+0x356/0xb40 [12612.342341] do_last+0x313/0xfc0 [12612.343474] ? nd_jump_root+0xe5/0x160 [12612.345867] ? path_init+0x437/0x520 [12612.347434] path_openat+0xf7/0x500 [12612.353498] ? do_raw_spin_unlock+0x75/0x190 [12612.357029] do_filp_open+0x99/0x140 [12612.359693] ? getname_flags+0x6e/0x330 [12612.364439] ? __check_object_size+0xff/0x256 [12612.372062] ? do_raw_spin_unlock+0x75/0x190 [12612.374749] ? _raw_spin_unlock+0x12/0x30 [12612.384517] do_sys_openat2+0x2b4/0x410 [12612.386174] do_sys_open+0x73/0xa0 [12612.391868] __x64_sys_openat+0x24/0x30 [12612.401552] do_syscall_64+0xc1/0x440 [12612.402638] entry_SYSCALL_64_after_hwframe+0x49/0xae [12612.408873] RIP: 0033:0x7ff54af45332 [12612.410155] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [12612.429466] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [12612.442192] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [12612.448728] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [12612.454848] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [12612.461624] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [12612.468534] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [12612.575285] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10653s! [12612.581722] Showing busy workqueues and worker pools: [12612.587792] workqueue events: flags=0x0 [12612.593471] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12612.597469] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [12612.613938] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [12612.622490] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [12612.633518] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12612.635973] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [12612.643107] workqueue events_long: flags=0x0 [12612.646104] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12612.651540] pending: lu_objects_flush [obdclass] [12612.655328] workqueue events_unbound: flags=0x2 [12612.659488] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [12612.669148] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [12612.674468] workqueue events_power_efficient: flags=0x80 [12612.677201] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12612.681187] pending: neigh_periodic_work [12612.683344] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12612.689828] pending: do_cache_clean, check_lifetime [12612.697721] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12612.707430] pending: neigh_periodic_work [12612.710576] workqueue writeback: flags=0x4a [12612.712626] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [12612.717813] pending: wb_workfn, wb_workfn [12612.721565] workqueue xprtiod: flags=0xa [12612.723518] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [12612.733886] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [12612.743476] workqueue ipv6_addrconf: flags=0x40008 [12612.749226] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12612.751673] pending: addrconf_verify_work [12612.766787] workqueue knbd0-recv: flags=0x1a [12612.768733] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12612.775283] in-flight: 657:recv_work [12612.777512] workqueue ptlrpc_pinger: flags=0x2 [12612.779506] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12612.784483] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [12612.788088] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10653s workers=4 idle: 20831 12642 28859 [12612.792912] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12603s workers=2 idle: 659 [12620.058037] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [12620.063434] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12620.120518] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12620.136609] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12620.146692] RIP: 0010:native_safe_halt+0x2e/0x50 [12620.153209] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12620.171159] RSP: 0018:ffffaf9fc1543ae0 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [12620.177704] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [12620.187330] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c20b4154 [12620.192065] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [12620.196095] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [12620.206853] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [12620.213086] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [12620.218545] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12620.221944] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [12620.231804] Call Trace: [12620.235664] [12620.238578] ? show_regs.cold.9+0x22/0x2f [12620.242483] ? watchdog_timer_fn.cold.6+0x62/0x10a [12620.246650] ? __hrtimer_run_queues+0x1c2/0x4d0 [12620.248360] ? watchdog+0x40/0x40 [12620.254093] ? hrtimer_interrupt+0x128/0x2c0 [12620.257768] ? smp_apic_timer_interrupt+0xa6/0x270 [12620.260767] ? apic_timer_interrupt+0xf/0x20 [12620.263645] [12620.264786] ? native_safe_halt+0x2e/0x50 [12620.271552] kvm_wait+0xcd/0xe0 [12620.274178] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12620.278337] do_raw_spin_lock+0x177/0x190 [12620.281890] _raw_spin_lock+0x21/0x30 [12620.283137] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12620.288531] ? do_raw_spin_unlock+0x75/0x190 [12620.291390] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12620.294922] ? do_wait_intr+0xf0/0xf0 [12620.306614] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12620.317112] mdc_close+0x53b/0xd70 [mdc] [12620.322597] lmv_close+0x18d/0x490 [lmv] [12620.327962] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [12620.333464] ll_close_inode_openhandle+0x493/0xea0 [lustre] [12620.335717] ll_md_real_close+0xa5/0x2a0 [lustre] [12620.338295] ll_md_close+0x274/0x800 [lustre] [12620.340564] ll_file_release+0x128/0xc50 [lustre] [12620.342995] __fput+0x10d/0x440 [12620.344554] ____fput+0x12/0x20 [12620.348439] task_work_run+0xc6/0x110 [12620.349670] exit_to_usermode_loop+0x1dd/0x1f0 [12620.350897] do_syscall_64+0x3d6/0x440 [12620.353766] entry_SYSCALL_64_after_hwframe+0x49/0xae [12620.356787] RIP: 0033:0x7ff54af45cd8 [12620.359341] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [12620.382250] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [12620.391029] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [12620.401655] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [12620.406508] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [12620.410937] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [12620.415696] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12632.059010] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [12632.061691] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12632.096323] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12632.109918] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12632.118872] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12632.131370] RIP: 0010:__pv_queued_spin_lock_slowpath+0x328/0x6d0 [12632.138343] Code: d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 05 18 d4 dc 02 01 e0 48 83 05 06 d4 dc 02 01 48 83 05 be d5 dc 02 01 8b 45 08 85 [12632.152234] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12632.155807] RAX: 00000000000023e3 RBX: ffff96848811cae0 RCX: 0000000000000001 [12632.163080] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [12632.169079] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [12632.178086] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [12632.186973] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [12632.195430] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [12632.205150] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12632.210359] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [12632.221620] Call Trace: [12632.222877] [12632.224185] ? show_regs.cold.9+0x22/0x2f [12632.234328] ? watchdog_timer_fn.cold.6+0x62/0x10a [12632.238519] ? __hrtimer_run_queues+0x1c2/0x4d0 [12632.241559] ? watchdog+0x40/0x40 [12632.245087] ? hrtimer_interrupt+0x128/0x2c0 [12632.251765] ? smp_apic_timer_interrupt+0xa6/0x270 [12632.259240] ? apic_timer_interrupt+0xf/0x20 [12632.265195] [12632.265784] ? __pv_queued_spin_lock_slowpath+0x328/0x6d0 [12632.269177] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12632.272069] do_raw_spin_lock+0x177/0x190 [12632.275188] _raw_spin_lock+0x21/0x30 [12632.279459] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12632.284542] process_one_work+0x2c8/0x700 [12632.290096] worker_thread+0x296/0x6e0 [12632.292713] ? rescuer_thread+0x570/0x570 [12632.296863] kthread+0x1d1/0x200 [12632.304218] ? set_kthread_struct+0x70/0x70 [12632.307583] ret_from_fork+0x1f/0x30 [12634.133207] rcu: INFO: rcu_sched self-detected stall on CPU [12634.139141] rcu: 0-...!: (5675287 ticks this GP) idle=98a/1/0x4000000000000002 softirq=125154/125154 fqs=959 [12634.154190] (t=10680293 jiffies g=253585 q=6297) [12634.165584] rcu: rcu_sched kthread starved for 10675253 jiffies! g253585 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=3 [12634.173215] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [12634.183922] rcu: RCU grace-period kthread stack dump: [12634.188097] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [12634.194025] Call Trace: [12634.194676] __schedule+0x351/0xcb0 [12634.210511] schedule+0xc0/0x180 [12634.211700] schedule_timeout+0xb4/0x190 [12634.227574] ? do_raw_spin_unlock+0x75/0x190 [12634.228929] ? __next_timer_interrupt+0x160/0x160 [12634.230299] ? prepare_to_swait_event+0xf3/0x140 [12634.235257] rcu_gp_kthread+0x8dd/0x15b0 [12634.237237] ? force_qs_rnp+0x2c0/0x2c0 [12634.239205] kthread+0x1d1/0x200 [12634.243219] ? set_kthread_struct+0x70/0x70 [12634.246006] ret_from_fork+0x1f/0x30 [12634.249393] rcu: Stack dump where RCU GP kthread last ran: [12634.251888] Sending NMI from CPU 0 to CPUs 3: [12634.264381] NMI backtrace for cpu 3 [12634.264389] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12634.264391] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12634.264393] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12634.264395] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12634.264397] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 [12634.264400] RAX: 0000000000007e9e RBX: ffff96848811cae0 RCX: 0000000000000001 [12634.264401] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [12634.264403] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [12634.264404] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [12634.264406] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [12634.264407] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [12634.264409] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12634.264410] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [12634.264411] Call Trace: [12634.264412] [12634.264413] ? show_regs.cold.9+0x22/0x2f [12634.264414] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [12634.264415] ? nmi_cpu_backtrace_handler+0x15/0x40 [12634.264417] ? nmi_handle+0xbd/0x210 [12634.264418] ? default_do_nmi+0x4b/0x1b0 [12634.264419] ? do_nmi+0x280/0x380 [12634.264420] ? end_repeat_nmi+0x16/0x1f [12634.264421] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12634.264422] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12634.264424] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12634.264425] [12634.264426] do_raw_spin_lock+0x177/0x190 [12634.264427] _raw_spin_lock+0x21/0x30 [12634.264428] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12634.264429] ? do_raw_spin_unlock+0x75/0x190 [12634.264430] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12634.264433] ? do_wait_intr+0xf0/0xf0 [12634.264434] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12634.264435] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [12634.264436] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12634.264437] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12634.264438] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12634.264439] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [12634.264440] mdc_intent_lock+0x353/0x7e0 [mdc] [12634.264442] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12634.264443] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12634.264444] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12634.264445] lmv_intent_open+0x338/0x1150 [lmv] [12634.264446] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12634.264447] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12634.264449] lmv_intent_lock+0x1dd/0x4c0 [lmv] [12634.264450] ll_intent_lock+0xc5/0xaa0 [lustre] [12634.264451] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12634.264453] ll_lookup_it+0xe0e/0x2270 [lustre] [12634.264454] ? do_raw_spin_unlock+0x75/0x190 [12634.264456] ? _raw_spin_unlock+0x12/0x30 [12634.264457] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [12634.264458] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [12634.264459] ll_atomic_open+0x36a/0xd70 [lustre] [12634.264460] lookup_open+0x356/0xb40 [12634.264461] do_last+0x313/0xfc0 [12634.264462] ? nd_jump_root+0xe5/0x160 [12634.264463] ? path_init+0x437/0x520 [12634.264464] path_openat+0xf7/0x500 [12634.264465] ? do_raw_spin_unlock+0x75/0x190 [12634.264466] do_filp_open+0x99/0x140 [12634.264467] ? getname_flags+0x6e/0x330 [12634.264468] ? __check_object_size+0xff/0x256 [12634.264469] ? do_raw_spin_unlock+0x75/0x190 [12634.264470] ? _raw_spin_unlock+0x12/0x30 [12634.264471] do_sys_openat2+0x2b4/0x410 [12634.264472] do_sys_open+0x73/0xa0 [12634.264473] __x64_sys_openat+0x24/0x30 [12634.264475] do_syscall_64+0xc1/0x440 [12634.264476] entry_SYSCALL_64_after_hwframe+0x49/0xae [12634.264477] RIP: 0033:0x7ff54af45332 [12634.264478] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [12634.264481] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [12634.264484] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [12634.264486] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [12634.264964] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [12634.264967] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [12634.264968] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [12634.268219] NMI backtrace for cpu 0 [12634.727200] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12634.732632] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12634.737443] Call Trace: [12634.738617] [12634.739895] ? dump_stack+0xbb/0x10e [12634.741646] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [12634.748499] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [12634.750242] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [12634.752349] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [12634.755127] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [12634.757507] ? rcu_dump_cpu_stacks+0x11b/0x171 [12634.762383] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [12634.764679] ? update_process_times+0x6d/0xc0 [12634.766020] ? tick_sched_handle.isra.18+0x27/0x90 [12634.768634] ? tick_sched_timer+0x53/0xb0 [12634.770380] ? __hrtimer_run_queues+0x1c2/0x4d0 [12634.772431] ? tick_sched_handle.isra.18+0x90/0x90 [12634.776112] ? hrtimer_interrupt+0x128/0x2c0 [12634.777198] ? smp_apic_timer_interrupt+0xa6/0x270 [12634.781048] ? apic_timer_interrupt+0xf/0x20 [12634.783280] [12634.784075] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [12634.785850] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12634.788099] ? do_raw_spin_lock+0x177/0x190 [12634.790228] ? _raw_spin_lock+0x21/0x30 [12634.792536] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12634.798423] ? obd_mod_free+0x46/0x140 [ptlrpc] [12634.801512] ? mdc_commit_open+0x91/0xb0 [mdc] [12634.803353] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12634.806083] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12634.819050] ? after_reply+0xe2c/0x1780 [ptlrpc] [12634.824847] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12634.833354] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [12634.843168] ? do_wait_intr+0xf0/0xf0 [12634.847634] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12634.859214] ? kthread+0x1d1/0x200 [12634.863487] ? set_kthread_struct+0x70/0x70 [12634.867203] ? ret_from_fork+0x1f/0x30 [12643.296282] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10684s! [12643.300078] Showing busy workqueues and worker pools: [12643.302311] workqueue events: flags=0x0 [12643.305514] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12643.309436] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [12643.316455] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [12643.322291] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [12643.329441] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12643.334406] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [12643.339345] workqueue events_long: flags=0x0 [12643.345953] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12643.349299] pending: lu_objects_flush [obdclass] [12643.353313] workqueue events_unbound: flags=0x2 [12643.357303] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [12643.361295] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [12643.367707] workqueue events_power_efficient: flags=0x80 [12643.370968] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12643.375349] pending: neigh_periodic_work [12643.377215] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12643.381501] pending: do_cache_clean, check_lifetime [12643.385433] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12643.391120] pending: neigh_periodic_work [12643.394127] workqueue writeback: flags=0x4a [12643.400943] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [12643.407152] pending: wb_workfn, wb_workfn [12643.409446] workqueue xprtiod: flags=0xa [12643.412225] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [12643.416243] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [12643.420981] workqueue ipv6_addrconf: flags=0x40008 [12643.422729] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12643.426400] pending: addrconf_verify_work [12643.428422] workqueue knbd0-recv: flags=0x1a [12643.430134] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12643.433219] in-flight: 657:recv_work [12643.435240] workqueue ptlrpc_pinger: flags=0x2 [12643.439237] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12643.443815] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [12643.447725] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10684s workers=4 idle: 20831 12642 28859 [12643.452097] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12633s workers=2 idle: 659 [12648.060331] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [12648.076123] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12648.129921] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12648.135444] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12648.140254] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12648.147404] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12648.170326] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12648.180040] RAX: 0000000000005c04 RBX: ffff96848811cae0 RCX: 0000000000000001 [12648.184220] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [12648.188157] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [12648.193230] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [12648.208682] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [12648.211074] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [12648.220833] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12648.224384] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [12648.226716] Call Trace: [12648.229111] [12648.231112] ? show_regs.cold.9+0x22/0x2f [12648.234393] ? watchdog_timer_fn.cold.6+0x62/0x10a [12648.243203] ? __hrtimer_run_queues+0x1c2/0x4d0 [12648.248378] ? watchdog+0x40/0x40 [12648.252250] ? hrtimer_interrupt+0x128/0x2c0 [12648.254170] ? smp_apic_timer_interrupt+0xa6/0x270 [12648.259107] ? apic_timer_interrupt+0xf/0x20 [12648.260436] [12648.263541] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12648.274905] do_raw_spin_lock+0x177/0x190 [12648.276912] _raw_spin_lock+0x21/0x30 [12648.279038] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12648.285937] ? do_raw_spin_unlock+0x75/0x190 [12648.288392] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12648.292860] ? do_wait_intr+0xf0/0xf0 [12648.297971] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12648.303857] mdc_close+0x53b/0xd70 [mdc] [12648.309103] lmv_close+0x18d/0x490 [lmv] [12648.312309] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [12648.319202] ll_close_inode_openhandle+0x493/0xea0 [lustre] [12648.329216] ll_md_real_close+0xa5/0x2a0 [lustre] [12648.336243] ll_md_close+0x274/0x800 [lustre] [12648.342234] ll_file_release+0x128/0xc50 [lustre] [12648.346092] __fput+0x10d/0x440 [12648.347336] ____fput+0x12/0x20 [12648.349420] task_work_run+0xc6/0x110 [12648.351307] exit_to_usermode_loop+0x1dd/0x1f0 [12648.353884] do_syscall_64+0x3d6/0x440 [12648.360436] entry_SYSCALL_64_after_hwframe+0x49/0xae [12648.362248] RIP: 0033:0x7ff54af45cd8 [12648.365389] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [12648.379945] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [12648.388920] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [12648.397377] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [12648.401882] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [12648.408434] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [12648.416386] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12660.060022] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [12660.060990] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [12660.079585] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12660.079690] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12660.079701] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12660.079712] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12660.079731] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12660.079736] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12660.079741] RAX: 0000000000004715 RBX: ffff96848811cae0 RCX: 0000000000000001 [12660.079743] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [12660.079746] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [12660.079748] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [12660.079751] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [12660.079754] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [12660.079758] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12660.079760] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [12660.079769] Call Trace: [12660.079775] [12660.079782] ? show_regs.cold.9+0x22/0x2f [12660.079792] ? watchdog_timer_fn.cold.6+0x62/0x10a [12660.079800] ? __hrtimer_run_queues+0x1c2/0x4d0 [12660.079804] ? watchdog+0x40/0x40 [12660.079809] ? hrtimer_interrupt+0x128/0x2c0 [12660.079813] ? smp_apic_timer_interrupt+0xa6/0x270 [12660.079824] ? apic_timer_interrupt+0xf/0x20 [12660.079828] [12660.079832] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12660.079839] do_raw_spin_lock+0x177/0x190 [12660.079844] _raw_spin_lock+0x21/0x30 [12660.079849] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12660.086492] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12660.086971] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12660.086980] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12660.086986] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12660.087310] RIP: 0010:native_safe_halt+0x2e/0x50 [12660.087321] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12660.087325] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12660.087330] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [12660.087332] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c2134154 [12660.087335] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [12660.087337] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [12660.087339] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [12660.087343] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [12660.087345] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12660.087348] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [12660.087357] Call Trace: [12660.087363] [12660.087370] ? show_regs.cold.9+0x22/0x2f [12660.087380] ? watchdog_timer_fn.cold.6+0x62/0x10a [12660.087387] ? __hrtimer_run_queues+0x1c2/0x4d0 [12660.087391] ? watchdog+0x40/0x40 [12660.087395] ? hrtimer_interrupt+0x128/0x2c0 [12660.087398] ? smp_apic_timer_interrupt+0xa6/0x270 [12660.087405] ? apic_timer_interrupt+0xf/0x20 [12660.087410] [12660.087412] ? native_safe_halt+0x2e/0x50 [12660.087416] kvm_wait+0xcd/0xe0 [12660.087425] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12660.087433] do_raw_spin_lock+0x177/0x190 [12660.087438] _raw_spin_lock+0x21/0x30 [12660.087443] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12660.087674] process_one_work+0x2c8/0x700 [12660.087683] worker_thread+0x296/0x6e0 [12660.087688] ? rescuer_thread+0x570/0x570 [12660.087691] kthread+0x1d1/0x200 [12660.087695] ? set_kthread_struct+0x70/0x70 [12660.087700] ret_from_fork+0x1f/0x30 [12660.694006] ? do_raw_spin_unlock+0x75/0x190 [12660.701638] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12660.709218] ? do_wait_intr+0xf0/0xf0 [12660.710528] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12660.718923] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [12660.734806] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12660.747370] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12660.752108] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12660.762656] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [12660.771277] mdc_intent_lock+0x353/0x7e0 [mdc] [12660.772972] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12660.789411] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12660.800734] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12660.813653] lmv_intent_open+0x338/0x1150 [lmv] [12660.822421] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12660.833015] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12660.849630] lmv_intent_lock+0x1dd/0x4c0 [lmv] [12660.858315] ll_intent_lock+0xc5/0xaa0 [lustre] [12660.863939] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12660.870844] ll_lookup_it+0xe0e/0x2270 [lustre] [12660.890418] ? do_raw_spin_unlock+0x75/0x190 [12660.892027] ? _raw_spin_unlock+0x12/0x30 [12660.903654] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [12660.905477] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [12660.919072] ll_atomic_open+0x36a/0xd70 [lustre] [12660.931807] lookup_open+0x356/0xb40 [12660.940323] do_last+0x313/0xfc0 [12660.943384] ? nd_jump_root+0xe5/0x160 [12660.950870] ? path_init+0x437/0x520 [12660.961706] path_openat+0xf7/0x500 [12660.965393] ? do_raw_spin_unlock+0x75/0x190 [12660.972092] do_filp_open+0x99/0x140 [12660.976821] ? getname_flags+0x6e/0x330 [12660.978931] ? __check_object_size+0xff/0x256 [12660.988469] ? do_raw_spin_unlock+0x75/0x190 [12660.995457] ? _raw_spin_unlock+0x12/0x30 [12661.001222] do_sys_openat2+0x2b4/0x410 [12661.003920] do_sys_open+0x73/0xa0 [12661.006914] __x64_sys_openat+0x24/0x30 [12661.013173] do_syscall_64+0xc1/0x440 [12661.014880] entry_SYSCALL_64_after_hwframe+0x49/0xae [12661.017395] RIP: 0033:0x7ff54af45332 [12661.019125] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [12661.026025] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [12661.028152] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [12661.030572] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [12661.032872] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [12661.035814] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [12661.038826] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [12668.061613] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [12668.071582] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12668.141064] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12668.153679] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12668.163989] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [12668.167004] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [12668.177851] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12668.193947] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [12668.200160] RDX: 0000000000000bee RSI: 0000000000000003 RDI: ffff96848811cae0 [12668.218627] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [12668.224688] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [12668.232792] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12668.239853] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [12668.246577] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12668.250253] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [12668.260728] Call Trace: [12668.262340] [12668.265310] ? show_regs.cold.9+0x22/0x2f [12668.270217] ? watchdog_timer_fn.cold.6+0x62/0x10a [12668.276312] ? __hrtimer_run_queues+0x1c2/0x4d0 [12668.279270] ? watchdog+0x40/0x40 [12668.284627] ? hrtimer_interrupt+0x128/0x2c0 [12668.293987] ? smp_apic_timer_interrupt+0xa6/0x270 [12668.300913] ? apic_timer_interrupt+0xf/0x20 [12668.307319] [12668.309283] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [12668.315074] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12668.319218] do_raw_spin_lock+0x177/0x190 [12668.321440] _raw_spin_lock+0x21/0x30 [12668.323675] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12668.327696] obd_mod_free+0x46/0x140 [ptlrpc] [12668.331755] mdc_commit_open+0x91/0xb0 [mdc] [12668.334230] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12668.337795] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12668.340945] after_reply+0xe2c/0x1780 [ptlrpc] [12668.344070] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12668.346911] ptlrpcd+0xa82/0xe40 [ptlrpc] [12668.350192] ? do_wait_intr+0xf0/0xf0 [12668.352706] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12668.364034] kthread+0x1d1/0x200 [12668.366194] ? set_kthread_struct+0x70/0x70 [12668.375934] ret_from_fork+0x1f/0x30 [12673.507453] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10714s! [12673.522962] Showing busy workqueues and worker pools: [12673.529091] workqueue events: flags=0x0 [12673.532984] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12673.540262] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [12673.553086] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [12673.559732] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [12673.577573] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12673.588187] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [12673.597156] workqueue events_long: flags=0x0 [12673.608181] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12673.614890] pending: lu_objects_flush [obdclass] [12673.620053] workqueue events_unbound: flags=0x2 [12673.624177] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [12673.639678] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [12673.650929] workqueue events_power_efficient: flags=0x80 [12673.657214] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12673.661554] pending: neigh_periodic_work [12673.664280] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12673.673213] pending: do_cache_clean, check_lifetime [12673.683477] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12673.691945] pending: neigh_periodic_work [12673.695825] workqueue writeback: flags=0x4a [12673.701071] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [12673.703413] pending: wb_workfn, wb_workfn [12673.715231] workqueue xprtiod: flags=0xa [12673.728373] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [12673.744997] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [12673.756819] workqueue ipv6_addrconf: flags=0x40008 [12673.768669] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12673.777838] pending: addrconf_verify_work [12673.783403] workqueue knbd0-recv: flags=0x1a [12673.787793] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12673.796973] in-flight: 657:recv_work [12673.798305] workqueue ptlrpc_pinger: flags=0x2 [12673.804769] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12673.810112] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [12673.817735] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10714s workers=4 idle: 20831 12642 28859 [12673.824443] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12664s workers=2 idle: 659 [12676.059598] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [12676.071051] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12676.137799] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12676.156152] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12676.176178] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12676.182783] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12676.201182] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12676.209442] RAX: 0000000000005686 RBX: ffff96848811cae0 RCX: 0000000000000001 [12676.217038] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [12676.225566] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [12676.233417] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [12676.243914] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [12676.246292] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [12676.263355] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12676.270791] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [12676.278063] Call Trace: [12676.279402] [12676.284257] ? show_regs.cold.9+0x22/0x2f [12676.289123] ? watchdog_timer_fn.cold.6+0x62/0x10a [12676.295695] ? __hrtimer_run_queues+0x1c2/0x4d0 [12676.305479] ? watchdog+0x40/0x40 [12676.307449] ? hrtimer_interrupt+0x128/0x2c0 [12676.313394] ? smp_apic_timer_interrupt+0xa6/0x270 [12676.321073] ? apic_timer_interrupt+0xf/0x20 [12676.326925] [12676.328994] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12676.335829] do_raw_spin_lock+0x177/0x190 [12676.342535] _raw_spin_lock+0x21/0x30 [12676.344835] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12676.352914] ? do_raw_spin_unlock+0x75/0x190 [12676.360641] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12676.365326] ? do_wait_intr+0xf0/0xf0 [12676.368497] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12676.374799] mdc_close+0x53b/0xd70 [mdc] [12676.376194] lmv_close+0x18d/0x490 [lmv] [12676.381137] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [12676.384050] ll_close_inode_openhandle+0x493/0xea0 [lustre] [12676.388809] ll_md_real_close+0xa5/0x2a0 [lustre] [12676.392012] ll_md_close+0x274/0x800 [lustre] [12676.397139] ll_file_release+0x128/0xc50 [lustre] [12676.404184] __fput+0x10d/0x440 [12676.406636] ____fput+0x12/0x20 [12676.410686] task_work_run+0xc6/0x110 [12676.413686] exit_to_usermode_loop+0x1dd/0x1f0 [12676.415867] do_syscall_64+0x3d6/0x440 [12676.419418] entry_SYSCALL_64_after_hwframe+0x49/0xae [12676.425475] RIP: 0033:0x7ff54af45cd8 [12676.429738] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [12676.445900] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [12676.451563] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [12676.458586] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [12676.465102] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [12676.472460] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [12676.479934] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12688.059025] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [12688.061662] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [12688.064519] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12688.064596] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12688.064602] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12688.064607] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12688.064858] RIP: 0010:native_safe_halt+0x2e/0x50 [12688.064868] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12688.064872] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [12688.064875] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [12688.064877] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c2134154 [12688.064879] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [12688.064880] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [12688.064882] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [12688.064884] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [12688.064886] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12688.064888] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [12688.064895] Call Trace: [12688.064899] [12688.064904] ? show_regs.cold.9+0x22/0x2f [12688.064913] ? watchdog_timer_fn.cold.6+0x62/0x10a [12688.064917] ? __hrtimer_run_queues+0x1c2/0x4d0 [12688.064920] ? watchdog+0x40/0x40 [12688.064923] ? hrtimer_interrupt+0x128/0x2c0 [12688.064926] ? smp_apic_timer_interrupt+0xa6/0x270 [12688.064930] ? apic_timer_interrupt+0xf/0x20 [12688.064934] [12688.064936] ? native_safe_halt+0x2e/0x50 [12688.064938] kvm_wait+0xcd/0xe0 [12688.064944] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12688.064951] do_raw_spin_lock+0x177/0x190 [12688.064954] _raw_spin_lock+0x21/0x30 [12688.064957] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12688.065121] process_one_work+0x2c8/0x700 [12688.065129] worker_thread+0x296/0x6e0 [12688.065134] ? rescuer_thread+0x570/0x570 [12688.065138] kthread+0x1d1/0x200 [12688.065142] ? set_kthread_struct+0x70/0x70 [12688.065145] ret_from_fork+0x1f/0x30 [12688.308269] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12688.339466] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12688.344892] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12688.349187] RIP: 0010:native_safe_halt+0x2e/0x50 [12688.351073] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12688.364345] RSP: 0000:ffffaf9fc15234f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12688.376485] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [12688.379745] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c21b4154 [12688.390126] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [12688.401169] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [12688.409470] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [12688.414272] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [12688.419759] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12688.422457] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [12688.434820] Call Trace: [12688.436690] [12688.442521] ? show_regs.cold.9+0x22/0x2f [12688.443886] ? watchdog_timer_fn.cold.6+0x62/0x10a [12688.454037] ? __hrtimer_run_queues+0x1c2/0x4d0 [12688.456273] ? watchdog+0x40/0x40 [12688.464079] ? hrtimer_interrupt+0x128/0x2c0 [12688.466369] ? smp_apic_timer_interrupt+0xa6/0x270 [12688.473827] ? apic_timer_interrupt+0xf/0x20 [12688.493491] [12688.494282] ? native_safe_halt+0x2e/0x50 [12688.495749] kvm_wait+0xcd/0xe0 [12688.498007] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12688.503768] do_raw_spin_lock+0x177/0x190 [12688.509546] _raw_spin_lock+0x21/0x30 [12688.510806] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12688.520475] ? do_raw_spin_unlock+0x75/0x190 [12688.523163] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12688.531744] ? do_wait_intr+0xf0/0xf0 [12688.540782] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12688.545368] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [12688.562701] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12688.567966] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12688.575659] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12688.582356] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [12688.587285] mdc_intent_lock+0x353/0x7e0 [mdc] [12688.591166] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12688.599655] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12688.611826] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12688.615128] lmv_intent_open+0x338/0x1150 [lmv] [12688.618068] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12688.626263] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12688.630439] lmv_intent_lock+0x1dd/0x4c0 [lmv] [12688.638877] ll_intent_lock+0xc5/0xaa0 [lustre] [12688.640553] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12688.650090] ll_lookup_it+0xe0e/0x2270 [lustre] [12688.658542] ? do_raw_spin_unlock+0x75/0x190 [12688.660013] ? _raw_spin_unlock+0x12/0x30 [12688.667559] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [12688.672038] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [12688.681069] ll_atomic_open+0x36a/0xd70 [lustre] [12688.682847] lookup_open+0x356/0xb40 [12688.687981] do_last+0x313/0xfc0 [12688.691136] ? nd_jump_root+0xe5/0x160 [12688.694670] ? path_init+0x437/0x520 [12688.700594] path_openat+0xf7/0x500 [12688.702771] ? do_raw_spin_unlock+0x75/0x190 [12688.710145] do_filp_open+0x99/0x140 [12688.715124] ? getname_flags+0x6e/0x330 [12688.718886] ? __check_object_size+0xff/0x256 [12688.720554] ? do_raw_spin_unlock+0x75/0x190 [12688.723987] ? _raw_spin_unlock+0x12/0x30 [12688.725730] do_sys_openat2+0x2b4/0x410 [12688.727958] do_sys_open+0x73/0xa0 [12688.729511] __x64_sys_openat+0x24/0x30 [12688.731486] do_syscall_64+0xc1/0x440 [12688.733013] entry_SYSCALL_64_after_hwframe+0x49/0xae [12688.735298] RIP: 0033:0x7ff54af45332 [12688.736678] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [12688.743491] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [12688.747732] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [12688.750651] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [12688.753798] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [12688.756518] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [12688.759593] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [12700.060490] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [12700.072795] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12700.122991] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12700.150764] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12700.164329] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [12700.166106] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [12700.191275] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12700.198820] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [12700.203791] RDX: 0000000000006788 RSI: 0000000000000003 RDI: ffff96848811cae0 [12700.208954] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [12700.216795] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [12700.222903] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12700.228575] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [12700.235206] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12700.242761] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [12700.247074] Call Trace: [12700.248674] [12700.249654] ? show_regs.cold.9+0x22/0x2f [12700.253391] ? watchdog_timer_fn.cold.6+0x62/0x10a [12700.257942] ? __hrtimer_run_queues+0x1c2/0x4d0 [12700.261994] ? watchdog+0x40/0x40 [12700.264906] ? hrtimer_interrupt+0x128/0x2c0 [12700.273652] ? smp_apic_timer_interrupt+0xa6/0x270 [12700.279709] ? apic_timer_interrupt+0xf/0x20 [12700.284517] [12700.287495] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [12700.293896] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12700.308047] do_raw_spin_lock+0x177/0x190 [12700.309468] _raw_spin_lock+0x21/0x30 [12700.316837] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12700.326413] obd_mod_free+0x46/0x140 [ptlrpc] [12700.335951] mdc_commit_open+0x91/0xb0 [mdc] [12700.339699] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12700.341681] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12700.349814] after_reply+0xe2c/0x1780 [ptlrpc] [12700.355660] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12700.359748] ptlrpcd+0xa82/0xe40 [ptlrpc] [12700.362926] ? do_wait_intr+0xf0/0xf0 [12700.368857] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12700.371741] kthread+0x1d1/0x200 [12700.374515] ? set_kthread_struct+0x70/0x70 [12700.378051] ret_from_fork+0x1f/0x30 [12704.058295] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [12704.061001] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12704.085434] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12704.102480] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12704.118594] RIP: 0010:native_safe_halt+0x2e/0x50 [12704.121815] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12704.137471] RSP: 0018:ffffaf9fc1543ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12704.145275] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [12704.152177] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c20b4154 [12704.154580] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [12704.156174] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [12704.164533] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [12704.167826] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [12704.174541] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12704.179463] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [12704.188778] Call Trace: [12704.190149] [12704.191220] ? show_regs.cold.9+0x22/0x2f [12704.195020] ? watchdog_timer_fn.cold.6+0x62/0x10a [12704.199917] ? __hrtimer_run_queues+0x1c2/0x4d0 [12704.207875] ? watchdog+0x40/0x40 [12704.211594] ? hrtimer_interrupt+0x128/0x2c0 [12704.214187] ? smp_apic_timer_interrupt+0xa6/0x270 [12704.221171] ? apic_timer_interrupt+0xf/0x20 [12704.224267] [12704.229217] BUG: workqueue lockup - pool [12704.230678] ? native_safe_halt+0x2e/0x50 [12704.230698] kvm_wait+0xcd/0xe0 [12704.230707] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12704.230717] do_raw_spin_lock+0x177/0x190 [12704.230721] _raw_spin_lock+0x21/0x30 [12704.230726] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12704.239444] cpus=0-3 [12704.240522] ? do_raw_spin_unlock+0x75/0x190 [12704.240537] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12704.250973] flags=0x4 nice=0 stuck for 10745s! [12704.250992] Showing busy workqueues and worker pools: [12704.250995] workqueue events: flags=0x0 [12704.251025] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12704.251037] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [12704.252091] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [12704.252149] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [12704.252354] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12704.252360] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [12704.260657] ? do_wait_intr+0xf0/0xf0 [12704.264649] [12704.264684] workqueue events_long: flags=0x0 [12704.264691] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12704.264708] pending: lu_objects_flush [obdclass] [12704.270592] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12704.274891] [12704.274928] workqueue events_unbound: flags=0x2 [12704.274937] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [12704.274949] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [12704.274980] workqueue events_power_efficient: flags=0x80 [12704.274982] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12704.274990] pending: neigh_periodic_work [12704.274997] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12704.275007] pending: do_cache_clean, check_lifetime [12704.275032] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12704.275041] pending: neigh_periodic_work [12704.275055] workqueue writeback: flags=0x4a [12704.275057] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [12704.275063] pending: wb_workfn, wb_workfn [12704.275081] workqueue xprtiod: flags=0xa [12704.275083] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [12704.275092] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [12704.275112] workqueue ipv6_addrconf: flags=0x40008 [12704.275115] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12704.275122] pending: addrconf_verify_work [12704.275132] workqueue knbd0-recv: flags=0x1a [12704.275138] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12704.275145] in-flight: 657:recv_work [12704.275155] workqueue ptlrpc_pinger: flags=0x2 [12704.275160] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12704.275166] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [12704.275524] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10745s workers=4 idle: 20831 12642 28859 [12704.275538] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12694s workers=2 idle: 659 [12704.451187] mdc_close+0x53b/0xd70 [mdc] [12704.452271] lmv_close+0x18d/0x490 [lmv] [12704.457235] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [12704.459704] ll_close_inode_openhandle+0x493/0xea0 [lustre] [12704.461148] ll_md_real_close+0xa5/0x2a0 [lustre] [12704.463525] ll_md_close+0x274/0x800 [lustre] [12704.465563] ll_file_release+0x128/0xc50 [lustre] [12704.467928] __fput+0x10d/0x440 [12704.468857] ____fput+0x12/0x20 [12704.470143] task_work_run+0xc6/0x110 [12704.471182] exit_to_usermode_loop+0x1dd/0x1f0 [12704.472376] do_syscall_64+0x3d6/0x440 [12704.473434] entry_SYSCALL_64_after_hwframe+0x49/0xae [12704.474828] RIP: 0033:0x7ff54af45cd8 [12704.475770] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [12704.486543] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [12704.488854] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [12704.496542] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [12704.498034] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [12704.510841] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [12704.515718] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12716.059149] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [12716.065195] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [12716.068323] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12716.068402] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12716.068408] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12716.068416] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12716.068692] RIP: 0010:native_safe_halt+0x2e/0x50 [12716.068701] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12716.068704] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12716.068709] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [12716.068711] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c2134154 [12716.068713] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [12716.068715] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [12716.068718] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [12716.068721] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [12716.068723] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12716.068726] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [12716.068736] Call Trace: [12716.068742] [12716.068747] ? show_regs.cold.9+0x22/0x2f [12716.073153] ? watchdog_timer_fn.cold.6+0x62/0x10a [12716.073161] ? __hrtimer_run_queues+0x1c2/0x4d0 [12716.073166] ? watchdog+0x40/0x40 [12716.073170] ? hrtimer_interrupt+0x128/0x2c0 [12716.073175] ? smp_apic_timer_interrupt+0xa6/0x270 [12716.073183] ? apic_timer_interrupt+0xf/0x20 [12716.073189] [12716.073194] ? native_safe_halt+0x2e/0x50 [12716.073198] kvm_wait+0xcd/0xe0 [12716.073207] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12716.073217] do_raw_spin_lock+0x177/0x190 [12716.073222] _raw_spin_lock+0x21/0x30 [12716.073227] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12716.073744] process_one_work+0x2c8/0x700 [12716.073752] worker_thread+0x296/0x6e0 [12716.073756] ? rescuer_thread+0x570/0x570 [12716.073760] kthread+0x1d1/0x200 [12716.073765] ? set_kthread_struct+0x70/0x70 [12716.073769] ret_from_fork+0x1f/0x30 [12716.505767] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12716.559955] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12716.566040] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12716.570671] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12716.573636] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12716.601143] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12716.616364] RAX: 0000000000007f89 RBX: ffff96848811cae0 RCX: 0000000000000001 [12716.630395] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [12716.635835] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [12716.641766] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [12716.650830] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [12716.658645] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [12716.668893] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12716.676451] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [12716.681648] Call Trace: [12716.684974] [12716.686483] ? show_regs.cold.9+0x22/0x2f [12716.688347] ? watchdog_timer_fn.cold.6+0x62/0x10a [12716.689969] ? __hrtimer_run_queues+0x1c2/0x4d0 [12716.700010] ? watchdog+0x40/0x40 [12716.701642] ? hrtimer_interrupt+0x128/0x2c0 [12716.708092] ? smp_apic_timer_interrupt+0xa6/0x270 [12716.717385] ? apic_timer_interrupt+0xf/0x20 [12716.722132] [12716.723273] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12716.732258] do_raw_spin_lock+0x177/0x190 [12716.737700] _raw_spin_lock+0x21/0x30 [12716.740882] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12716.749019] ? do_raw_spin_unlock+0x75/0x190 [12716.758659] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12716.763913] ? do_wait_intr+0xf0/0xf0 [12716.770680] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12716.777484] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [12716.789231] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12716.804523] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12716.812678] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12716.824757] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [12716.832138] mdc_intent_lock+0x353/0x7e0 [mdc] [12716.837062] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12716.844664] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12716.859435] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12716.867898] lmv_intent_open+0x338/0x1150 [lmv] [12716.869672] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12716.883154] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12716.891991] lmv_intent_lock+0x1dd/0x4c0 [lmv] [12716.898330] ll_intent_lock+0xc5/0xaa0 [lustre] [12716.900286] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12716.904462] ll_lookup_it+0xe0e/0x2270 [lustre] [12716.906591] ? do_raw_spin_unlock+0x75/0x190 [12716.913885] ? _raw_spin_unlock+0x12/0x30 [12716.916572] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [12716.922868] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [12716.925568] ll_atomic_open+0x36a/0xd70 [lustre] [12716.928200] lookup_open+0x356/0xb40 [12716.932539] do_last+0x313/0xfc0 [12716.934890] ? nd_jump_root+0xe5/0x160 [12716.938187] ? path_init+0x437/0x520 [12716.939811] path_openat+0xf7/0x500 [12716.942678] ? do_raw_spin_unlock+0x75/0x190 [12716.944987] do_filp_open+0x99/0x140 [12716.950237] ? getname_flags+0x6e/0x330 [12716.955254] ? __check_object_size+0xff/0x256 [12716.956861] ? do_raw_spin_unlock+0x75/0x190 [12716.965619] ? _raw_spin_unlock+0x12/0x30 [12716.967091] do_sys_openat2+0x2b4/0x410 [12716.968445] do_sys_open+0x73/0xa0 [12716.974662] __x64_sys_openat+0x24/0x30 [12716.977181] do_syscall_64+0xc1/0x440 [12716.981724] entry_SYSCALL_64_after_hwframe+0x49/0xae [12716.985101] RIP: 0033:0x7ff54af45332 [12716.990813] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [12717.017031] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [12717.022852] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [12717.029331] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [12717.036381] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [12717.041772] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [12717.047097] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [12732.055029] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [12732.058223] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12732.059813] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [12732.075885] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12732.075900] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12732.075903] RIP: 0010:native_safe_halt+0x2e/0x50 [12732.075917] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12732.075921] RSP: 0000:ffffaf9fc1b97bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12732.075939] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [12732.075941] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff96848811cae0 [12732.075944] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [12732.075946] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [12732.075949] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12732.075951] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [12732.075954] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12732.075956] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [12732.075963] Call Trace: [12732.075969] [12732.075975] ? show_regs.cold.9+0x22/0x2f [12732.075985] ? watchdog_timer_fn.cold.6+0x62/0x10a [12732.075991] ? __hrtimer_run_queues+0x1c2/0x4d0 [12732.075995] ? watchdog+0x40/0x40 [12732.076059] ? hrtimer_interrupt+0x128/0x2c0 [12732.076063] ? smp_apic_timer_interrupt+0xa6/0x270 [12732.076070] ? apic_timer_interrupt+0xf/0x20 [12732.076075] [12732.076078] ? native_safe_halt+0x2e/0x50 [12732.076082] kvm_wait+0xcd/0xe0 [12732.076089] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12732.076096] do_raw_spin_lock+0x177/0x190 [12732.076101] _raw_spin_lock+0x21/0x30 [12732.076107] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12732.076385] obd_mod_free+0x46/0x140 [ptlrpc] [12732.076584] mdc_commit_open+0x91/0xb0 [mdc] [12732.076624] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12732.077084] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12732.077290] after_reply+0xe2c/0x1780 [ptlrpc] [12732.077496] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12732.077854] ptlrpcd+0xa82/0xe40 [ptlrpc] [12732.078199] ? do_wait_intr+0xf0/0xf0 [12732.078210] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12732.078430] kthread+0x1d1/0x200 [12732.078435] ? set_kthread_struct+0x70/0x70 [12732.078439] ret_from_fork+0x1f/0x30 [12732.223741] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12732.254352] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12732.267476] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12732.274425] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12732.277522] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12732.291394] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12732.301449] RAX: 0000000000006d91 RBX: ffff96848811cae0 RCX: 0000000000000001 [12732.308841] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [12732.314306] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [12732.318154] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [12732.323445] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [12732.328523] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [12732.334579] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12732.342730] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [12732.350806] Call Trace: [12732.355407] [12732.356600] ? show_regs.cold.9+0x22/0x2f [12732.358377] ? watchdog_timer_fn.cold.6+0x62/0x10a [12732.362354] ? __hrtimer_run_queues+0x1c2/0x4d0 [12732.367759] ? watchdog+0x40/0x40 [12732.369149] ? hrtimer_interrupt+0x128/0x2c0 [12732.373993] ? smp_apic_timer_interrupt+0xa6/0x270 [12732.376836] ? apic_timer_interrupt+0xf/0x20 [12732.379596] [12732.384218] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12732.387648] do_raw_spin_lock+0x177/0x190 [12732.390738] _raw_spin_lock+0x21/0x30 [12732.395248] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12732.397213] ? do_raw_spin_unlock+0x75/0x190 [12732.408966] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12732.416121] ? do_wait_intr+0xf0/0xf0 [12732.421685] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12732.429353] mdc_close+0x53b/0xd70 [mdc] [12732.433100] lmv_close+0x18d/0x490 [lmv] [12732.439731] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [12732.449152] ll_close_inode_openhandle+0x493/0xea0 [lustre] [12732.457640] ll_md_real_close+0xa5/0x2a0 [lustre] [12732.465110] ll_md_close+0x274/0x800 [lustre] [12732.466709] ll_file_release+0x128/0xc50 [lustre] [12732.480699] __fput+0x10d/0x440 [12732.485720] ____fput+0x12/0x20 [12732.487465] task_work_run+0xc6/0x110 [12732.490081] exit_to_usermode_loop+0x1dd/0x1f0 [12732.495712] do_syscall_64+0x3d6/0x440 [12732.503473] entry_SYSCALL_64_after_hwframe+0x49/0xae [12732.511102] RIP: 0033:0x7ff54af45cd8 [12732.515479] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [12732.539817] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [12732.548738] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [12732.560660] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [12732.568168] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [12732.570591] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [12732.579630] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12734.431051] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10775s! [12734.435856] Showing busy workqueues and worker pools: [12734.438897] workqueue events: flags=0x0 [12734.441388] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12734.445742] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [12734.451379] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [12734.455657] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [12734.463243] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12734.467373] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [12734.471815] workqueue events_long: flags=0x0 [12734.477427] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12734.481699] pending: lu_objects_flush [obdclass] [12734.484962] workqueue events_unbound: flags=0x2 [12734.488689] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [12734.492477] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [12734.498493] workqueue events_power_efficient: flags=0x80 [12734.501590] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12734.505433] pending: neigh_periodic_work [12734.508331] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12734.512267] pending: do_cache_clean, check_lifetime [12734.515486] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12734.519593] pending: neigh_periodic_work [12734.522313] workqueue writeback: flags=0x4a [12734.524525] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [12734.528580] pending: wb_workfn, wb_workfn [12734.531276] workqueue xprtiod: flags=0xa [12734.533516] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [12734.536985] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [12734.541818] workqueue ipv6_addrconf: flags=0x40008 [12734.544551] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12734.548312] pending: addrconf_verify_work [12734.550647] workqueue knbd0-recv: flags=0x1a [12734.553190] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12734.556515] in-flight: 657:recv_work [12734.559491] workqueue ptlrpc_pinger: flags=0x2 [12734.561671] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12734.564951] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [12734.567561] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10775s workers=4 idle: 20831 12642 28859 [12734.572740] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12724s workers=2 idle: 659 [12744.059273] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [12744.063087] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [12744.067707] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12744.068382] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12744.068395] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12744.068402] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12744.069966] RIP: 0010:native_safe_halt+0x2e/0x50 [12744.069980] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12744.069985] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12744.069990] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [12744.069993] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c2134154 [12744.069995] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [12744.069997] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [12744.070000] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [12744.070003] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [12744.070006] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12744.070009] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [12744.070017] Call Trace: [12744.070026] [12744.070033] ? show_regs.cold.9+0x22/0x2f [12744.070043] ? watchdog_timer_fn.cold.6+0x62/0x10a [12744.070134] ? __hrtimer_run_queues+0x1c2/0x4d0 [12744.070139] ? watchdog+0x40/0x40 [12744.070143] ? hrtimer_interrupt+0x128/0x2c0 [12744.070147] ? smp_apic_timer_interrupt+0xa6/0x270 [12744.070153] ? apic_timer_interrupt+0xf/0x20 [12744.070157] [12744.070161] ? native_safe_halt+0x2e/0x50 [12744.070164] kvm_wait+0xcd/0xe0 [12744.070172] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12744.070180] do_raw_spin_lock+0x177/0x190 [12744.070185] _raw_spin_lock+0x21/0x30 [12744.070190] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12744.070439] process_one_work+0x2c8/0x700 [12744.070445] worker_thread+0x296/0x6e0 [12744.070449] ? rescuer_thread+0x570/0x570 [12744.070453] kthread+0x1d1/0x200 [12744.070458] ? set_kthread_struct+0x70/0x70 [12744.070461] ret_from_fork+0x1f/0x30 [12744.310483] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12744.350526] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12744.357530] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12744.362271] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12744.365211] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12744.373040] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12744.376318] RAX: 0000000000007e65 RBX: ffff96848811cae0 RCX: 0000000000000001 [12744.379513] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [12744.382292] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [12744.385091] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [12744.388021] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [12744.397568] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [12744.402519] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12744.405315] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [12744.408499] Call Trace: [12744.410184] [12744.411194] ? show_regs.cold.9+0x22/0x2f [12744.413345] ? watchdog_timer_fn.cold.6+0x62/0x10a [12744.415894] ? __hrtimer_run_queues+0x1c2/0x4d0 [12744.418250] ? watchdog+0x40/0x40 [12744.419884] ? hrtimer_interrupt+0x128/0x2c0 [12744.421881] ? smp_apic_timer_interrupt+0xa6/0x270 [12744.424566] ? apic_timer_interrupt+0xf/0x20 [12744.431880] [12744.435816] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12744.447275] do_raw_spin_lock+0x177/0x190 [12744.455183] _raw_spin_lock+0x21/0x30 [12744.458522] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12744.469944] ? do_raw_spin_unlock+0x75/0x190 [12744.476901] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12744.485800] ? do_wait_intr+0xf0/0xf0 [12744.487164] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12744.494211] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [12744.501448] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12744.510419] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12744.517815] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12744.522388] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [12744.525439] mdc_intent_lock+0x353/0x7e0 [mdc] [12744.528406] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12744.531975] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12744.535104] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12744.537914] lmv_intent_open+0x338/0x1150 [lmv] [12744.539947] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12744.542324] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12744.546990] lmv_intent_lock+0x1dd/0x4c0 [lmv] [12744.550920] ll_intent_lock+0xc5/0xaa0 [lustre] [12744.553397] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12744.561206] ll_lookup_it+0xe0e/0x2270 [lustre] [12744.565810] ? do_raw_spin_unlock+0x75/0x190 [12744.572822] ? _raw_spin_unlock+0x12/0x30 [12744.580440] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [12744.590368] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [12744.599234] ll_atomic_open+0x36a/0xd70 [lustre] [12744.604235] lookup_open+0x356/0xb40 [12744.606723] do_last+0x313/0xfc0 [12744.608947] ? nd_jump_root+0xe5/0x160 [12744.611126] ? path_init+0x437/0x520 [12744.613536] path_openat+0xf7/0x500 [12744.619165] ? do_raw_spin_unlock+0x75/0x190 [12744.621285] do_filp_open+0x99/0x140 [12744.626442] ? getname_flags+0x6e/0x330 [12744.631262] ? __check_object_size+0xff/0x256 [12744.635452] ? do_raw_spin_unlock+0x75/0x190 [12744.637702] ? _raw_spin_unlock+0x12/0x30 [12744.640687] do_sys_openat2+0x2b4/0x410 [12744.643404] do_sys_open+0x73/0xa0 [12744.649093] __x64_sys_openat+0x24/0x30 [12744.655107] do_syscall_64+0xc1/0x440 [12744.656384] entry_SYSCALL_64_after_hwframe+0x49/0xae [12744.662462] RIP: 0033:0x7ff54af45332 [12744.663729] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [12744.678782] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [12744.686194] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [12744.690961] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [12744.701919] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [12744.706762] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [12744.712405] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [12760.059108] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [12760.061766] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [12760.067639] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12760.067731] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12760.067742] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12760.067744] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [12760.067760] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [12760.067765] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12760.067770] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [12760.067772] RDX: 0000000000006b29 RSI: 0000000000000003 RDI: ffff96848811cae0 [12760.067775] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [12760.067777] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [12760.067780] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12760.067783] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [12760.067787] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12760.067789] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [12760.067796] Call Trace: [12760.067808] [12760.067814] ? show_regs.cold.9+0x22/0x2f [12760.067822] ? watchdog_timer_fn.cold.6+0x62/0x10a [12760.067828] ? __hrtimer_run_queues+0x1c2/0x4d0 [12760.067832] ? watchdog+0x40/0x40 [12760.067836] ? hrtimer_interrupt+0x128/0x2c0 [12760.067841] ? smp_apic_timer_interrupt+0xa6/0x270 [12760.067848] ? apic_timer_interrupt+0xf/0x20 [12760.067853] [12760.068083] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [12760.068090] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12760.068095] do_raw_spin_lock+0x177/0x190 [12760.068100] _raw_spin_lock+0x21/0x30 [12760.068106] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12760.068675] obd_mod_free+0x46/0x140 [ptlrpc] [12760.069015] mdc_commit_open+0x91/0xb0 [mdc] [12760.069051] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12760.069422] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12760.079449] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12760.079704] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12760.079711] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12760.079714] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12760.079729] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12760.079733] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12760.079738] RAX: 0000000000006394 RBX: ffff96848811cae0 RCX: 0000000000000001 [12760.079740] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [12760.079742] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [12760.079744] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [12760.079747] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [12760.079750] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [12760.079752] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12760.079754] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [12760.079763] Call Trace: [12760.079769] [12760.079775] ? show_regs.cold.9+0x22/0x2f [12760.079784] ? watchdog_timer_fn.cold.6+0x62/0x10a [12760.079791] ? __hrtimer_run_queues+0x1c2/0x4d0 [12760.080091] ? watchdog+0x40/0x40 [12760.080096] ? hrtimer_interrupt+0x128/0x2c0 [12760.080100] ? smp_apic_timer_interrupt+0xa6/0x270 [12760.080109] ? apic_timer_interrupt+0xf/0x20 [12760.080114] [12760.080117] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12760.080124] do_raw_spin_lock+0x177/0x190 [12760.080130] _raw_spin_lock+0x21/0x30 [12760.080136] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12760.131249] after_reply+0xe2c/0x1780 [ptlrpc] [12760.131548] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12760.131763] ptlrpcd+0xa82/0xe40 [ptlrpc] [12760.132073] ? do_wait_intr+0xf0/0xf0 [12760.132083] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12760.132269] kthread+0x1d1/0x200 [12760.132275] ? set_kthread_struct+0x70/0x70 [12760.132279] ret_from_fork+0x1f/0x30 [12760.602316] ? do_raw_spin_unlock+0x75/0x190 [12760.607122] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12760.611258] ? do_wait_intr+0xf0/0xf0 [12760.620426] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12760.626290] mdc_close+0x53b/0xd70 [mdc] [12760.629145] lmv_close+0x18d/0x490 [lmv] [12760.632270] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [12760.636105] ll_close_inode_openhandle+0x493/0xea0 [lustre] [12760.639377] ll_md_real_close+0xa5/0x2a0 [lustre] [12760.643458] ll_md_close+0x274/0x800 [lustre] [12760.646364] ll_file_release+0x128/0xc50 [lustre] [12760.650974] __fput+0x10d/0x440 [12760.653380] ____fput+0x12/0x20 [12760.655826] task_work_run+0xc6/0x110 [12760.658980] exit_to_usermode_loop+0x1dd/0x1f0 [12760.663200] do_syscall_64+0x3d6/0x440 [12760.666592] entry_SYSCALL_64_after_hwframe+0x49/0xae [12760.675000] RIP: 0033:0x7ff54af45cd8 [12760.676763] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [12760.690447] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [12760.697368] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [12760.701906] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [12760.707520] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [12760.714141] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [12760.717502] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12764.643932] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10805s! [12764.653251] Showing busy workqueues and worker pools: [12764.659731] workqueue events: flags=0x0 [12764.668394] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12764.685283] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [12764.706346] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [12764.711382] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [12764.719350] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12764.726351] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [12764.742438] workqueue events_long: flags=0x0 [12764.743929] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12764.762173] pending: lu_objects_flush [obdclass] [12764.768167] workqueue events_unbound: flags=0x2 [12764.775066] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [12764.787879] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [12764.797057] workqueue events_power_efficient: flags=0x80 [12764.802516] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12764.808225] pending: neigh_periodic_work [12764.810218] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12764.813524] pending: do_cache_clean, check_lifetime [12764.817207] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12764.821163] pending: neigh_periodic_work [12764.830831] workqueue writeback: flags=0x4a [12764.832283] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [12764.847799] pending: wb_workfn, wb_workfn [12764.850371] workqueue xprtiod: flags=0xa [12764.853030] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [12764.861293] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [12764.868080] workqueue ipv6_addrconf: flags=0x40008 [12764.872426] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12764.886782] pending: addrconf_verify_work [12764.891541] workqueue knbd0-recv: flags=0x1a [12764.902991] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12764.913017] in-flight: 657:recv_work [12764.919121] workqueue ptlrpc_pinger: flags=0x2 [12764.927540] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12764.935153] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [12764.937399] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10806s workers=4 idle: 20831 12642 28859 [12764.960183] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12755s workers=2 idle: 659 [12772.061360] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [12772.064631] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [12772.070452] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12772.070608] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12772.070615] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12772.070621] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12772.070999] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12772.071011] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12772.071015] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12772.071020] RAX: 00000000000072b2 RBX: ffff96848811cae0 RCX: 0000000000000001 [12772.071023] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [12772.071025] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [12772.071027] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [12772.071030] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [12772.071033] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [12772.071035] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12772.071038] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [12772.071048] Call Trace: [12772.071054] [12772.071059] ? show_regs.cold.9+0x22/0x2f [12772.071069] ? watchdog_timer_fn.cold.6+0x62/0x10a [12772.071075] ? __hrtimer_run_queues+0x1c2/0x4d0 [12772.071079] ? watchdog+0x40/0x40 [12772.071084] ? hrtimer_interrupt+0x128/0x2c0 [12772.071088] ? smp_apic_timer_interrupt+0xa6/0x270 [12772.071096] ? apic_timer_interrupt+0xf/0x20 [12772.071101] [12772.071104] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12772.071109] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12772.071113] do_raw_spin_lock+0x177/0x190 [12772.071118] _raw_spin_lock+0x21/0x30 [12772.071123] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12772.077718] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12772.077808] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12772.077815] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12772.077818] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12772.077832] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12772.077836] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12772.077841] RAX: 0000000000007c24 RBX: ffff96848811cae0 RCX: 0000000000000001 [12772.077844] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [12772.077847] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [12772.077849] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [12772.077851] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [12772.077854] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [12772.077858] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12772.077860] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [12772.077869] Call Trace: [12772.077875] [12772.077881] ? show_regs.cold.9+0x22/0x2f [12772.077891] ? watchdog_timer_fn.cold.6+0x62/0x10a [12772.077898] ? __hrtimer_run_queues+0x1c2/0x4d0 [12772.077903] ? watchdog+0x40/0x40 [12772.077907] ? hrtimer_interrupt+0x128/0x2c0 [12772.077911] ? smp_apic_timer_interrupt+0xa6/0x270 [12772.077919] ? apic_timer_interrupt+0xf/0x20 [12772.077924] [12772.077927] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12772.077935] do_raw_spin_lock+0x177/0x190 [12772.077939] _raw_spin_lock+0x21/0x30 [12772.077944] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12772.078221] ? do_raw_spin_unlock+0x75/0x190 [12772.078227] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12772.078424] ? do_wait_intr+0xf0/0xf0 [12772.078432] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12772.079319] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [12772.079711] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12772.079992] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12772.080118] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12772.080319] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [12772.080356] mdc_intent_lock+0x353/0x7e0 [mdc] [12772.080450] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12772.116788] process_one_work+0x2c8/0x700 [12772.116809] worker_thread+0x296/0x6e0 [12772.116814] ? rescuer_thread+0x570/0x570 [12772.116818] kthread+0x1d1/0x200 [12772.116823] ? set_kthread_struct+0x70/0x70 [12772.116827] ret_from_fork+0x1f/0x30 [12772.402942] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12772.412300] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12772.415991] lmv_intent_open+0x338/0x1150 [lmv] [12772.421053] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12772.429717] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12772.433836] lmv_intent_lock+0x1dd/0x4c0 [lmv] [12772.436951] ll_intent_lock+0xc5/0xaa0 [lustre] [12772.439713] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12772.443180] ll_lookup_it+0xe0e/0x2270 [lustre] [12772.446333] ? do_raw_spin_unlock+0x75/0x190 [12772.448568] ? _raw_spin_unlock+0x12/0x30 [12772.450072] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [12772.453860] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [12772.456127] ll_atomic_open+0x36a/0xd70 [lustre] [12772.458232] lookup_open+0x356/0xb40 [12772.459255] do_last+0x313/0xfc0 [12772.460439] ? nd_jump_root+0xe5/0x160 [12772.461975] ? path_init+0x437/0x520 [12772.463930] path_openat+0xf7/0x500 [12772.466089] ? do_raw_spin_unlock+0x75/0x190 [12772.469985] do_filp_open+0x99/0x140 [12772.471330] ? getname_flags+0x6e/0x330 [12772.477479] ? __check_object_size+0xff/0x256 [12772.479761] ? do_raw_spin_unlock+0x75/0x190 [12772.483817] ? _raw_spin_unlock+0x12/0x30 [12772.489507] do_sys_openat2+0x2b4/0x410 [12772.493700] do_sys_open+0x73/0xa0 [12772.497683] __x64_sys_openat+0x24/0x30 [12772.501762] do_syscall_64+0xc1/0x440 [12772.504399] entry_SYSCALL_64_after_hwframe+0x49/0xae [12772.507306] RIP: 0033:0x7ff54af45332 [12772.509498] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [12772.528618] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [12772.533469] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [12772.542378] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [12772.548883] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [12772.552928] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [12772.561832] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [12788.062725] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [12788.069972] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12788.104451] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12788.110107] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12788.113360] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12788.115356] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12788.124672] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12788.129505] RAX: 0000000000003687 RBX: ffff96848811cae0 RCX: 0000000000000001 [12788.132572] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [12788.135518] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [12788.137201] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [12788.139723] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [12788.142783] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [12788.145802] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12788.148190] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [12788.151296] Call Trace: [12788.152139] [12788.153283] ? show_regs.cold.9+0x22/0x2f [12788.154967] ? watchdog_timer_fn.cold.6+0x62/0x10a [12788.157133] ? __hrtimer_run_queues+0x1c2/0x4d0 [12788.158863] ? watchdog+0x40/0x40 [12788.160392] ? hrtimer_interrupt+0x128/0x2c0 [12788.162072] ? smp_apic_timer_interrupt+0xa6/0x270 [12788.164151] ? apic_timer_interrupt+0xf/0x20 [12788.165969] [12788.166906] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12788.169287] do_raw_spin_lock+0x177/0x190 [12788.171509] _raw_spin_lock+0x21/0x30 [12788.172936] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12788.175705] ? do_raw_spin_unlock+0x75/0x190 [12788.177802] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12788.180069] ? do_wait_intr+0xf0/0xf0 [12788.181529] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12788.182853] mdc_close+0x53b/0xd70 [mdc] [12788.183753] lmv_close+0x18d/0x490 [lmv] [12788.184679] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [12788.186318] ll_close_inode_openhandle+0x493/0xea0 [lustre] [12788.187678] ll_md_real_close+0xa5/0x2a0 [lustre] [12788.188648] ll_md_close+0x274/0x800 [lustre] [12788.189677] ll_file_release+0x128/0xc50 [lustre] [12788.190817] __fput+0x10d/0x440 [12788.191489] ____fput+0x12/0x20 [12788.192159] task_work_run+0xc6/0x110 [12788.192891] exit_to_usermode_loop+0x1dd/0x1f0 [12788.193871] do_syscall_64+0x3d6/0x440 [12788.194703] entry_SYSCALL_64_after_hwframe+0x49/0xae [12788.195776] RIP: 0033:0x7ff54af45cd8 [12788.196576] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [12788.200635] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [12788.202826] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [12788.204577] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [12788.206772] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [12788.208367] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [12788.210172] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12792.058246] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [12792.067999] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12792.128701] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12792.138922] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12792.145290] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [12792.149698] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [12792.163086] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12792.170615] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [12792.175692] RDX: 00000000000052a1 RSI: 0000000000000003 RDI: ffff96848811cae0 [12792.181384] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [12792.189729] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [12792.200400] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12792.216951] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [12792.227448] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12792.235142] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [12792.243830] Call Trace: [12792.245667] [12792.254934] ? show_regs.cold.9+0x22/0x2f [12792.258058] ? watchdog_timer_fn.cold.6+0x62/0x10a [12792.269366] ? __hrtimer_run_queues+0x1c2/0x4d0 [12792.271568] ? watchdog+0x40/0x40 [12792.273233] ? hrtimer_interrupt+0x128/0x2c0 [12792.280254] ? smp_apic_timer_interrupt+0xa6/0x270 [12792.289472] ? apic_timer_interrupt+0xf/0x20 [12792.294444] [12792.296021] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [12792.302527] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12792.308516] do_raw_spin_lock+0x177/0x190 [12792.314370] _raw_spin_lock+0x21/0x30 [12792.320276] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12792.324508] obd_mod_free+0x46/0x140 [ptlrpc] [12792.335011] mdc_commit_open+0x91/0xb0 [mdc] [12792.339005] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12792.344910] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12792.357529] after_reply+0xe2c/0x1780 [ptlrpc] [12792.368027] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12792.376816] ptlrpcd+0xa82/0xe40 [ptlrpc] [12792.381323] ? do_wait_intr+0xf0/0xf0 [12792.383120] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12792.389947] kthread+0x1d1/0x200 [12792.393894] ? set_kthread_struct+0x70/0x70 [12792.398663] ret_from_fork+0x1f/0x30 [12795.359017] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10836s! [12795.366698] Showing busy workqueues and worker pools: [12795.374016] workqueue events: flags=0x0 [12795.384571] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12795.395623] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [12795.410534] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [12795.420377] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [12795.438551] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12795.448982] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [12795.456363] workqueue events_long: flags=0x0 [12795.460110] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12795.471508] pending: lu_objects_flush [obdclass] [12795.478630] workqueue events_unbound: flags=0x2 [12795.485742] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [12795.490081] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [12795.502346] workqueue events_power_efficient: flags=0x80 [12795.507540] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12795.515916] pending: neigh_periodic_work [12795.521013] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12795.525178] pending: do_cache_clean, check_lifetime [12795.527289] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12795.530368] pending: neigh_periodic_work [12795.536409] workqueue writeback: flags=0x4a [12795.542103] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [12795.550510] pending: wb_workfn, wb_workfn [12795.558460] workqueue xprtiod: flags=0xa [12795.564499] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [12795.571552] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [12795.576300] workqueue ipv6_addrconf: flags=0x40008 [12795.583016] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12795.589867] pending: addrconf_verify_work [12795.597632] workqueue knbd0-recv: flags=0x1a [12795.599242] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12795.602918] in-flight: 657:recv_work [12795.604944] workqueue ptlrpc_pinger: flags=0x2 [12795.607195] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12795.613271] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [12795.616845] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10836s workers=4 idle: 20831 12642 28859 [12795.622278] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12785s workers=2 idle: 659 [12800.059028] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [12800.067360] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [12800.069647] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12800.069725] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12800.069733] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12800.069739] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12800.070562] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12800.070574] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12800.070578] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12800.070583] RAX: 0000000000005b41 RBX: ffff96848811cae0 RCX: 0000000000000001 [12800.070586] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [12800.070587] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [12800.070589] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [12800.070591] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [12800.070594] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [12800.070597] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12800.070600] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [12800.070609] Call Trace: [12800.070616] [12800.070621] ? show_regs.cold.9+0x22/0x2f [12800.070630] ? watchdog_timer_fn.cold.6+0x62/0x10a [12800.070636] ? __hrtimer_run_queues+0x1c2/0x4d0 [12800.070640] ? watchdog+0x40/0x40 [12800.070644] ? hrtimer_interrupt+0x128/0x2c0 [12800.070648] ? smp_apic_timer_interrupt+0xa6/0x270 [12800.070656] ? apic_timer_interrupt+0xf/0x20 [12800.070660] [12800.070663] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12800.070668] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12800.070672] do_raw_spin_lock+0x177/0x190 [12800.070677] _raw_spin_lock+0x21/0x30 [12800.070682] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12800.071483] process_one_work+0x2c8/0x700 [12800.071491] worker_thread+0x296/0x6e0 [12800.071495] ? rescuer_thread+0x570/0x570 [12800.071499] kthread+0x1d1/0x200 [12800.071503] ? set_kthread_struct+0x70/0x70 [12800.071507] ret_from_fork+0x1f/0x30 [12800.304729] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12800.355984] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12800.365572] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12800.376824] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12800.383980] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12800.403969] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12800.409493] RAX: 0000000000003b3a RBX: ffff96848811cae0 RCX: 0000000000000001 [12800.416044] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [12800.419423] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [12800.426735] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [12800.438561] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [12800.448668] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [12800.451603] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12800.456943] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [12800.471018] Call Trace: [12800.475905] [12800.479689] ? show_regs.cold.9+0x22/0x2f [12800.483385] ? watchdog_timer_fn.cold.6+0x62/0x10a [12800.491080] ? __hrtimer_run_queues+0x1c2/0x4d0 [12800.492502] ? watchdog+0x40/0x40 [12800.493524] ? hrtimer_interrupt+0x128/0x2c0 [12800.498453] ? smp_apic_timer_interrupt+0xa6/0x270 [12800.500825] ? apic_timer_interrupt+0xf/0x20 [12800.507748] [12800.508500] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12800.510169] do_raw_spin_lock+0x177/0x190 [12800.518289] _raw_spin_lock+0x21/0x30 [12800.520757] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12800.525505] ? do_raw_spin_unlock+0x75/0x190 [12800.529943] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12800.533385] ? do_wait_intr+0xf0/0xf0 [12800.535016] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12800.540450] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [12800.546657] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12800.549941] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12800.554017] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12800.560822] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [12800.564103] mdc_intent_lock+0x353/0x7e0 [mdc] [12800.570334] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12800.575436] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12800.579413] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12800.581148] lmv_intent_open+0x338/0x1150 [lmv] [12800.587991] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12800.593009] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12800.595365] lmv_intent_lock+0x1dd/0x4c0 [lmv] [12800.599233] ll_intent_lock+0xc5/0xaa0 [lustre] [12800.603896] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12800.609904] ll_lookup_it+0xe0e/0x2270 [lustre] [12800.617826] ? do_raw_spin_unlock+0x75/0x190 [12800.621530] ? _raw_spin_unlock+0x12/0x30 [12800.623698] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [12800.628660] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [12800.630743] ll_atomic_open+0x36a/0xd70 [lustre] [12800.633856] lookup_open+0x356/0xb40 [12800.636638] do_last+0x313/0xfc0 [12800.640821] ? nd_jump_root+0xe5/0x160 [12800.642816] ? path_init+0x437/0x520 [12800.644250] path_openat+0xf7/0x500 [12800.645677] ? do_raw_spin_unlock+0x75/0x190 [12800.647767] do_filp_open+0x99/0x140 [12800.649715] ? getname_flags+0x6e/0x330 [12800.652598] ? __check_object_size+0xff/0x256 [12800.653944] ? do_raw_spin_unlock+0x75/0x190 [12800.655833] ? _raw_spin_unlock+0x12/0x30 [12800.657703] do_sys_openat2+0x2b4/0x410 [12800.660623] do_sys_open+0x73/0xa0 [12800.661872] __x64_sys_openat+0x24/0x30 [12800.664042] do_syscall_64+0xc1/0x440 [12800.665730] entry_SYSCALL_64_after_hwframe+0x49/0xae [12800.668077] RIP: 0033:0x7ff54af45332 [12800.669334] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [12800.678452] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [12800.681634] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [12800.686353] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [12800.692688] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [12800.699357] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [12800.703151] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [12814.139603] rcu: INFO: rcu_sched self-detected stall on CPU [12814.142948] rcu: 0-...!: (5759654 ticks this GP) idle=98a/1/0x4000000000000002 softirq=125154/125154 fqs=959 [12814.152721] (t=10860299 jiffies g=253585 q=6359) [12814.157175] rcu: rcu_sched kthread starved for 10855259 jiffies! g253585 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [12814.163911] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [12814.172196] rcu: RCU grace-period kthread stack dump: [12814.179534] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [12814.193964] Call Trace: [12814.201023] __schedule+0x351/0xcb0 [12814.202844] schedule+0xc0/0x180 [12814.209376] schedule_timeout+0xb4/0x190 [12814.216283] ? do_raw_spin_unlock+0x75/0x190 [12814.224590] ? __next_timer_interrupt+0x160/0x160 [12814.226070] ? prepare_to_swait_event+0xf3/0x140 [12814.231003] rcu_gp_kthread+0x8dd/0x15b0 [12814.232394] ? force_qs_rnp+0x2c0/0x2c0 [12814.239733] kthread+0x1d1/0x200 [12814.241200] ? set_kthread_struct+0x70/0x70 [12814.248619] ret_from_fork+0x1f/0x30 [12814.250892] rcu: Stack dump where RCU GP kthread last ran: [12814.257521] Sending NMI from CPU 0 to CPUs 1: [12814.262616] NMI backtrace for cpu 1 [12814.262623] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12814.262625] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12814.262627] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12814.262629] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12814.262631] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 [12814.262634] RAX: 0000000000001483 RBX: ffff96848811cae0 RCX: 0000000000000001 [12814.262635] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [12814.262637] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [12814.262638] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [12814.262639] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [12814.262641] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [12814.262643] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12814.262644] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [12814.262645] Call Trace: [12814.262646] [12814.262647] ? show_regs.cold.9+0x22/0x2f [12814.262649] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [12814.262650] ? nmi_cpu_backtrace_handler+0x15/0x40 [12814.262651] ? nmi_handle+0xbd/0x210 [12814.262652] ? default_do_nmi+0x4b/0x1b0 [12814.262654] ? do_nmi+0x280/0x380 [12814.262655] ? end_repeat_nmi+0x16/0x1f [12814.262656] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12814.262657] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12814.262658] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12814.262659] [12814.262660] do_raw_spin_lock+0x177/0x190 [12814.262661] _raw_spin_lock+0x21/0x30 [12814.262662] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12814.262663] ? do_raw_spin_unlock+0x75/0x190 [12814.262664] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12814.262665] ? do_wait_intr+0xf0/0xf0 [12814.262667] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12814.262668] mdc_close+0x53b/0xd70 [mdc] [12814.262669] lmv_close+0x18d/0x490 [lmv] [12814.262670] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [12814.262671] ll_close_inode_openhandle+0x493/0xea0 [lustre] [12814.262672] ll_md_real_close+0xa5/0x2a0 [lustre] [12814.262673] ll_md_close+0x274/0x800 [lustre] [12814.262674] ll_file_release+0x128/0xc50 [lustre] [12814.262675] __fput+0x10d/0x440 [12814.262676] ____fput+0x12/0x20 [12814.262677] task_work_run+0xc6/0x110 [12814.262678] exit_to_usermode_loop+0x1dd/0x1f0 [12814.262678] do_syscall_64+0x3d6/0x440 [12814.262680] entry_SYSCALL_64_after_hwframe+0x49/0xae [12814.262680] RIP: 0033:0x7ff54af45cd8 [12814.262682] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [12814.262684] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [12814.262686] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [12814.262687] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [12814.262689] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [12814.262690] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [12814.262691] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12814.267611] NMI backtrace for cpu 0 [12814.415386] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12814.419582] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12814.423156] Call Trace: [12814.423935] [12814.424892] ? dump_stack+0xbb/0x10e [12814.426293] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [12814.428083] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [12814.429640] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [12814.431596] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [12814.433254] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [12814.435381] ? rcu_dump_cpu_stacks+0x11b/0x171 [12814.437234] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [12814.439204] ? update_process_times+0x6d/0xc0 [12814.440842] ? tick_sched_handle.isra.18+0x27/0x90 [12814.442372] ? tick_sched_timer+0x53/0xb0 [12814.444437] ? __hrtimer_run_queues+0x1c2/0x4d0 [12814.453751] ? tick_sched_handle.isra.18+0x90/0x90 [12814.457722] ? hrtimer_interrupt+0x128/0x2c0 [12814.463221] ? smp_apic_timer_interrupt+0xa6/0x270 [12814.466582] ? apic_timer_interrupt+0xf/0x20 [12814.471280] [12814.473063] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [12814.482739] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12814.486778] ? do_raw_spin_lock+0x177/0x190 [12814.493937] ? _raw_spin_lock+0x21/0x30 [12814.497672] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12814.502388] ? obd_mod_free+0x46/0x140 [ptlrpc] [12814.510080] ? mdc_commit_open+0x91/0xb0 [mdc] [12814.517010] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12814.520716] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12814.526413] ? after_reply+0xe2c/0x1780 [ptlrpc] [12814.529259] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12814.532841] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [12814.536534] ? do_wait_intr+0xf0/0xf0 [12814.540879] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12814.549660] ? kthread+0x1d1/0x200 [12814.553879] ? set_kthread_struct+0x70/0x70 [12814.563204] ? ret_from_fork+0x1f/0x30 [12826.080341] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10867s! [12826.087803] Showing busy workqueues and worker pools: [12826.096684] workqueue events: flags=0x0 [12826.104241] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12826.106517] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [12826.119991] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [12826.132607] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [12826.145646] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12826.150210] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [12826.158822] workqueue events_long: flags=0x0 [12826.160077] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12826.172368] pending: lu_objects_flush [obdclass] [12826.181835] workqueue events_unbound: flags=0x2 [12826.187113] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [12826.194834] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [12826.208220] workqueue events_power_efficient: flags=0x80 [12826.213274] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12826.221579] pending: neigh_periodic_work [12826.225509] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12826.232768] pending: do_cache_clean, check_lifetime [12826.236261] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12826.248272] pending: neigh_periodic_work [12826.250316] workqueue writeback: flags=0x4a [12826.256751] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [12826.266670] pending: wb_workfn, wb_workfn [12826.271112] workqueue xprtiod: flags=0xa [12826.273968] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [12826.285062] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [12826.297300] workqueue ipv6_addrconf: flags=0x40008 [12826.303221] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12826.309657] pending: addrconf_verify_work [12826.316660] workqueue knbd0-recv: flags=0x1a [12826.319331] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12826.325127] in-flight: 657:recv_work [12826.331184] workqueue ptlrpc_pinger: flags=0x2 [12826.332770] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12826.337800] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [12826.347315] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10867s workers=4 idle: 20831 12642 28859 [12826.352561] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12816s workers=2 idle: 659 [12828.061086] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [12828.063283] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [12828.072149] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log [12828.087722] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12828.087803] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12828.087810] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12828.087923] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12828.089073] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12828.089087] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12828.089092] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12828.089097] RAX: 0000000000004185 RBX: ffff96848811cae0 RCX: 0000000000000001 [12828.089100] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [12828.089102] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [12828.089104] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [12828.089107] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [12828.089110] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [12828.089112] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12828.089114] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [12828.089121] Call Trace: [12828.089126] [12828.089134] ? show_regs.cold.9+0x22/0x2f [12828.089144] ? watchdog_timer_fn.cold.6+0x62/0x10a [12828.089150] ? __hrtimer_run_queues+0x1c2/0x4d0 [12828.089162] ? watchdog+0x40/0x40 [12828.089166] ? hrtimer_interrupt+0x128/0x2c0 [12828.089171] ? smp_apic_timer_interrupt+0xa6/0x270 [12828.089179] ? apic_timer_interrupt+0xf/0x20 [12828.089183] [12828.089186] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12828.089192] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12828.089197] do_raw_spin_lock+0x177/0x190 [12828.089202] _raw_spin_lock+0x21/0x30 [12828.089207] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12828.089457] process_one_work+0x2c8/0x700 [12828.089464] worker_thread+0x296/0x6e0 [12828.089467] ? rescuer_thread+0x570/0x570 [12828.089471] kthread+0x1d1/0x200 [12828.089476] ? set_kthread_struct+0x70/0x70 [12828.089479] ret_from_fork+0x1f/0x30 [12828.484957] dm_mod sha512_ssse3 sha512_generic [12828.490784] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12828.501819] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12828.509571] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12828.514128] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12828.526231] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12828.529321] RAX: 0000000000006738 RBX: ffff96848811cae0 RCX: 0000000000000001 [12828.533109] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [12828.536381] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [12828.539805] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [12828.546501] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [12828.551302] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [12828.553190] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12828.556387] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [12828.561674] Call Trace: [12828.562797] [12828.564309] ? show_regs.cold.9+0x22/0x2f [12828.566155] ? watchdog_timer_fn.cold.6+0x62/0x10a [12828.570847] ? __hrtimer_run_queues+0x1c2/0x4d0 [12828.580030] ? watchdog+0x40/0x40 [12828.586946] ? hrtimer_interrupt+0x128/0x2c0 [12828.592644] ? smp_apic_timer_interrupt+0xa6/0x270 [12828.596272] ? apic_timer_interrupt+0xf/0x20 [12828.599634] [12828.602715] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12828.607665] do_raw_spin_lock+0x177/0x190 [12828.611670] _raw_spin_lock+0x21/0x30 [12828.616063] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12828.622555] ? do_raw_spin_unlock+0x75/0x190 [12828.629135] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12828.632525] ? do_wait_intr+0xf0/0xf0 [12828.635790] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12828.639612] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [12828.643512] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12828.648984] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12828.650802] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12828.655104] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [12828.661266] mdc_intent_lock+0x353/0x7e0 [mdc] [12828.662804] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12828.666085] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12828.671805] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12828.677680] lmv_intent_open+0x338/0x1150 [lmv] [12828.680158] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12828.689108] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12828.695574] lmv_intent_lock+0x1dd/0x4c0 [lmv] [12828.698654] ll_intent_lock+0xc5/0xaa0 [lustre] [12828.702699] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12828.705632] ll_lookup_it+0xe0e/0x2270 [lustre] [12828.708580] ? do_raw_spin_unlock+0x75/0x190 [12828.711648] ? _raw_spin_unlock+0x12/0x30 [12828.713634] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [12828.716616] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [12828.719479] ll_atomic_open+0x36a/0xd70 [lustre] [12828.725779] lookup_open+0x356/0xb40 [12828.730353] do_last+0x313/0xfc0 [12828.731443] ? nd_jump_root+0xe5/0x160 [12828.732733] ? path_init+0x437/0x520 [12828.735738] path_openat+0xf7/0x500 [12828.737767] ? do_raw_spin_unlock+0x75/0x190 [12828.741578] do_filp_open+0x99/0x140 [12828.743383] ? getname_flags+0x6e/0x330 [12828.749110] ? __check_object_size+0xff/0x256 [12828.750698] ? do_raw_spin_unlock+0x75/0x190 [12828.758516] ? _raw_spin_unlock+0x12/0x30 [12828.760755] do_sys_openat2+0x2b4/0x410 [12828.762343] do_sys_open+0x73/0xa0 [12828.763565] __x64_sys_openat+0x24/0x30 [12828.773695] do_syscall_64+0xc1/0x440 [12828.775133] entry_SYSCALL_64_after_hwframe+0x49/0xae [12828.781711] RIP: 0033:0x7ff54af45332 [12828.784641] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [12828.799348] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [12828.807506] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [12828.813467] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [12828.820577] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [12828.827950] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [12828.835715] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [12840.058466] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [12840.062991] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12840.089751] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12840.102034] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12840.110260] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12840.121600] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12840.143086] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12840.148221] RAX: 0000000000002585 RBX: ffff96848811cae0 RCX: 0000000000000001 [12840.156460] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [12840.161822] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [12840.164886] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [12840.171897] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [12840.179787] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [12840.188574] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12840.195548] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [12840.199573] Call Trace: [12840.200525] [12840.201506] ? show_regs.cold.9+0x22/0x2f [12840.204598] ? watchdog_timer_fn.cold.6+0x62/0x10a [12840.206500] ? __hrtimer_run_queues+0x1c2/0x4d0 [12840.207685] ? watchdog+0x40/0x40 [12840.210238] ? hrtimer_interrupt+0x128/0x2c0 [12840.213628] ? smp_apic_timer_interrupt+0xa6/0x270 [12840.217982] ? apic_timer_interrupt+0xf/0x20 [12840.221141] [12840.223968] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12840.231220] do_raw_spin_lock+0x177/0x190 [12840.236720] _raw_spin_lock+0x21/0x30 [12840.241609] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12840.243174] ? do_raw_spin_unlock+0x75/0x190 [12840.248672] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12840.252422] ? do_wait_intr+0xf0/0xf0 [12840.253728] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12840.256454] mdc_close+0x53b/0xd70 [mdc] [12840.258603] lmv_close+0x18d/0x490 [lmv] [12840.260644] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [12840.263446] ll_close_inode_openhandle+0x493/0xea0 [lustre] [12840.266670] ll_md_real_close+0xa5/0x2a0 [lustre] [12840.269684] ll_md_close+0x274/0x800 [lustre] [12840.272526] ll_file_release+0x128/0xc50 [lustre] [12840.275602] __fput+0x10d/0x440 [12840.277233] ____fput+0x12/0x20 [12840.279129] task_work_run+0xc6/0x110 [12840.280570] exit_to_usermode_loop+0x1dd/0x1f0 [12840.290201] do_syscall_64+0x3d6/0x440 [12840.291340] entry_SYSCALL_64_after_hwframe+0x49/0xae [12840.301775] RIP: 0033:0x7ff54af45cd8 [12840.307055] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [12840.318024] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [12840.322252] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [12840.323748] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [12840.325159] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [12840.327317] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [12840.330929] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12852.056892] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [12852.073353] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12852.137111] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12852.153397] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12852.165667] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [12852.169398] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [12852.208467] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12852.217112] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [12852.230432] RDX: 0000000000002990 RSI: 0000000000000003 RDI: ffff96848811cae0 [12852.247169] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [12852.261067] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [12852.270441] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12852.274426] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [12852.287841] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12852.297299] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [12852.311566] Call Trace: [12852.312701] [12852.317653] ? show_regs.cold.9+0x22/0x2f [12852.321376] ? watchdog_timer_fn.cold.6+0x62/0x10a [12852.329547] ? __hrtimer_run_queues+0x1c2/0x4d0 [12852.339250] ? watchdog+0x40/0x40 [12852.341748] ? hrtimer_interrupt+0x128/0x2c0 [12852.346167] ? smp_apic_timer_interrupt+0xa6/0x270 [12852.356416] ? apic_timer_interrupt+0xf/0x20 [12852.362843] [12852.365331] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [12852.373532] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12852.380792] do_raw_spin_lock+0x177/0x190 [12852.384437] _raw_spin_lock+0x21/0x30 [12852.389105] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12852.392742] obd_mod_free+0x46/0x140 [ptlrpc] [12852.400450] mdc_commit_open+0x91/0xb0 [mdc] [12852.403612] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12852.410441] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12852.420419] after_reply+0xe2c/0x1780 [ptlrpc] [12852.424838] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12852.428483] ptlrpcd+0xa82/0xe40 [ptlrpc] [12852.438406] ? do_wait_intr+0xf0/0xf0 [12852.439594] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12852.446384] kthread+0x1d1/0x200 [12852.449818] ? set_kthread_struct+0x70/0x70 [12852.455267] ret_from_fork+0x1f/0x30 [12856.059206] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [12856.060020] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [12856.068751] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12856.068829] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12856.068836] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12856.068842] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12856.070296] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12856.070313] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12856.070317] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12856.070321] RAX: 0000000000002d0e RBX: ffff96848811cae0 RCX: 0000000000000001 [12856.070324] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [12856.070326] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [12856.070328] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [12856.070331] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [12856.070334] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [12856.070336] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12856.070338] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [12856.070347] Call Trace: [12856.070352] [12856.070358] ? show_regs.cold.9+0x22/0x2f [12856.070367] ? watchdog_timer_fn.cold.6+0x62/0x10a [12856.070374] ? __hrtimer_run_queues+0x1c2/0x4d0 [12856.070378] ? watchdog+0x40/0x40 [12856.070382] ? hrtimer_interrupt+0x128/0x2c0 [12856.070385] ? smp_apic_timer_interrupt+0xa6/0x270 [12856.070394] ? apic_timer_interrupt+0xf/0x20 [12856.070398] [12856.070401] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12856.070769] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12856.070777] do_raw_spin_lock+0x177/0x190 [12856.070782] _raw_spin_lock+0x21/0x30 [12856.070787] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12856.071047] process_one_work+0x2c8/0x700 [12856.071877] worker_thread+0x296/0x6e0 [12856.071889] ? rescuer_thread+0x570/0x570 [12856.071892] kthread+0x1d1/0x200 [12856.071897] ? set_kthread_struct+0x70/0x70 [12856.071900] ret_from_fork+0x1f/0x30 [12856.262265] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12856.283100] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12856.288980] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12856.294066] RIP: 0010:native_safe_halt+0x2e/0x50 [12856.296412] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12856.304969] RSP: 0000:ffffaf9fc15234f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12856.308436] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [12856.312792] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c21b4154 [12856.319098] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [12856.322860] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [12856.327895] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [12856.333840] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [12856.338355] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12856.341922] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [12856.344931] Call Trace: [12856.345885] [12856.346842] ? show_regs.cold.9+0x22/0x2f [12856.348392] ? watchdog_timer_fn.cold.6+0x62/0x10a [12856.350704] ? __hrtimer_run_queues+0x1c2/0x4d0 [12856.352426] ? watchdog+0x40/0x40 [12856.354445] ? hrtimer_interrupt+0x128/0x2c0 [12856.356277] ? smp_apic_timer_interrupt+0xa6/0x270 [12856.358345] ? apic_timer_interrupt+0xf/0x20 [12856.360148] [12856.361265] ? native_safe_halt+0x2e/0x50 [12856.362794] kvm_wait+0xcd/0xe0 [12856.364217] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12856.367171] do_raw_spin_lock+0x177/0x190 [12856.368887] _raw_spin_lock+0x21/0x30 [12856.370662] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12856.373815] ? do_raw_spin_unlock+0x75/0x190 [12856.376201] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12856.379182] ? do_wait_intr+0xf0/0xf0 [12856.380629] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12856.383715] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [12856.388869] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12856.393805] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12856.397054] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12856.401291] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [12856.407370] mdc_intent_lock+0x353/0x7e0 [mdc] [12856.415425] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12856.418033] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12856.421264] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12856.424327] lmv_intent_open+0x338/0x1150 [lmv] [12856.427022] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12856.430535] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12856.433375] lmv_intent_lock+0x1dd/0x4c0 [lmv] [12856.436158] ll_intent_lock+0xc5/0xaa0 [lustre] [12856.438449] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12856.442890] ll_lookup_it+0xe0e/0x2270 [lustre] [12856.447307] ? do_raw_spin_unlock+0x75/0x190 [12856.449921] ? _raw_spin_unlock+0x12/0x30 [12856.451932] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [12856.455022] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [12856.457417] ll_atomic_open+0x36a/0xd70 [lustre] [12856.461511] lookup_open+0x356/0xb40 [12856.465412] do_last+0x313/0xfc0 [12856.467916] ? nd_jump_root+0xe5/0x160 [12856.470020] ? path_init+0x437/0x520 [12856.472673] path_openat+0xf7/0x500 [12856.474138] ? do_raw_spin_unlock+0x75/0x190 [12856.476983] do_filp_open+0x99/0x140 [12856.480814] ? getname_flags+0x6e/0x330 [12856.483249] ? __check_object_size+0xff/0x256 [12856.490534] ? do_raw_spin_unlock+0x75/0x190 [12856.495098] ? _raw_spin_unlock+0x12/0x30 [12856.497633] do_sys_openat2+0x2b4/0x410 [12856.499053] do_sys_open+0x73/0xa0 [12856.500946] __x64_sys_openat+0x24/0x30 [12856.505169] do_syscall_64+0xc1/0x440 [12856.507369] entry_SYSCALL_64_after_hwframe+0x49/0xae [12856.512267] RIP: 0033:0x7ff54af45332 [12856.516034] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [12856.525804] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [12856.530369] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [12856.535226] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [12856.539867] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [12856.543565] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [12856.548229] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [12856.802022] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10897s! [12856.806082] Showing busy workqueues and worker pools: [12856.809537] workqueue events: flags=0x0 [12856.810620] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12856.813686] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [12856.820903] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [12856.826318] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [12856.831898] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12856.834946] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [12856.839295] workqueue events_long: flags=0x0 [12856.841365] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12856.844381] pending: lu_objects_flush [obdclass] [12856.846908] workqueue events_unbound: flags=0x2 [12856.848989] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [12856.852027] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [12856.864665] workqueue events_power_efficient: flags=0x80 [12856.866659] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12856.878720] pending: neigh_periodic_work [12856.880911] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12856.885890] pending: do_cache_clean, check_lifetime [12856.888490] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12856.892215] pending: neigh_periodic_work [12856.897014] workqueue writeback: flags=0x4a [12856.899383] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [12856.906030] pending: wb_workfn, wb_workfn [12856.908892] workqueue xprtiod: flags=0xa [12856.914221] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [12856.919938] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [12856.926375] workqueue ipv6_addrconf: flags=0x40008 [12856.932446] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12856.940488] pending: addrconf_verify_work [12856.943517] workqueue knbd0-recv: flags=0x1a [12856.947979] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12856.953390] in-flight: 657:recv_work [12856.957525] workqueue ptlrpc_pinger: flags=0x2 [12856.959143] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12856.965885] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [12856.969729] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10898s workers=4 idle: 20831 12642 28859 [12856.975464] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12847s workers=2 idle: 659 [12868.061151] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [12868.066635] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12868.109728] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12868.113980] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12868.119013] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12868.123822] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12868.143325] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12868.152721] RAX: 0000000000004321 RBX: ffff96848811cae0 RCX: 0000000000000001 [12868.158698] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [12868.169869] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [12868.179115] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [12868.185036] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [12868.192530] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [12868.204519] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12868.208590] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [12868.216137] Call Trace: [12868.216777] [12868.217315] ? show_regs.cold.9+0x22/0x2f [12868.225045] ? watchdog_timer_fn.cold.6+0x62/0x10a [12868.229883] ? __hrtimer_run_queues+0x1c2/0x4d0 [12868.238703] ? watchdog+0x40/0x40 [12868.241415] ? hrtimer_interrupt+0x128/0x2c0 [12868.245914] ? smp_apic_timer_interrupt+0xa6/0x270 [12868.252758] ? apic_timer_interrupt+0xf/0x20 [12868.256564] [12868.259530] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12868.268594] do_raw_spin_lock+0x177/0x190 [12868.274293] _raw_spin_lock+0x21/0x30 [12868.277017] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12868.283689] ? do_raw_spin_unlock+0x75/0x190 [12868.287162] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12868.291871] ? do_wait_intr+0xf0/0xf0 [12868.296911] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12868.305478] mdc_close+0x53b/0xd70 [mdc] [12868.308857] lmv_close+0x18d/0x490 [lmv] [12868.315086] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [12868.321621] ll_close_inode_openhandle+0x493/0xea0 [lustre] [12868.324307] ll_md_real_close+0xa5/0x2a0 [lustre] [12868.327302] ll_md_close+0x274/0x800 [lustre] [12868.337734] ll_file_release+0x128/0xc50 [lustre] [12868.342536] __fput+0x10d/0x440 [12868.347474] ____fput+0x12/0x20 [12868.348365] task_work_run+0xc6/0x110 [12868.349434] exit_to_usermode_loop+0x1dd/0x1f0 [12868.360538] do_syscall_64+0x3d6/0x440 [12868.361750] entry_SYSCALL_64_after_hwframe+0x49/0xae [12868.368254] RIP: 0033:0x7ff54af45cd8 [12868.370773] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [12868.383789] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [12868.394854] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [12868.396998] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [12868.406837] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [12868.414660] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [12868.416973] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12884.055024] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [12884.058627] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12884.061499] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [12884.061572] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [12884.061578] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12884.061653] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12884.061660] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12884.061666] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12884.062054] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12884.062066] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12884.062070] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12884.062074] RAX: 0000000000007a37 RBX: ffff96848811cae0 RCX: 0000000000000001 [12884.062077] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [12884.062079] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [12884.062081] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [12884.062084] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [12884.062087] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [12884.062090] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12884.062093] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [12884.062102] Call Trace: [12884.062107] [12884.062111] ? show_regs.cold.9+0x22/0x2f [12884.062120] ? watchdog_timer_fn.cold.6+0x62/0x10a [12884.062126] ? __hrtimer_run_queues+0x1c2/0x4d0 [12884.062130] ? watchdog+0x40/0x40 [12884.062134] ? hrtimer_interrupt+0x128/0x2c0 [12884.062138] ? smp_apic_timer_interrupt+0xa6/0x270 [12884.062146] ? apic_timer_interrupt+0xf/0x20 [12884.062150] [12884.062153] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12884.062158] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12884.062162] do_raw_spin_lock+0x177/0x190 [12884.062168] _raw_spin_lock+0x21/0x30 [12884.062173] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12884.062389] process_one_work+0x2c8/0x700 [12884.062396] worker_thread+0x296/0x6e0 [12884.062400] ? rescuer_thread+0x570/0x570 [12884.062404] kthread+0x1d1/0x200 [12884.062409] ? set_kthread_struct+0x70/0x70 [12884.062413] ret_from_fork+0x1f/0x30 [12884.076137] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12884.076163] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12884.076167] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [12884.076181] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [12884.076186] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12884.076192] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [12884.076194] RDX: 00000000000017ac RSI: 0000000000000003 RDI: ffff96848811cae0 [12884.076196] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [12884.076198] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [12884.076201] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12884.076204] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [12884.076207] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12884.076209] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [12884.076216] Call Trace: [12884.076221] [12884.076226] ? show_regs.cold.9+0x22/0x2f [12884.076236] ? watchdog_timer_fn.cold.6+0x62/0x10a [12884.076242] ? __hrtimer_run_queues+0x1c2/0x4d0 [12884.076246] ? watchdog+0x40/0x40 [12884.076249] ? hrtimer_interrupt+0x128/0x2c0 [12884.076253] ? smp_apic_timer_interrupt+0xa6/0x270 [12884.076261] ? apic_timer_interrupt+0xf/0x20 [12884.076265] [12884.076268] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [12884.076272] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12884.076337] do_raw_spin_lock+0x177/0x190 [12884.076342] _raw_spin_lock+0x21/0x30 [12884.076348] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12884.076638] obd_mod_free+0x46/0x140 [ptlrpc] [12884.077267] mdc_commit_open+0x91/0xb0 [mdc] [12884.077371] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12884.077594] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12884.077808] after_reply+0xe2c/0x1780 [ptlrpc] [12884.083374] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12884.083458] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12884.083465] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12884.083468] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12884.083481] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12884.083485] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12884.083489] RAX: 0000000000007f8d RBX: ffff96848811cae0 RCX: 0000000000000001 [12884.083492] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [12884.083494] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [12884.083496] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [12884.083499] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [12884.083502] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [12884.083504] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12884.083506] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [12884.083514] Call Trace: [12884.083520] [12884.083525] ? show_regs.cold.9+0x22/0x2f [12884.083535] ? watchdog_timer_fn.cold.6+0x62/0x10a [12884.083541] ? __hrtimer_run_queues+0x1c2/0x4d0 [12884.083545] ? watchdog+0x40/0x40 [12884.083549] ? hrtimer_interrupt+0x128/0x2c0 [12884.083553] ? smp_apic_timer_interrupt+0xa6/0x270 [12884.083561] ? apic_timer_interrupt+0xf/0x20 [12884.083565] [12884.083569] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12884.083575] do_raw_spin_lock+0x177/0x190 [12884.083580] _raw_spin_lock+0x21/0x30 [12884.083586] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12884.083970] ? do_raw_spin_unlock+0x75/0x190 [12884.083978] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12884.084189] ? do_wait_intr+0xf0/0xf0 [12884.084198] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12884.084477] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [12884.086942] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12884.087202] ptlrpcd+0xa82/0xe40 [ptlrpc] [12884.087408] ? do_wait_intr+0xf0/0xf0 [12884.087419] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12884.087643] kthread+0x1d1/0x200 [12884.087650] ? set_kthread_struct+0x70/0x70 [12884.087654] ret_from_fork+0x1f/0x30 [12884.789785] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12884.799892] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12884.805722] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12884.813740] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [12884.816661] mdc_intent_lock+0x353/0x7e0 [mdc] [12884.822850] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12884.824748] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12884.830785] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12884.832265] lmv_intent_open+0x338/0x1150 [lmv] [12884.835242] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12884.839766] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12884.844014] lmv_intent_lock+0x1dd/0x4c0 [lmv] [12884.850358] ll_intent_lock+0xc5/0xaa0 [lustre] [12884.859930] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12884.867806] ll_lookup_it+0xe0e/0x2270 [lustre] [12884.872531] ? do_raw_spin_unlock+0x75/0x190 [12884.875681] ? _raw_spin_unlock+0x12/0x30 [12884.880489] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [12884.887844] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [12884.895876] ll_atomic_open+0x36a/0xd70 [lustre] [12884.900052] lookup_open+0x356/0xb40 [12884.905714] do_last+0x313/0xfc0 [12884.911658] ? nd_jump_root+0xe5/0x160 [12884.915776] ? path_init+0x437/0x520 [12884.928124] path_openat+0xf7/0x500 [12884.929658] ? do_raw_spin_unlock+0x75/0x190 [12884.936155] do_filp_open+0x99/0x140 [12884.938877] ? getname_flags+0x6e/0x330 [12884.944899] ? __check_object_size+0xff/0x256 [12884.950261] ? do_raw_spin_unlock+0x75/0x190 [12884.952913] ? _raw_spin_unlock+0x12/0x30 [12884.954837] do_sys_openat2+0x2b4/0x410 [12884.958789] do_sys_open+0x73/0xa0 [12884.960602] __x64_sys_openat+0x24/0x30 [12884.964530] do_syscall_64+0xc1/0x440 [12884.965945] entry_SYSCALL_64_after_hwframe+0x49/0xae [12884.970283] RIP: 0033:0x7ff54af45332 [12884.971938] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [12884.989261] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [12885.002039] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [12885.016029] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [12885.022105] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [12885.029909] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [12885.041547] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [12887.010872] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10928s! [12887.013734] Showing busy workqueues and worker pools: [12887.021913] workqueue events: flags=0x0 [12887.024449] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12887.030765] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [12887.038043] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [12887.043436] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [12887.059067] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12887.065575] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [12887.074493] workqueue events_long: flags=0x0 [12887.076952] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12887.081810] pending: lu_objects_flush [obdclass] [12887.085495] workqueue events_unbound: flags=0x2 [12887.089866] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [12887.092705] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [12887.101348] workqueue events_power_efficient: flags=0x80 [12887.105402] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12887.109430] pending: neigh_periodic_work [12887.112823] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12887.116291] pending: do_cache_clean, check_lifetime [12887.119031] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12887.122233] pending: neigh_periodic_work [12887.124893] workqueue writeback: flags=0x4a [12887.128716] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [12887.138736] pending: wb_workfn, wb_workfn [12887.144020] workqueue xprtiod: flags=0xa [12887.147466] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [12887.152823] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [12887.158123] workqueue ipv6_addrconf: flags=0x40008 [12887.163364] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12887.167874] pending: addrconf_verify_work [12887.170486] workqueue knbd0-recv: flags=0x1a [12887.172991] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12887.178894] in-flight: 657:recv_work [12887.183045] workqueue ptlrpc_pinger: flags=0x2 [12887.192530] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12887.202989] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [12887.213587] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10928s workers=4 idle: 20831 12642 28859 [12887.223199] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12877s workers=2 idle: 659 [12896.058706] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [12896.069459] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12896.119600] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12896.131415] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12896.146481] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12896.153598] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12896.171210] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12896.175733] RAX: 000000000000536a RBX: ffff96848811cae0 RCX: 0000000000000001 [12896.182711] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [12896.191929] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [12896.197374] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [12896.202535] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [12896.209220] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [12896.215280] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12896.220792] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [12896.225951] Call Trace: [12896.228484] [12896.230555] ? show_regs.cold.9+0x22/0x2f [12896.232722] ? watchdog_timer_fn.cold.6+0x62/0x10a [12896.239736] ? __hrtimer_run_queues+0x1c2/0x4d0 [12896.244075] ? watchdog+0x40/0x40 [12896.246516] ? hrtimer_interrupt+0x128/0x2c0 [12896.249071] ? smp_apic_timer_interrupt+0xa6/0x270 [12896.255283] ? apic_timer_interrupt+0xf/0x20 [12896.261664] [12896.262667] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12896.268074] do_raw_spin_lock+0x177/0x190 [12896.272630] _raw_spin_lock+0x21/0x30 [12896.277258] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12896.286165] ? do_raw_spin_unlock+0x75/0x190 [12896.288670] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12896.291852] ? do_wait_intr+0xf0/0xf0 [12896.295152] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12896.298611] mdc_close+0x53b/0xd70 [mdc] [12896.300571] lmv_close+0x18d/0x490 [lmv] [12896.302621] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [12896.306951] ll_close_inode_openhandle+0x493/0xea0 [lustre] [12896.309458] ll_md_real_close+0xa5/0x2a0 [lustre] [12896.311669] ll_md_close+0x274/0x800 [lustre] [12896.313422] ll_file_release+0x128/0xc50 [lustre] [12896.317618] __fput+0x10d/0x440 [12896.320956] ____fput+0x12/0x20 [12896.323677] task_work_run+0xc6/0x110 [12896.325008] exit_to_usermode_loop+0x1dd/0x1f0 [12896.329680] do_syscall_64+0x3d6/0x440 [12896.331695] entry_SYSCALL_64_after_hwframe+0x49/0xae [12896.338938] RIP: 0033:0x7ff54af45cd8 [12896.340655] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [12896.353697] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [12896.358475] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [12896.361718] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [12896.365722] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [12896.369588] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [12896.373391] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12912.056686] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [12912.063343] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [12912.064134] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [12912.064142] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12912.064405] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12912.064411] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12912.064414] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12912.064441] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12912.064445] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12912.064449] RAX: 0000000000007a94 RBX: ffff96848811cae0 RCX: 0000000000000001 [12912.064451] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [12912.064453] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [12912.064455] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [12912.064458] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [12912.064461] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [12912.064463] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12912.064465] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [12912.064473] Call Trace: [12912.064490] [12912.064497] ? show_regs.cold.9+0x22/0x2f [12912.064509] ? watchdog_timer_fn.cold.6+0x62/0x10a [12912.064517] ? __hrtimer_run_queues+0x1c2/0x4d0 [12912.064521] ? watchdog+0x40/0x40 [12912.064525] ? hrtimer_interrupt+0x128/0x2c0 [12912.064529] ? smp_apic_timer_interrupt+0xa6/0x270 [12912.064540] ? apic_timer_interrupt+0xf/0x20 [12912.064545] [12912.064548] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12912.064555] do_raw_spin_lock+0x177/0x190 [12912.064559] _raw_spin_lock+0x21/0x30 [12912.064565] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12912.066001] ? do_raw_spin_unlock+0x75/0x190 [12912.066008] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12912.066399] ? do_wait_intr+0xf0/0xf0 [12912.066408] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12912.067096] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12912.067169] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12912.067176] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12912.067191] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [12912.067202] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [12912.067205] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12912.067210] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [12912.067213] RDX: 0000000000005243 RSI: 0000000000000003 RDI: ffff96848811cae0 [12912.067215] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [12912.067217] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [12912.067220] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12912.067223] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [12912.067226] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12912.067229] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [12912.067237] Call Trace: [12912.067242] [12912.067245] ? show_regs.cold.9+0x22/0x2f [12912.067254] ? watchdog_timer_fn.cold.6+0x62/0x10a [12912.067259] ? __hrtimer_run_queues+0x1c2/0x4d0 [12912.067264] ? watchdog+0x40/0x40 [12912.067265] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [12912.071708] ? hrtimer_interrupt+0x128/0x2c0 [12912.071716] ? smp_apic_timer_interrupt+0xa6/0x270 [12912.071723] ? apic_timer_interrupt+0xf/0x20 [12912.071728] [12912.071733] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [12912.071740] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12912.071746] do_raw_spin_lock+0x177/0x190 [12912.071753] _raw_spin_lock+0x21/0x30 [12912.071760] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12912.072065] obd_mod_free+0x46/0x140 [ptlrpc] [12912.073727] mdc_commit_open+0x91/0xb0 [mdc] [12912.073771] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12912.074321] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12912.074434] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12912.074438] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12912.074444] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12912.075007] RIP: 0010:native_safe_halt+0x2e/0x50 [12912.075018] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12912.075022] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12912.075026] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [12912.075028] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [12912.075030] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [12912.075032] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [12912.075034] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [12912.075037] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [12912.075040] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12912.075042] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [12912.075051] Call Trace: [12912.075055] [12912.075059] ? show_regs.cold.9+0x22/0x2f [12912.075080] ? watchdog_timer_fn.cold.6+0x62/0x10a [12912.075086] ? __hrtimer_run_queues+0x1c2/0x4d0 [12912.075090] ? watchdog+0x40/0x40 [12912.075094] ? hrtimer_interrupt+0x128/0x2c0 [12912.075098] ? smp_apic_timer_interrupt+0xa6/0x270 [12912.075105] ? apic_timer_interrupt+0xf/0x20 [12912.075109] [12912.075112] ? native_safe_halt+0x2e/0x50 [12912.075116] kvm_wait+0xcd/0xe0 [12912.075125] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12912.075133] do_raw_spin_lock+0x177/0x190 [12912.075137] _raw_spin_lock+0x21/0x30 [12912.075141] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12912.075561] process_one_work+0x2c8/0x700 [12912.075570] worker_thread+0x296/0x6e0 [12912.075574] ? rescuer_thread+0x570/0x570 [12912.075578] kthread+0x1d1/0x200 [12912.075582] ? set_kthread_struct+0x70/0x70 [12912.075586] ret_from_fork+0x1f/0x30 [12912.073989] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12912.076629] after_reply+0xe2c/0x1780 [ptlrpc] [12912.076830] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12912.091866] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12912.162599] ptlrpcd+0xa82/0xe40 [ptlrpc] [12912.162925] ? do_wait_intr+0xf0/0xf0 [12912.162938] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12912.176976] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12912.177380] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12912.177428] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [12912.177463] mdc_intent_lock+0x353/0x7e0 [mdc] [12912.177494] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12912.177583] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12912.178278] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12912.178313] lmv_intent_open+0x338/0x1150 [lmv] [12912.178347] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12912.178446] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12912.178538] lmv_intent_lock+0x1dd/0x4c0 [lmv] [12912.178560] ll_intent_lock+0xc5/0xaa0 [lustre] [12912.178937] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12912.179394] ll_lookup_it+0xe0e/0x2270 [lustre] [12912.179479] ? do_raw_spin_unlock+0x75/0x190 [12912.179488] ? _raw_spin_unlock+0x12/0x30 [12912.179495] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [12912.179525] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [12912.179543] ll_atomic_open+0x36a/0xd70 [lustre] [12912.180292] lookup_open+0x356/0xb40 [12912.180303] do_last+0x313/0xfc0 [12912.180307] ? nd_jump_root+0xe5/0x160 [12912.180312] ? path_init+0x437/0x520 [12912.180316] path_openat+0xf7/0x500 [12912.180321] ? do_raw_spin_unlock+0x75/0x190 [12912.180325] do_filp_open+0x99/0x140 [12912.180331] ? getname_flags+0x6e/0x330 [12912.180335] ? __check_object_size+0xff/0x256 [12912.180343] ? do_raw_spin_unlock+0x75/0x190 [12912.180346] ? _raw_spin_unlock+0x12/0x30 [12912.180351] do_sys_openat2+0x2b4/0x410 [12912.180355] do_sys_open+0x73/0xa0 [12912.180360] __x64_sys_openat+0x24/0x30 [12912.180363] do_syscall_64+0xc1/0x440 [12912.180370] entry_SYSCALL_64_after_hwframe+0x49/0xae [12912.180377] RIP: 0033:0x7ff54af45332 [12912.180406] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [12912.180411] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [12912.180416] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [12912.180418] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [12912.180421] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [12912.180423] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [12912.180425] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [12912.900773] kthread+0x1d1/0x200 [12912.904644] ? set_kthread_struct+0x70/0x70 [12912.907773] ret_from_fork+0x1f/0x30 [12917.735069] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10958s! [12917.744076] Showing busy workqueues and worker pools: [12917.750656] workqueue events: flags=0x0 [12917.755054] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12917.763068] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [12917.768696] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [12917.772397] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [12917.779622] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12917.788725] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [12917.791263] workqueue events_long: flags=0x0 [12917.796196] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12917.802481] pending: lu_objects_flush [obdclass] [12917.808761] workqueue events_unbound: flags=0x2 [12917.813501] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [12917.819694] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [12917.829196] workqueue events_power_efficient: flags=0x80 [12917.832466] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12917.836068] pending: neigh_periodic_work [12917.838347] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12917.841598] pending: do_cache_clean, check_lifetime [12917.845400] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12917.852409] pending: neigh_periodic_work [12917.854489] workqueue writeback: flags=0x4a [12917.855825] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [12917.858078] pending: wb_workfn, wb_workfn [12917.862395] workqueue xprtiod: flags=0xa [12917.864945] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [12917.868279] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [12917.872737] workqueue ipv6_addrconf: flags=0x40008 [12917.875185] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12917.877630] pending: addrconf_verify_work [12917.879246] workqueue knbd0-recv: flags=0x1a [12917.882225] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12917.884421] in-flight: 657:recv_work [12917.887472] workqueue ptlrpc_pinger: flags=0x2 [12917.890261] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12917.893242] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [12917.896135] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10959s workers=4 idle: 20831 12642 28859 [12917.901308] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12908s workers=2 idle: 659 [12924.058633] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [12924.063399] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12924.103461] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12924.113453] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12924.124475] RIP: 0010:native_safe_halt+0x2e/0x50 [12924.135416] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12924.157126] RSP: 0018:ffffaf9fc1543ae0 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [12924.166371] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [12924.168691] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c20b4154 [12924.177510] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [12924.184871] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [12924.193817] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [12924.201350] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [12924.214153] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12924.224665] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [12924.238103] Call Trace: [12924.241248] [12924.243905] ? show_regs.cold.9+0x22/0x2f [12924.254282] ? watchdog_timer_fn.cold.6+0x62/0x10a [12924.263044] ? __hrtimer_run_queues+0x1c2/0x4d0 [12924.266992] ? watchdog+0x40/0x40 [12924.269633] ? hrtimer_interrupt+0x128/0x2c0 [12924.273230] ? smp_apic_timer_interrupt+0xa6/0x270 [12924.281225] ? apic_timer_interrupt+0xf/0x20 [12924.288262] [12924.291670] ? native_safe_halt+0x2e/0x50 [12924.298755] kvm_wait+0xcd/0xe0 [12924.300875] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12924.308863] do_raw_spin_lock+0x177/0x190 [12924.315197] _raw_spin_lock+0x21/0x30 [12924.319541] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12924.326224] ? do_raw_spin_unlock+0x75/0x190 [12924.333363] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12924.344431] ? do_wait_intr+0xf0/0xf0 [12924.345819] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12924.360526] mdc_close+0x53b/0xd70 [mdc] [12924.361970] lmv_close+0x18d/0x490 [lmv] [12924.369907] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [12924.378967] ll_close_inode_openhandle+0x493/0xea0 [lustre] [12924.384778] ll_md_real_close+0xa5/0x2a0 [lustre] [12924.387117] ll_md_close+0x274/0x800 [lustre] [12924.392556] ll_file_release+0x128/0xc50 [lustre] [12924.401030] __fput+0x10d/0x440 [12924.404199] ____fput+0x12/0x20 [12924.406872] task_work_run+0xc6/0x110 [12924.409806] exit_to_usermode_loop+0x1dd/0x1f0 [12924.412121] do_syscall_64+0x3d6/0x440 [12924.415164] entry_SYSCALL_64_after_hwframe+0x49/0xae [12924.418765] RIP: 0033:0x7ff54af45cd8 [12924.421227] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [12924.433611] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [12924.439961] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [12924.444187] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [12924.451178] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [12924.456822] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [12924.463494] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12940.061274] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [12940.063323] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [12940.071952] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12940.072036] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12940.072043] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12940.072051] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12940.072613] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12940.072627] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12940.072632] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [12940.072637] RAX: 0000000000006e3d RBX: ffff96848811cae0 RCX: 0000000000000001 [12940.072640] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [12940.072642] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [12940.072644] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [12940.072647] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [12940.072651] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [12940.072654] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12940.072657] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [12940.072666] Call Trace: [12940.072673] [12940.072679] ? show_regs.cold.9+0x22/0x2f [12940.072689] ? watchdog_timer_fn.cold.6+0x62/0x10a [12940.072697] ? __hrtimer_run_queues+0x1c2/0x4d0 [12940.072701] ? watchdog+0x40/0x40 [12940.072705] ? hrtimer_interrupt+0x128/0x2c0 [12940.072709] ? smp_apic_timer_interrupt+0xa6/0x270 [12940.072718] ? apic_timer_interrupt+0xf/0x20 [12940.072723] [12940.072726] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12940.072828] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12940.072835] do_raw_spin_lock+0x177/0x190 [12940.072839] _raw_spin_lock+0x21/0x30 [12940.072844] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12940.073216] process_one_work+0x2c8/0x700 [12940.073223] worker_thread+0x296/0x6e0 [12940.073228] ? rescuer_thread+0x570/0x570 [12940.073231] kthread+0x1d1/0x200 [12940.073236] ? set_kthread_struct+0x70/0x70 [12940.073240] ret_from_fork+0x1f/0x30 [12940.348986] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12940.408162] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12940.420809] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12940.431349] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12940.434314] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12940.442586] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [12940.451615] RAX: 0000000000005103 RBX: ffff96848811cae0 RCX: 0000000000000001 [12940.455702] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [12940.458557] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [12940.462403] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [12940.465949] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [12940.469551] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [12940.473595] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12940.477615] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [12940.481330] Call Trace: [12940.483111] [12940.483825] ? show_regs.cold.9+0x22/0x2f [12940.488715] ? watchdog_timer_fn.cold.6+0x62/0x10a [12940.491142] ? __hrtimer_run_queues+0x1c2/0x4d0 [12940.493446] ? watchdog+0x40/0x40 [12940.494831] ? hrtimer_interrupt+0x128/0x2c0 [12940.496927] ? smp_apic_timer_interrupt+0xa6/0x270 [12940.499055] ? apic_timer_interrupt+0xf/0x20 [12940.500949] [12940.501686] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12940.504310] do_raw_spin_lock+0x177/0x190 [12940.506694] _raw_spin_lock+0x21/0x30 [12940.508348] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12940.511623] ? do_raw_spin_unlock+0x75/0x190 [12940.515410] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12940.519506] ? do_wait_intr+0xf0/0xf0 [12940.521689] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12940.528246] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [12940.533596] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12940.540278] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12940.543280] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12940.548800] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [12940.550659] mdc_intent_lock+0x353/0x7e0 [mdc] [12940.554197] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12940.556678] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12940.561404] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12940.563920] lmv_intent_open+0x338/0x1150 [lmv] [12940.566208] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12940.568654] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12940.574646] lmv_intent_lock+0x1dd/0x4c0 [lmv] [12940.579013] ll_intent_lock+0xc5/0xaa0 [lustre] [12940.581144] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12940.583274] ll_lookup_it+0xe0e/0x2270 [lustre] [12940.585283] ? do_raw_spin_unlock+0x75/0x190 [12940.587069] ? _raw_spin_unlock+0x12/0x30 [12940.588895] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [12940.591053] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [12940.593246] ll_atomic_open+0x36a/0xd70 [lustre] [12940.595413] lookup_open+0x356/0xb40 [12940.596868] do_last+0x313/0xfc0 [12940.598401] ? nd_jump_root+0xe5/0x160 [12940.599944] ? path_init+0x437/0x520 [12940.605048] path_openat+0xf7/0x500 [12940.611092] ? do_raw_spin_unlock+0x75/0x190 [12940.613568] do_filp_open+0x99/0x140 [12940.615085] ? getname_flags+0x6e/0x330 [12940.616958] ? __check_object_size+0xff/0x256 [12940.619467] ? do_raw_spin_unlock+0x75/0x190 [12940.620706] ? _raw_spin_unlock+0x12/0x30 [12940.622186] do_sys_openat2+0x2b4/0x410 [12940.623437] do_sys_open+0x73/0xa0 [12940.624381] __x64_sys_openat+0x24/0x30 [12940.625794] do_syscall_64+0xc1/0x440 [12940.628652] entry_SYSCALL_64_after_hwframe+0x49/0xae [12940.638240] RIP: 0033:0x7ff54af45332 [12940.642997] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [12940.656925] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [12940.666079] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [12940.674413] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [12940.681305] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [12940.695243] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [12940.706831] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [12944.055022] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [12944.056733] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12944.077800] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12944.085840] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12944.092813] RIP: 0010:native_safe_halt+0x2e/0x50 [12944.096323] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12944.109514] RSP: 0000:ffffaf9fc1b97bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12944.113481] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [12944.119805] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff96848811cae0 [12944.124462] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [12944.129882] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [12944.131805] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12944.134275] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [12944.138921] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12944.141344] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [12944.147453] Call Trace: [12944.149852] [12944.150889] ? show_regs.cold.9+0x22/0x2f [12944.153103] ? watchdog_timer_fn.cold.6+0x62/0x10a [12944.156628] ? __hrtimer_run_queues+0x1c2/0x4d0 [12944.158418] ? watchdog+0x40/0x40 [12944.161773] ? hrtimer_interrupt+0x128/0x2c0 [12944.166531] ? smp_apic_timer_interrupt+0xa6/0x270 [12944.167589] ? apic_timer_interrupt+0xf/0x20 [12944.169727] [12944.170405] ? native_safe_halt+0x2e/0x50 [12944.171676] kvm_wait+0xcd/0xe0 [12944.174208] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12944.179938] do_raw_spin_lock+0x177/0x190 [12944.181592] _raw_spin_lock+0x21/0x30 [12944.184896] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12944.188527] obd_mod_free+0x46/0x140 [ptlrpc] [12944.193280] mdc_commit_open+0x91/0xb0 [mdc] [12944.196364] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12944.200019] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12944.203654] after_reply+0xe2c/0x1780 [ptlrpc] [12944.207332] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12944.211241] ptlrpcd+0xa82/0xe40 [ptlrpc] [12944.215321] ? do_wait_intr+0xf0/0xf0 [12944.216863] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12944.220898] kthread+0x1d1/0x200 [12944.224472] ? set_kthread_struct+0x70/0x70 [12944.227660] ret_from_fork+0x1f/0x30 [12947.935243] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10989s! [12947.940034] Showing busy workqueues and worker pools: [12947.942987] workqueue events: flags=0x0 [12947.945104] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12947.948515] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [12947.957509] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [12947.962345] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [12947.970419] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12947.980433] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [12947.988707] workqueue events_long: flags=0x0 [12947.990948] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12948.001114] pending: lu_objects_flush [obdclass] [12948.014759] workqueue events_unbound: flags=0x2 [12948.028527] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [12948.036614] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [12948.054507] workqueue events_power_efficient: flags=0x80 [12948.058119] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12948.084449] pending: neigh_periodic_work [12948.086888] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12948.095672] pending: do_cache_clean, check_lifetime [12948.107450] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12948.124424] pending: neigh_periodic_work [12948.131052] workqueue writeback: flags=0x4a [12948.136416] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [12948.138625] pending: wb_workfn, wb_workfn [12948.147526] workqueue xprtiod: flags=0xa [12948.150644] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [12948.159133] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [12948.162771] workqueue ipv6_addrconf: flags=0x40008 [12948.166799] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12948.171967] pending: addrconf_verify_work [12948.175824] workqueue knbd0-recv: flags=0x1a [12948.177770] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12948.184315] in-flight: 657:recv_work [12948.188295] workqueue ptlrpc_pinger: flags=0x2 [12948.192526] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12948.202945] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [12948.206787] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10989s workers=4 idle: 20831 12642 28859 [12948.218112] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12938s workers=2 idle: 659 [12952.058623] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [12952.069542] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12952.117181] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12952.125121] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12952.129558] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12952.134529] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12952.152486] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12952.159187] RAX: 0000000000006b87 RBX: ffff96848811cae0 RCX: 0000000000000001 [12952.169262] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [12952.174540] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [12952.178258] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [12952.191520] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [12952.199062] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [12952.204250] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12952.214493] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [12952.220954] Call Trace: [12952.221902] [12952.222604] ? show_regs.cold.9+0x22/0x2f [12952.232029] ? watchdog_timer_fn.cold.6+0x62/0x10a [12952.234158] ? __hrtimer_run_queues+0x1c2/0x4d0 [12952.237393] ? watchdog+0x40/0x40 [12952.239724] ? hrtimer_interrupt+0x128/0x2c0 [12952.243904] ? smp_apic_timer_interrupt+0xa6/0x270 [12952.249969] ? apic_timer_interrupt+0xf/0x20 [12952.256409] [12952.257468] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12952.269302] do_raw_spin_lock+0x177/0x190 [12952.272226] _raw_spin_lock+0x21/0x30 [12952.275783] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12952.281094] ? do_raw_spin_unlock+0x75/0x190 [12952.285092] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12952.288466] ? do_wait_intr+0xf0/0xf0 [12952.292245] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12952.297604] mdc_close+0x53b/0xd70 [mdc] [12952.305098] lmv_close+0x18d/0x490 [lmv] [12952.311219] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [12952.316771] ll_close_inode_openhandle+0x493/0xea0 [lustre] [12952.324583] ll_md_real_close+0xa5/0x2a0 [lustre] [12952.332900] ll_md_close+0x274/0x800 [lustre] [12952.340026] ll_file_release+0x128/0xc50 [lustre] [12952.343230] __fput+0x10d/0x440 [12952.347179] ____fput+0x12/0x20 [12952.349604] task_work_run+0xc6/0x110 [12952.353550] exit_to_usermode_loop+0x1dd/0x1f0 [12952.359181] do_syscall_64+0x3d6/0x440 [12952.363016] entry_SYSCALL_64_after_hwframe+0x49/0xae [12952.366830] RIP: 0033:0x7ff54af45cd8 [12952.374100] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [12952.390422] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [12952.397471] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [12952.404727] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [12952.407251] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [12952.413463] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [12952.418353] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12968.059025] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [12968.060017] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [12968.062173] Modules linked in: lustre(O) osp(O) [12968.074728] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12968.074819] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12968.075365] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12968.078303] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12968.078378] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12968.078385] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12968.078391] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12968.078992] RIP: 0010:native_safe_halt+0x2e/0x50 [12968.079003] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12968.079007] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [12968.079012] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [12968.079014] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c2134154 [12968.079016] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [12968.079018] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [12968.079020] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [12968.079023] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [12968.079026] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12968.079028] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [12968.079036] Call Trace: [12968.079043] [12968.079049] ? show_regs.cold.9+0x22/0x2f [12968.079060] ? watchdog_timer_fn.cold.6+0x62/0x10a [12968.079066] ? __hrtimer_run_queues+0x1c2/0x4d0 [12968.079071] ? watchdog+0x40/0x40 [12968.079075] ? hrtimer_interrupt+0x128/0x2c0 [12968.079079] ? smp_apic_timer_interrupt+0xa6/0x270 [12968.079085] ? apic_timer_interrupt+0xf/0x20 [12968.079090] [12968.079092] ? native_safe_halt+0x2e/0x50 [12968.079096] kvm_wait+0xcd/0xe0 [12968.079103] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12968.079111] do_raw_spin_lock+0x177/0x190 [12968.079115] _raw_spin_lock+0x21/0x30 [12968.079119] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12968.079757] process_one_work+0x2c8/0x700 [12968.079766] worker_thread+0x296/0x6e0 [12968.079771] ? rescuer_thread+0x570/0x570 [12968.079775] kthread+0x1d1/0x200 [12968.079779] ? set_kthread_struct+0x70/0x70 [12968.079783] ret_from_fork+0x1f/0x30 [12968.361760] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12968.368474] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12968.392205] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12968.395628] RAX: 000000000000548b RBX: ffff96848811cae0 RCX: 0000000000000001 [12968.411111] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [12968.420251] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [12968.430520] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [12968.436365] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [12968.440405] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [12968.444524] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12968.451097] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [12968.471355] Call Trace: [12968.477104] [12968.480073] ? show_regs.cold.9+0x22/0x2f [12968.486533] ? watchdog_timer_fn.cold.6+0x62/0x10a [12968.493332] ? __hrtimer_run_queues+0x1c2/0x4d0 [12968.499149] ? watchdog+0x40/0x40 [12968.502193] ? hrtimer_interrupt+0x128/0x2c0 [12968.507249] ? smp_apic_timer_interrupt+0xa6/0x270 [12968.511147] ? apic_timer_interrupt+0xf/0x20 [12968.513401] [12968.514725] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12968.517605] do_raw_spin_lock+0x177/0x190 [12968.519957] _raw_spin_lock+0x21/0x30 [12968.521670] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12968.524450] ? do_raw_spin_unlock+0x75/0x190 [12968.526869] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12968.530416] ? do_wait_intr+0xf0/0xf0 [12968.532765] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12968.535882] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [12968.539352] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12968.543352] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12968.553683] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12968.556035] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [12968.562868] mdc_intent_lock+0x353/0x7e0 [mdc] [12968.567214] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12968.571441] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12968.576117] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12968.582326] lmv_intent_open+0x338/0x1150 [lmv] [12968.588017] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12968.596012] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12968.602277] lmv_intent_lock+0x1dd/0x4c0 [lmv] [12968.604720] ll_intent_lock+0xc5/0xaa0 [lustre] [12968.611916] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12968.620961] ll_lookup_it+0xe0e/0x2270 [lustre] [12968.625916] ? do_raw_spin_unlock+0x75/0x190 [12968.631410] ? _raw_spin_unlock+0x12/0x30 [12968.636766] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [12968.639344] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [12968.642003] ll_atomic_open+0x36a/0xd70 [lustre] [12968.643994] lookup_open+0x356/0xb40 [12968.645591] do_last+0x313/0xfc0 [12968.650489] ? nd_jump_root+0xe5/0x160 [12968.652055] ? path_init+0x437/0x520 [12968.657572] path_openat+0xf7/0x500 [12968.658923] ? do_raw_spin_unlock+0x75/0x190 [12968.661545] do_filp_open+0x99/0x140 [12968.663422] ? getname_flags+0x6e/0x330 [12968.665872] ? __check_object_size+0xff/0x256 [12968.667909] ? do_raw_spin_unlock+0x75/0x190 [12968.670019] ? _raw_spin_unlock+0x12/0x30 [12968.671624] do_sys_openat2+0x2b4/0x410 [12968.673859] do_sys_open+0x73/0xa0 [12968.675227] __x64_sys_openat+0x24/0x30 [12968.676875] do_syscall_64+0xc1/0x440 [12968.678431] entry_SYSCALL_64_after_hwframe+0x49/0xae [12968.681074] RIP: 0033:0x7ff54af45332 [12968.682523] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [12968.696001] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [12968.703446] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [12968.707346] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [12968.710758] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [12968.714013] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [12968.718567] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [12976.055836] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [12976.059885] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12976.108945] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12976.120702] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12976.127028] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [12976.129125] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [12976.148060] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12976.158045] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [12976.164721] RDX: 0000000000004b2d RSI: 0000000000000003 RDI: ffff96848811cae0 [12976.171786] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [12976.176888] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [12976.187672] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12976.197216] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [12976.207584] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12976.214324] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [12976.220318] Call Trace: [12976.222574] [12976.223248] ? show_regs.cold.9+0x22/0x2f [12976.224813] ? watchdog_timer_fn.cold.6+0x62/0x10a [12976.227199] ? __hrtimer_run_queues+0x1c2/0x4d0 [12976.229524] ? watchdog+0x40/0x40 [12976.230845] ? hrtimer_interrupt+0x128/0x2c0 [12976.232874] ? smp_apic_timer_interrupt+0xa6/0x270 [12976.235711] ? apic_timer_interrupt+0xf/0x20 [12976.240645] [12976.243534] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [12976.254901] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12976.263627] do_raw_spin_lock+0x177/0x190 [12976.271886] _raw_spin_lock+0x21/0x30 [12976.273916] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12976.281233] obd_mod_free+0x46/0x140 [ptlrpc] [12976.282992] mdc_commit_open+0x91/0xb0 [mdc] [12976.291779] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12976.306623] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12976.319166] after_reply+0xe2c/0x1780 [ptlrpc] [12976.326489] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12976.332721] ptlrpcd+0xa82/0xe40 [ptlrpc] [12976.340724] ? do_wait_intr+0xf0/0xf0 [12976.344364] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12976.356260] kthread+0x1d1/0x200 [12976.365080] ? set_kthread_struct+0x70/0x70 [12976.372344] ret_from_fork+0x1f/0x30 [12978.655381] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11019s! [12978.672880] Showing busy workqueues and worker pools: [12978.674870] workqueue events: flags=0x0 [12978.684274] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12978.689820] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [12978.697052] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [12978.700923] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [12978.711255] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12978.715929] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [12978.720792] workqueue events_long: flags=0x0 [12978.723685] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12978.732973] pending: lu_objects_flush [obdclass] [12978.738602] workqueue events_unbound: flags=0x2 [12978.749363] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [12978.763802] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [12978.776601] workqueue events_power_efficient: flags=0x80 [12978.793801] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12978.803223] pending: neigh_periodic_work [12978.810129] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12978.813754] pending: do_cache_clean, check_lifetime [12978.822201] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12978.824370] pending: neigh_periodic_work [12978.829288] workqueue writeback: flags=0x4a [12978.833367] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [12978.837033] pending: wb_workfn, wb_workfn [12978.849273] workqueue xprtiod: flags=0xa [12978.856224] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [12978.863186] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [12978.869760] workqueue ipv6_addrconf: flags=0x40008 [12978.873298] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12978.877538] pending: addrconf_verify_work [12978.884819] workqueue knbd0-recv: flags=0x1a [12978.890784] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12978.894412] in-flight: 657:recv_work [12978.897860] workqueue ptlrpc_pinger: flags=0x2 [12978.901077] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12978.903408] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [12978.912880] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11020s workers=4 idle: 20831 12642 28859 [12978.919923] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12969s workers=2 idle: 659 [12980.058244] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [12980.061065] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12980.098481] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12980.106579] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12980.116804] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12980.119112] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12980.130778] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12980.141045] RAX: 0000000000003b83 RBX: ffff96848811cae0 RCX: 0000000000000001 [12980.148264] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [12980.159060] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [12980.172849] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [12980.185914] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [12980.193859] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [12980.213065] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12980.221648] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [12980.225754] Call Trace: [12980.226576] [12980.227356] ? show_regs.cold.9+0x22/0x2f [12980.235130] ? watchdog_timer_fn.cold.6+0x62/0x10a [12980.237784] ? __hrtimer_run_queues+0x1c2/0x4d0 [12980.240925] ? watchdog+0x40/0x40 [12980.242881] ? hrtimer_interrupt+0x128/0x2c0 [12980.248710] ? smp_apic_timer_interrupt+0xa6/0x270 [12980.252620] ? apic_timer_interrupt+0xf/0x20 [12980.258543] [12980.262129] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12980.271865] do_raw_spin_lock+0x177/0x190 [12980.274771] _raw_spin_lock+0x21/0x30 [12980.279971] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12980.293158] ? do_raw_spin_unlock+0x75/0x190 [12980.304240] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12980.307631] ? do_wait_intr+0xf0/0xf0 [12980.315846] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12980.324678] mdc_close+0x53b/0xd70 [mdc] [12980.327636] lmv_close+0x18d/0x490 [lmv] [12980.333622] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [12980.338541] ll_close_inode_openhandle+0x493/0xea0 [lustre] [12980.341211] ll_md_real_close+0xa5/0x2a0 [lustre] [12980.344056] ll_md_close+0x274/0x800 [lustre] [12980.346479] ll_file_release+0x128/0xc50 [lustre] [12980.349915] __fput+0x10d/0x440 [12980.352375] ____fput+0x12/0x20 [12980.353487] task_work_run+0xc6/0x110 [12980.354843] exit_to_usermode_loop+0x1dd/0x1f0 [12980.358851] do_syscall_64+0x3d6/0x440 [12980.360857] entry_SYSCALL_64_after_hwframe+0x49/0xae [12980.366224] RIP: 0033:0x7ff54af45cd8 [12980.368809] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [12980.378665] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [12980.383024] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [12980.387004] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [12980.389874] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [12980.393551] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [12980.396994] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12994.142431] rcu: INFO: rcu_sched self-detected stall on CPU [12994.145032] rcu: 0-...!: (5852368 ticks this GP) idle=98a/1/0x4000000000000002 softirq=125154/125154 fqs=959 [12994.149736] (t=11040302 jiffies g=253585 q=6436) [12994.152321] rcu: rcu_sched kthread starved for 11035262 jiffies! g253585 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [12994.157761] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [12994.163123] rcu: RCU grace-period kthread stack dump: [12994.166785] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [12994.170896] Call Trace: [12994.171884] __schedule+0x351/0xcb0 [12994.173373] schedule+0xc0/0x180 [12994.175353] schedule_timeout+0xb4/0x190 [12994.177759] ? do_raw_spin_unlock+0x75/0x190 [12994.179952] ? __next_timer_interrupt+0x160/0x160 [12994.182447] ? prepare_to_swait_event+0xf3/0x140 [12994.184783] rcu_gp_kthread+0x8dd/0x15b0 [12994.186752] ? force_qs_rnp+0x2c0/0x2c0 [12994.188891] kthread+0x1d1/0x200 [12994.190629] ? set_kthread_struct+0x70/0x70 [12994.193017] ret_from_fork+0x1f/0x30 [12994.195045] rcu: Stack dump where RCU GP kthread last ran: [12994.197854] NMI backtrace for cpu 0 [12994.199725] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12994.205156] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12994.209036] Call Trace: [12994.210410] [12994.211425] ? dump_stack+0xbb/0x10e [12994.213317] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [12994.215833] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [12994.218353] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [12994.220929] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [12994.223418] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [12994.226720] ? rcu_check_gp_kthread_starvation+0x16f/0x188 [12994.229651] ? rcu_sched_clock_irq.cold.51+0xce/0x307 [12994.232298] ? update_process_times+0x6d/0xc0 [12994.234393] ? tick_sched_handle.isra.18+0x27/0x90 [12994.236823] ? tick_sched_timer+0x53/0xb0 [12994.239379] ? __hrtimer_run_queues+0x1c2/0x4d0 [12994.241472] ? tick_sched_handle.isra.18+0x90/0x90 [12994.244503] ? hrtimer_interrupt+0x128/0x2c0 [12994.246633] ? smp_apic_timer_interrupt+0xa6/0x270 [12994.248871] ? apic_timer_interrupt+0xf/0x20 [12994.250980] [12994.251980] ? native_safe_halt+0x2e/0x50 [12994.253970] ? kvm_wait+0xcd/0xe0 [12994.256446] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12994.259543] ? do_raw_spin_lock+0x177/0x190 [12994.261826] ? _raw_spin_lock+0x21/0x30 [12994.263757] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12994.266411] ? obd_mod_free+0x46/0x140 [ptlrpc] [12994.268972] ? mdc_commit_open+0x91/0xb0 [mdc] [12994.271506] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12994.275029] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12994.277963] ? after_reply+0xe2c/0x1780 [ptlrpc] [12994.280658] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12994.283488] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [12994.285793] ? do_wait_intr+0xf0/0xf0 [12994.287688] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12994.290453] ? kthread+0x1d1/0x200 [12994.291969] ? set_kthread_struct+0x70/0x70 [12994.294101] ? ret_from_fork+0x1f/0x30 [12994.295942] NMI backtrace for cpu 0 [12994.297716] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12994.302906] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12994.306240] Call Trace: [12994.307436] [12994.308279] ? dump_stack+0xbb/0x10e [12994.309448] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [12994.311737] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [12994.312961] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [12994.314638] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [12994.316185] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [12994.317755] ? rcu_dump_cpu_stacks+0x11b/0x171 [12994.318889] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [12994.320496] ? update_process_times+0x6d/0xc0 [12994.321723] ? tick_sched_handle.isra.18+0x27/0x90 [12994.322965] ? tick_sched_timer+0x53/0xb0 [12994.324065] ? __hrtimer_run_queues+0x1c2/0x4d0 [12994.325698] ? tick_sched_handle.isra.18+0x90/0x90 [12994.326877] ? hrtimer_interrupt+0x128/0x2c0 [12994.328803] ? smp_apic_timer_interrupt+0xa6/0x270 [12994.330994] ? apic_timer_interrupt+0xf/0x20 [12994.332447] [12994.333424] ? native_safe_halt+0x2e/0x50 [12994.335304] ? kvm_wait+0xcd/0xe0 [12994.336780] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12994.339161] ? do_raw_spin_lock+0x177/0x190 [12994.341355] ? _raw_spin_lock+0x21/0x30 [12994.343094] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12994.345920] ? obd_mod_free+0x46/0x140 [ptlrpc] [12994.348427] ? mdc_commit_open+0x91/0xb0 [mdc] [12994.350496] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12994.353019] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12994.355869] ? after_reply+0xe2c/0x1780 [ptlrpc] [12994.358996] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12994.361902] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [12994.364310] ? do_wait_intr+0xf0/0xf0 [12994.366101] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12994.368747] ? kthread+0x1d1/0x200 [12994.370421] ? set_kthread_struct+0x70/0x70 [12994.372314] ? ret_from_fork+0x1f/0x30 [12996.059026] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [12996.061202] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [12996.067190] Modules linked in: lustre(O) osp(O) [12996.073004] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12996.073123] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12996.073133] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12996.073137] RIP: 0010:native_safe_halt+0x2e/0x50 [12996.073159] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12996.073165] RSP: 0000:ffffaf9fc15234f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12996.073171] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [12996.073174] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [12996.073176] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [12996.073180] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [12996.073183] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [12996.073199] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [12996.073203] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12996.073206] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [12996.073214] Call Trace: [12996.073223] [12996.073238] ? show_regs.cold.9+0x22/0x2f [12996.073251] ? watchdog_timer_fn.cold.6+0x62/0x10a [12996.073258] ? __hrtimer_run_queues+0x1c2/0x4d0 [12996.073262] ? watchdog+0x40/0x40 [12996.073266] ? hrtimer_interrupt+0x128/0x2c0 [12996.073274] ? smp_apic_timer_interrupt+0xa6/0x270 [12996.073282] ? apic_timer_interrupt+0xf/0x20 [12996.073287] [12996.073289] ? native_safe_halt+0x2e/0x50 [12996.073294] kvm_wait+0xcd/0xe0 [12996.073303] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12996.073312] do_raw_spin_lock+0x177/0x190 [12996.073317] _raw_spin_lock+0x21/0x30 [12996.073321] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12996.073806] ? do_raw_spin_unlock+0x75/0x190 [12996.073813] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12996.074002] ? do_wait_intr+0xf0/0xf0 [12996.074013] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12996.074208] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [12996.079238] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12996.079327] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12996.079337] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12996.079348] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12996.104310] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12996.110393] [12996.110406] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12996.110434] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12996.110445] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12996.110450] RAX: 0000000000007e06 RBX: ffff96848811cae0 RCX: 0000000000000001 [12996.110453] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [12996.110456] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [12996.110458] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [12996.110461] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [12996.110465] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [12996.110468] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12996.110471] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [12996.110481] Call Trace: [12996.110489] [12996.110496] ? show_regs.cold.9+0x22/0x2f [12996.110510] ? watchdog_timer_fn.cold.6+0x62/0x10a [12996.110518] ? __hrtimer_run_queues+0x1c2/0x4d0 [12996.110524] ? watchdog+0x40/0x40 [12996.110528] ? hrtimer_interrupt+0x128/0x2c0 [12996.110532] ? smp_apic_timer_interrupt+0xa6/0x270 [12996.110545] ? apic_timer_interrupt+0xf/0x20 [12996.110550] [12996.110561] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12996.110567] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12996.110576] do_raw_spin_lock+0x177/0x190 [12996.110582] _raw_spin_lock+0x21/0x30 [12996.110588] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12996.111154] process_one_work+0x2c8/0x700 [12996.111164] worker_thread+0x296/0x6e0 [12996.111170] ? rescuer_thread+0x570/0x570 [12996.111241] kthread+0x1d1/0x200 [12996.111248] ? set_kthread_struct+0x70/0x70 [12996.111251] ret_from_fork+0x1f/0x30 [12996.555883] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12996.567627] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12996.571374] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [12996.584885] mdc_intent_lock+0x353/0x7e0 [mdc] [12996.589053] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12996.596749] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12996.600808] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12996.609492] lmv_intent_open+0x338/0x1150 [lmv] [12996.622019] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12996.625974] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12996.630526] lmv_intent_lock+0x1dd/0x4c0 [lmv] [12996.633407] ll_intent_lock+0xc5/0xaa0 [lustre] [12996.641856] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12996.661147] ll_lookup_it+0xe0e/0x2270 [lustre] [12996.670843] ? do_raw_spin_unlock+0x75/0x190 [12996.675895] ? _raw_spin_unlock+0x12/0x30 [12996.679411] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [12996.689081] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [12996.693309] ll_atomic_open+0x36a/0xd70 [lustre] [12996.708384] lookup_open+0x356/0xb40 [12996.719030] do_last+0x313/0xfc0 [12996.722626] ? nd_jump_root+0xe5/0x160 [12996.732402] ? path_init+0x437/0x520 [12996.735005] path_openat+0xf7/0x500 [12996.742226] ? do_raw_spin_unlock+0x75/0x190 [12996.746171] do_filp_open+0x99/0x140 [12996.751345] ? getname_flags+0x6e/0x330 [12996.758928] ? __check_object_size+0xff/0x256 [12996.761149] ? do_raw_spin_unlock+0x75/0x190 [12996.773847] ? _raw_spin_unlock+0x12/0x30 [12996.776292] do_sys_openat2+0x2b4/0x410 [12996.784165] do_sys_open+0x73/0xa0 [12996.791652] __x64_sys_openat+0x24/0x30 [12996.796807] do_syscall_64+0xc1/0x440 [12996.803239] entry_SYSCALL_64_after_hwframe+0x49/0xae [12996.809309] RIP: 0033:0x7ff54af45332 [12996.811997] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [12996.831915] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [12996.838894] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [12996.845859] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [12996.851233] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [12996.860975] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [12996.870762] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [13008.058296] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [13008.061866] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13008.080673] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13008.086020] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13008.090843] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13008.093785] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13008.103616] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13008.107065] RAX: 0000000000005219 RBX: ffff96848811cae0 RCX: 0000000000000001 [13008.111769] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [13008.114949] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [13008.118130] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [13008.121937] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [13008.127668] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [13008.130714] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13008.132959] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [13008.135879] Call Trace: [13008.136925] [13008.138490] ? show_regs.cold.9+0x22/0x2f [13008.140356] ? watchdog_timer_fn.cold.6+0x62/0x10a [13008.142452] ? __hrtimer_run_queues+0x1c2/0x4d0 [13008.144549] ? watchdog+0x40/0x40 [13008.145948] ? hrtimer_interrupt+0x128/0x2c0 [13008.147741] ? smp_apic_timer_interrupt+0xa6/0x270 [13008.149549] ? apic_timer_interrupt+0xf/0x20 [13008.151602] [13008.152525] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13008.155506] do_raw_spin_lock+0x177/0x190 [13008.157189] _raw_spin_lock+0x21/0x30 [13008.159530] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13008.162823] ? do_raw_spin_unlock+0x75/0x190 [13008.164969] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13008.167153] ? do_wait_intr+0xf0/0xf0 [13008.168794] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13008.171631] mdc_close+0x53b/0xd70 [mdc] [13008.173489] lmv_close+0x18d/0x490 [lmv] [13008.174748] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [13008.177778] ll_close_inode_openhandle+0x493/0xea0 [lustre] [13008.181460] ll_md_real_close+0xa5/0x2a0 [lustre] [13008.184129] ll_md_close+0x274/0x800 [lustre] [13008.186606] ll_file_release+0x128/0xc50 [lustre] [13008.189652] __fput+0x10d/0x440 [13008.191405] ____fput+0x12/0x20 [13008.194803] task_work_run+0xc6/0x110 [13008.197117] exit_to_usermode_loop+0x1dd/0x1f0 [13008.200050] do_syscall_64+0x3d6/0x440 [13008.201966] entry_SYSCALL_64_after_hwframe+0x49/0xae [13008.204952] RIP: 0033:0x7ff54af45cd8 [13008.206213] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [13008.214539] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [13008.217321] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [13008.220265] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [13008.223735] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [13008.226715] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [13008.229988] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13009.376554] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11050s! [13009.387352] Showing busy workqueues and worker pools: [13009.389908] workqueue events: flags=0x0 [13009.391656] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13009.395162] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [13009.400816] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [13009.407022] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [13009.413705] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13009.420178] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [13009.423787] workqueue events_long: flags=0x0 [13009.426472] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13009.430625] pending: lu_objects_flush [obdclass] [13009.433456] workqueue events_unbound: flags=0x2 [13009.434993] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [13009.437330] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [13009.443153] workqueue events_power_efficient: flags=0x80 [13009.445803] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13009.449656] pending: neigh_periodic_work [13009.452210] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13009.455794] pending: do_cache_clean, check_lifetime [13009.460431] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13009.464708] pending: neigh_periodic_work [13009.466951] workqueue writeback: flags=0x4a [13009.470773] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [13009.475256] pending: wb_workfn, wb_workfn [13009.478553] workqueue xprtiod: flags=0xa [13009.481319] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [13009.486997] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [13009.493520] workqueue ipv6_addrconf: flags=0x40008 [13009.497676] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13009.501177] pending: addrconf_verify_work [13009.503572] workqueue knbd0-recv: flags=0x1a [13009.506048] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13009.510836] in-flight: 657:recv_work [13009.513629] workqueue ptlrpc_pinger: flags=0x2 [13009.515719] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13009.520708] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [13009.524967] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11050s workers=4 idle: 20831 12642 28859 [13009.532479] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12999s workers=2 idle: 659 [13024.059461] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [13024.062969] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13024.063071] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [13024.100687] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13024.119182] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13024.119347] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13024.119354] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13024.119357] RIP: 0010:native_safe_halt+0x2e/0x50 [13024.119484] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13024.119489] RSP: 0000:ffffaf9fc15234f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13024.119498] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [13024.119500] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c21b4154 [13024.119503] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [13024.119505] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [13024.119507] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [13024.119511] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [13024.119514] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13024.119520] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [13024.119528] Call Trace: [13024.119535] [13024.119546] ? show_regs.cold.9+0x22/0x2f [13024.119558] ? watchdog_timer_fn.cold.6+0x62/0x10a [13024.119711] ? __hrtimer_run_queues+0x1c2/0x4d0 [13024.119717] ? watchdog+0x40/0x40 [13024.119721] ? hrtimer_interrupt+0x128/0x2c0 [13024.119731] ? smp_apic_timer_interrupt+0xa6/0x270 [13024.119738] ? apic_timer_interrupt+0xf/0x20 [13024.119746] [13024.119750] ? native_safe_halt+0x2e/0x50 [13024.119754] kvm_wait+0xcd/0xe0 [13024.119765] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13024.119773] do_raw_spin_lock+0x177/0x190 [13024.119778] _raw_spin_lock+0x21/0x30 [13024.119786] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13024.143042] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13024.144656] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13024.146000] RIP: 0010:native_safe_halt+0x2e/0x50 [13024.146011] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13024.146014] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13024.146018] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [13024.146021] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c2134154 [13024.146024] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [13024.146026] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [13024.146028] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [13024.146031] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [13024.146035] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13024.146037] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [13024.146044] Call Trace: [13024.146049] [13024.146053] ? show_regs.cold.9+0x22/0x2f [13024.146062] ? watchdog_timer_fn.cold.6+0x62/0x10a [13024.146067] ? __hrtimer_run_queues+0x1c2/0x4d0 [13024.146071] ? watchdog+0x40/0x40 [13024.146074] ? hrtimer_interrupt+0x128/0x2c0 [13024.146077] ? smp_apic_timer_interrupt+0xa6/0x270 [13024.146083] ? apic_timer_interrupt+0xf/0x20 [13024.146087] [13024.146089] ? native_safe_halt+0x2e/0x50 [13024.146106] kvm_wait+0xcd/0xe0 [13024.146112] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13024.146119] do_raw_spin_lock+0x177/0x190 [13024.146122] _raw_spin_lock+0x21/0x30 [13024.146126] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13024.147182] process_one_work+0x2c8/0x700 [13024.147189] worker_thread+0x296/0x6e0 [13024.147193] ? rescuer_thread+0x570/0x570 [13024.147196] kthread+0x1d1/0x200 [13024.147200] ? set_kthread_struct+0x70/0x70 [13024.147203] ret_from_fork+0x1f/0x30 [13024.596771] ? do_raw_spin_unlock+0x75/0x190 [13024.599657] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13024.603681] ? do_wait_intr+0xf0/0xf0 [13024.605884] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13024.610879] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [13024.613545] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13024.616810] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13024.620481] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13024.623464] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [13024.626197] mdc_intent_lock+0x353/0x7e0 [mdc] [13024.629574] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13024.640000] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13024.647115] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13024.655294] lmv_intent_open+0x338/0x1150 [lmv] [13024.660191] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13024.664361] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13024.673289] lmv_intent_lock+0x1dd/0x4c0 [lmv] [13024.678634] ll_intent_lock+0xc5/0xaa0 [lustre] [13024.685149] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13024.693062] ll_lookup_it+0xe0e/0x2270 [lustre] [13024.696288] ? do_raw_spin_unlock+0x75/0x190 [13024.702859] ? _raw_spin_unlock+0x12/0x30 [13024.705267] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [13024.719135] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [13024.734842] ll_atomic_open+0x36a/0xd70 [lustre] [13024.746994] lookup_open+0x356/0xb40 [13024.758844] do_last+0x313/0xfc0 [13024.763619] ? nd_jump_root+0xe5/0x160 [13024.769157] ? path_init+0x437/0x520 [13024.771741] path_openat+0xf7/0x500 [13024.774591] ? do_raw_spin_unlock+0x75/0x190 [13024.777684] do_filp_open+0x99/0x140 [13024.779948] ? getname_flags+0x6e/0x330 [13024.783034] ? __check_object_size+0xff/0x256 [13024.785437] ? do_raw_spin_unlock+0x75/0x190 [13024.788612] ? _raw_spin_unlock+0x12/0x30 [13024.790681] do_sys_openat2+0x2b4/0x410 [13024.793082] do_sys_open+0x73/0xa0 [13024.794610] __x64_sys_openat+0x24/0x30 [13024.796778] do_syscall_64+0xc1/0x440 [13024.798611] entry_SYSCALL_64_after_hwframe+0x49/0xae [13024.801107] RIP: 0033:0x7ff54af45332 [13024.802667] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [13024.817206] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [13024.821474] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [13024.825563] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [13024.829803] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [13024.833413] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [13024.837421] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [13036.055362] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [13036.058823] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [13036.063131] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) [13036.069142] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13036.069277] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13036.069285] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13036.069288] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13036.069304] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13036.069309] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13036.069314] RAX: 0000000000003486 RBX: ffff96848811cae0 RCX: 0000000000000001 [13036.069317] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [13036.069319] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [13036.069321] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [13036.069323] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [13036.069326] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [13036.069328] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13036.069330] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [13036.069338] Call Trace: [13036.069344] [13036.069350] ? show_regs.cold.9+0x22/0x2f [13036.069358] ? watchdog_timer_fn.cold.6+0x62/0x10a [13036.069364] ? __hrtimer_run_queues+0x1c2/0x4d0 [13036.069367] ? watchdog+0x40/0x40 [13036.069371] ? hrtimer_interrupt+0x128/0x2c0 [13036.069374] ? smp_apic_timer_interrupt+0xa6/0x270 [13036.069388] ? apic_timer_interrupt+0xf/0x20 [13036.069393] [13036.069396] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13036.069401] do_raw_spin_lock+0x177/0x190 [13036.069406] _raw_spin_lock+0x21/0x30 [13036.069411] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13036.069697] ? do_raw_spin_unlock+0x75/0x190 [13036.069702] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13036.070556] ? do_wait_intr+0xf0/0xf0 [13036.070566] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13036.070773] mdc_close+0x53b/0xd70 [mdc] [13036.070970] lmv_close+0x18d/0x490 [lmv] [13036.071010] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [13036.071173] ll_close_inode_openhandle+0x493/0xea0 [lustre] [13036.071449] ll_md_real_close+0xa5/0x2a0 [lustre] [13036.071561] ll_md_close+0x274/0x800 [lustre] [13036.071661] ll_file_release+0x128/0xc50 [lustre] [13036.071738] __fput+0x10d/0x440 [13036.071748] ____fput+0x12/0x20 [13036.071752] task_work_run+0xc6/0x110 [13036.071759] exit_to_usermode_loop+0x1dd/0x1f0 [13036.071768] do_syscall_64+0x3d6/0x440 [13036.071771] entry_SYSCALL_64_after_hwframe+0x49/0xae [13036.071780] RIP: 0033:0x7ff54af45cd8 [13036.072056] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [13036.072062] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [13036.072067] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [13036.072070] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [13036.072078] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [13036.072079] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [13036.072082] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13036.421624] osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13036.461883] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13036.475282] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13036.484536] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [13036.492652] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [13036.520930] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13036.532207] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [13036.543518] RDX: 0000000000006f0a RSI: 0000000000000003 RDI: ffff96848811cae0 [13036.549565] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [13036.560177] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [13036.567363] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13036.575216] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [13036.579549] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13036.582155] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [13036.591954] Call Trace: [13036.598523] [13036.601567] ? show_regs.cold.9+0x22/0x2f [13036.608439] ? watchdog_timer_fn.cold.6+0x62/0x10a [13036.612769] ? __hrtimer_run_queues+0x1c2/0x4d0 [13036.614919] ? watchdog+0x40/0x40 [13036.616554] ? hrtimer_interrupt+0x128/0x2c0 [13036.618016] ? smp_apic_timer_interrupt+0xa6/0x270 [13036.620494] ? apic_timer_interrupt+0xf/0x20 [13036.622474] [13036.624547] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [13036.629402] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13036.633522] do_raw_spin_lock+0x177/0x190 [13036.643547] _raw_spin_lock+0x21/0x30 [13036.649738] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13036.658634] obd_mod_free+0x46/0x140 [ptlrpc] [13036.663519] mdc_commit_open+0x91/0xb0 [mdc] [13036.667061] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13036.671296] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13036.674144] after_reply+0xe2c/0x1780 [ptlrpc] [13036.676647] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13036.681255] ptlrpcd+0xa82/0xe40 [ptlrpc] [13036.683462] ? do_wait_intr+0xf0/0xf0 [13036.685002] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13036.687204] kthread+0x1d1/0x200 [13036.688462] ? set_kthread_struct+0x70/0x70 [13036.689865] ret_from_fork+0x1f/0x30 [13039.586965] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11080s! [13039.593743] Showing busy workqueues and worker pools: [13039.596161] workqueue events: flags=0x0 [13039.597387] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13039.602287] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [13039.607548] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [13039.612577] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [13039.620374] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13039.624636] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [13039.637631] workqueue events_long: flags=0x0 [13039.641657] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13039.647290] pending: lu_objects_flush [obdclass] [13039.650583] workqueue events_unbound: flags=0x2 [13039.656080] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [13039.661030] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [13039.667512] workqueue events_power_efficient: flags=0x80 [13039.669071] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13039.672104] pending: neigh_periodic_work [13039.673983] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13039.677405] pending: do_cache_clean, check_lifetime [13039.679908] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13039.682934] pending: neigh_periodic_work [13039.685684] workqueue writeback: flags=0x4a [13039.691385] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [13039.695171] pending: wb_workfn, wb_workfn [13039.696743] workqueue xprtiod: flags=0xa [13039.698996] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [13039.705409] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [13039.710126] workqueue ipv6_addrconf: flags=0x40008 [13039.712777] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13039.718806] pending: addrconf_verify_work [13039.720458] workqueue knbd0-recv: flags=0x1a [13039.726047] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13039.728889] in-flight: 657:recv_work [13039.734761] workqueue ptlrpc_pinger: flags=0x2 [13039.737010] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13039.740143] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [13039.743523] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11080s workers=4 idle: 20831 12642 28859 [13039.751271] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13029s workers=2 idle: 659 [13052.060976] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [13052.067346] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [13052.069124] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13052.069207] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13052.069215] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13052.069219] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13052.069234] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13052.069238] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13052.069242] RAX: 0000000000005f0b RBX: ffff96848811cae0 RCX: 0000000000000001 [13052.069244] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [13052.069247] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [13052.069249] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [13052.069252] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [13052.069255] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [13052.069258] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13052.069261] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [13052.069270] Call Trace: [13052.069276] [13052.069283] ? show_regs.cold.9+0x22/0x2f [13052.069293] ? watchdog_timer_fn.cold.6+0x62/0x10a [13052.069299] ? __hrtimer_run_queues+0x1c2/0x4d0 [13052.069304] ? watchdog+0x40/0x40 [13052.069309] ? hrtimer_interrupt+0x128/0x2c0 [13052.069314] ? smp_apic_timer_interrupt+0xa6/0x270 [13052.069323] ? apic_timer_interrupt+0xf/0x20 [13052.069328] [13052.069331] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13052.069337] do_raw_spin_lock+0x177/0x190 [13052.069342] _raw_spin_lock+0x21/0x30 [13052.069347] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13052.071223] ? do_raw_spin_unlock+0x75/0x190 [13052.071236] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13052.071523] ? do_wait_intr+0xf0/0xf0 [13052.071532] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13052.073164] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [13052.073407] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13052.073617] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13052.074205] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13052.074245] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [13052.074274] mdc_intent_lock+0x353/0x7e0 [mdc] [13052.074303] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13052.074386] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13052.074629] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13052.074903] lmv_intent_open+0x338/0x1150 [lmv] [13052.074936] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13052.075024] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13052.075104] lmv_intent_lock+0x1dd/0x4c0 [lmv] [13052.075125] ll_intent_lock+0xc5/0xaa0 [lustre] [13052.075212] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13052.075293] ll_lookup_it+0xe0e/0x2270 [lustre] [13052.075374] ? do_raw_spin_unlock+0x75/0x190 [13052.075383] ? _raw_spin_unlock+0x12/0x30 [13052.075390] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [13052.075416] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [13052.075435] ll_atomic_open+0x36a/0xd70 [lustre] [13052.075520] lookup_open+0x356/0xb40 [13052.075529] do_last+0x313/0xfc0 [13052.075534] ? nd_jump_root+0xe5/0x160 [13052.075539] ? path_init+0x437/0x520 [13052.075544] path_openat+0xf7/0x500 [13052.075549] ? do_raw_spin_unlock+0x75/0x190 [13052.075554] do_filp_open+0x99/0x140 [13052.075561] ? getname_flags+0x6e/0x330 [13052.075566] ? __check_object_size+0xff/0x256 [13052.075573] ? do_raw_spin_unlock+0x75/0x190 [13052.075577] ? _raw_spin_unlock+0x12/0x30 [13052.075581] do_sys_openat2+0x2b4/0x410 [13052.075585] do_sys_open+0x73/0xa0 [13052.075589] __x64_sys_openat+0x24/0x30 [13052.075593] do_syscall_64+0xc1/0x440 [13052.075600] entry_SYSCALL_64_after_hwframe+0x49/0xae [13052.075606] RIP: 0033:0x7ff54af45332 [13052.075615] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [13052.075619] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [13052.075624] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [13052.075626] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [13052.075628] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [13052.075630] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [13052.075632] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [13052.614804] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13052.642248] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13052.656029] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13052.667229] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13052.669723] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13052.683588] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13052.716216] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13052.724055] RAX: 0000000000003b0b RBX: ffff96848811cae0 RCX: 0000000000000001 [13052.736148] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [13052.749496] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [13052.757146] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [13052.765885] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [13052.768327] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [13052.778189] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13052.784383] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [13052.788545] Call Trace: [13052.790829] [13052.791598] ? show_regs.cold.9+0x22/0x2f [13052.797544] ? watchdog_timer_fn.cold.6+0x62/0x10a [13052.807451] ? __hrtimer_run_queues+0x1c2/0x4d0 [13052.811533] ? watchdog+0x40/0x40 [13052.817112] ? hrtimer_interrupt+0x128/0x2c0 [13052.818659] ? smp_apic_timer_interrupt+0xa6/0x270 [13052.824581] ? apic_timer_interrupt+0xf/0x20 [13052.826154] [13052.829649] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13052.831624] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13052.835451] do_raw_spin_lock+0x177/0x190 [13052.836961] _raw_spin_lock+0x21/0x30 [13052.839569] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13052.841707] process_one_work+0x2c8/0x700 [13052.845180] worker_thread+0x296/0x6e0 [13052.846586] ? rescuer_thread+0x570/0x570 [13052.854440] kthread+0x1d1/0x200 [13052.855586] ? set_kthread_struct+0x70/0x70 [13052.857123] ret_from_fork+0x1f/0x30 [13064.057893] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [13064.061871] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [13064.066437] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13064.066519] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13064.066528] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13064.066595] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [13064.066610] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [13064.066614] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13064.066618] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [13064.066621] RDX: 00000000000006f0 RSI: 0000000000000003 RDI: ffff96848811cae0 [13064.066623] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [13064.066625] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [13064.066627] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13064.066631] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [13064.066634] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13064.066636] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [13064.066770] Call Trace: [13064.066777] [13064.066784] ? show_regs.cold.9+0x22/0x2f [13064.066794] ? watchdog_timer_fn.cold.6+0x62/0x10a [13064.066800] ? __hrtimer_run_queues+0x1c2/0x4d0 [13064.066805] ? watchdog+0x40/0x40 [13064.066808] ? hrtimer_interrupt+0x128/0x2c0 [13064.066812] ? smp_apic_timer_interrupt+0xa6/0x270 [13064.066821] ? apic_timer_interrupt+0xf/0x20 [13064.066825] [13064.066828] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [13064.066833] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13064.066838] do_raw_spin_lock+0x177/0x190 [13064.066843] _raw_spin_lock+0x21/0x30 [13064.066849] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13064.078440] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13064.078521] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13064.078843] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13064.078848] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13064.078862] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13064.078866] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13064.078871] RAX: 0000000000006d74 RBX: ffff96848811cae0 RCX: 0000000000000001 [13064.078873] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [13064.078875] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [13064.078877] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [13064.078880] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [13064.078883] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [13064.078886] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13064.078888] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [13064.078897] Call Trace: [13064.078903] [13064.078909] ? show_regs.cold.9+0x22/0x2f [13064.078919] ? watchdog_timer_fn.cold.6+0x62/0x10a [13064.078926] ? __hrtimer_run_queues+0x1c2/0x4d0 [13064.078932] ? watchdog+0x40/0x40 [13064.078937] ? hrtimer_interrupt+0x128/0x2c0 [13064.078940] ? smp_apic_timer_interrupt+0xa6/0x270 [13064.079075] ? apic_timer_interrupt+0xf/0x20 [13064.079080] [13064.079083] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13064.079091] do_raw_spin_lock+0x177/0x190 [13064.079096] _raw_spin_lock+0x21/0x30 [13064.079101] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13064.081497] ? do_raw_spin_unlock+0x75/0x190 [13064.081507] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13064.162176] obd_mod_free+0x46/0x140 [ptlrpc] [13064.173561] ? do_wait_intr+0xf0/0xf0 [13064.173593] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13064.174309] mdc_close+0x53b/0xd70 [mdc] [13064.174349] lmv_close+0x18d/0x490 [lmv] [13064.174384] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [13064.174545] ll_close_inode_openhandle+0x493/0xea0 [lustre] [13064.174847] ll_md_real_close+0xa5/0x2a0 [lustre] [13064.197155] mdc_commit_open+0x91/0xb0 [mdc] [13064.197220] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13064.197546] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13064.198338] after_reply+0xe2c/0x1780 [ptlrpc] [13064.199083] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13064.200098] ptlrpcd+0xa82/0xe40 [ptlrpc] [13064.200319] ? do_wait_intr+0xf0/0xf0 [13064.200331] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13064.200531] kthread+0x1d1/0x200 [13064.200538] ? set_kthread_struct+0x70/0x70 [13064.200542] ret_from_fork+0x1f/0x30 [13064.599022] ll_md_close+0x274/0x800 [lustre] [13064.601068] ll_file_release+0x128/0xc50 [lustre] [13064.604262] __fput+0x10d/0x440 [13064.606511] ____fput+0x12/0x20 [13064.607980] task_work_run+0xc6/0x110 [13064.609250] exit_to_usermode_loop+0x1dd/0x1f0 [13064.613849] do_syscall_64+0x3d6/0x440 [13064.617012] entry_SYSCALL_64_after_hwframe+0x49/0xae [13064.619786] RIP: 0033:0x7ff54af45cd8 [13064.621242] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [13064.629262] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [13064.632680] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [13064.635779] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [13064.638857] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [13064.642205] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [13064.646446] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13069.793697] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11110s! [13069.804173] Showing busy workqueues and worker pools: [13069.809285] workqueue events: flags=0x0 [13069.813691] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13069.824160] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [13069.837672] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [13069.843787] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [13069.854669] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13069.867154] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [13069.873270] workqueue events_long: flags=0x0 [13069.879093] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13069.884294] pending: lu_objects_flush [obdclass] [13069.887332] workqueue events_unbound: flags=0x2 [13069.894460] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [13069.900775] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [13069.910180] workqueue events_power_efficient: flags=0x80 [13069.919776] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13069.928143] pending: neigh_periodic_work [13069.934137] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13069.950886] pending: do_cache_clean, check_lifetime [13069.953817] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13069.964179] pending: neigh_periodic_work [13069.979319] workqueue writeback: flags=0x4a [13069.991334] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [13069.996923] pending: wb_workfn, wb_workfn [13070.001620] workqueue xprtiod: flags=0xa [13070.009857] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [13070.020153] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [13070.032426] workqueue ipv6_addrconf: flags=0x40008 [13070.040432] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13070.053535] pending: addrconf_verify_work [13070.055152] workqueue knbd0-recv: flags=0x1a [13070.060677] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13070.066815] in-flight: 657:recv_work [13070.069446] workqueue ptlrpc_pinger: flags=0x2 [13070.079859] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13070.094130] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [13070.101424] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11111s workers=4 idle: 20831 12642 28859 [13070.107407] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13060s workers=2 idle: 659 [13080.060296] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [13080.064716] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [13080.065748] Modules linked in: [13080.073525] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13080.073631] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13080.073640] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13080.073643] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13080.073661] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13080.073666] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13080.073671] RAX: 0000000000002609 RBX: ffff96848811cae0 RCX: 0000000000000001 [13080.073673] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [13080.073677] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [13080.073679] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [13080.073682] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [13080.073687] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [13080.073690] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13080.073693] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [13080.073701] Call Trace: [13080.073708] [13080.073716] ? show_regs.cold.9+0x22/0x2f [13080.073726] ? watchdog_timer_fn.cold.6+0x62/0x10a [13080.073735] ? __hrtimer_run_queues+0x1c2/0x4d0 [13080.073739] ? watchdog+0x40/0x40 [13080.073744] ? hrtimer_interrupt+0x128/0x2c0 [13080.073749] ? smp_apic_timer_interrupt+0xa6/0x270 [13080.073758] ? apic_timer_interrupt+0xf/0x20 [13080.073763] [13080.073766] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13080.073773] do_raw_spin_lock+0x177/0x190 [13080.073778] _raw_spin_lock+0x21/0x30 [13080.073783] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13080.074164] ? do_raw_spin_unlock+0x75/0x190 [13080.074169] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13080.092093] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13080.092180] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13080.092190] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13080.092199] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13080.092677] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13080.092693] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13080.092698] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13080.092703] RAX: 0000000000007b84 RBX: ffff96848811cae0 RCX: 0000000000000001 [13080.092706] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [13080.092708] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [13080.092710] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [13080.092713] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [13080.092717] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [13080.092891] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13080.092894] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [13080.092902] Call Trace: [13080.092909] [13080.092915] ? show_regs.cold.9+0x22/0x2f [13080.092927] ? watchdog_timer_fn.cold.6+0x62/0x10a [13080.092934] ? __hrtimer_run_queues+0x1c2/0x4d0 [13080.092939] ? watchdog+0x40/0x40 [13080.092943] ? hrtimer_interrupt+0x128/0x2c0 [13080.092947] ? smp_apic_timer_interrupt+0xa6/0x270 [13080.092957] ? apic_timer_interrupt+0xf/0x20 [13080.092961] [13080.092965] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13080.092970] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13080.092976] do_raw_spin_lock+0x177/0x190 [13080.092981] _raw_spin_lock+0x21/0x30 [13080.092986] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13080.093229] process_one_work+0x2c8/0x700 [13080.093238] worker_thread+0x296/0x6e0 [13080.093242] ? rescuer_thread+0x570/0x570 [13080.093246] kthread+0x1d1/0x200 [13080.093251] ? set_kthread_struct+0x70/0x70 [13080.093254] ret_from_fork+0x1f/0x30 [13080.299150] ? do_wait_intr+0xf0/0xf0 [13080.305860] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13080.309210] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [13080.315139] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13080.320184] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13080.322530] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13080.330666] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [13080.333126] mdc_intent_lock+0x353/0x7e0 [mdc] [13080.335222] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13080.343007] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13080.346261] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13080.348684] lmv_intent_open+0x338/0x1150 [lmv] [13080.350765] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13080.353362] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13080.355807] lmv_intent_lock+0x1dd/0x4c0 [lmv] [13080.361707] ll_intent_lock+0xc5/0xaa0 [lustre] [13080.364359] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13080.368051] ll_lookup_it+0xe0e/0x2270 [lustre] [13080.370189] ? do_raw_spin_unlock+0x75/0x190 [13080.372328] ? _raw_spin_unlock+0x12/0x30 [13080.374783] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [13080.377331] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [13080.380132] ll_atomic_open+0x36a/0xd70 [lustre] [13080.384892] lookup_open+0x356/0xb40 [13080.387599] do_last+0x313/0xfc0 [13080.388924] ? nd_jump_root+0xe5/0x160 [13080.392672] ? path_init+0x437/0x520 [13080.394279] path_openat+0xf7/0x500 [13080.395994] ? do_raw_spin_unlock+0x75/0x190 [13080.397795] do_filp_open+0x99/0x140 [13080.399315] ? getname_flags+0x6e/0x330 [13080.400921] ? __check_object_size+0xff/0x256 [13080.405083] ? do_raw_spin_unlock+0x75/0x190 [13080.406586] ? _raw_spin_unlock+0x12/0x30 [13080.409728] do_sys_openat2+0x2b4/0x410 [13080.414300] do_sys_open+0x73/0xa0 [13080.417326] __x64_sys_openat+0x24/0x30 [13080.420205] do_syscall_64+0xc1/0x440 [13080.422058] entry_SYSCALL_64_after_hwframe+0x49/0xae [13080.423969] RIP: 0033:0x7ff54af45332 [13080.427225] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [13080.434050] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [13080.436761] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [13080.440169] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [13080.442619] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [13080.444516] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [13080.447182] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [13092.058028] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [13092.060734] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13092.090737] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13092.101238] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13092.113026] RIP: 0010:native_safe_halt+0x2e/0x50 [13092.114925] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13092.133859] RSP: 0018:ffffaf9fc1543ae0 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [13092.140623] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [13092.144144] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c20b4154 [13092.147694] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [13092.151584] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [13092.155408] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [13092.159653] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [13092.163918] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13092.166983] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [13092.171219] Call Trace: [13092.172800] [13092.173874] ? show_regs.cold.9+0x22/0x2f [13092.176059] ? watchdog_timer_fn.cold.6+0x62/0x10a [13092.180815] ? __hrtimer_run_queues+0x1c2/0x4d0 [13092.185687] ? watchdog+0x40/0x40 [13092.187557] ? hrtimer_interrupt+0x128/0x2c0 [13092.193106] ? smp_apic_timer_interrupt+0xa6/0x270 [13092.196469] ? apic_timer_interrupt+0xf/0x20 [13092.199448] [13092.201899] ? native_safe_halt+0x2e/0x50 [13092.210786] kvm_wait+0xcd/0xe0 [13092.213357] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13092.219555] do_raw_spin_lock+0x177/0x190 [13092.225136] _raw_spin_lock+0x21/0x30 [13092.230679] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13092.236479] ? do_raw_spin_unlock+0x75/0x190 [13092.240037] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13092.244428] ? do_wait_intr+0xf0/0xf0 [13092.248128] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13092.253521] mdc_close+0x53b/0xd70 [mdc] [13092.255365] lmv_close+0x18d/0x490 [lmv] [13092.260664] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [13092.268120] ll_close_inode_openhandle+0x493/0xea0 [lustre] [13092.272621] ll_md_real_close+0xa5/0x2a0 [lustre] [13092.277852] ll_md_close+0x274/0x800 [lustre] [13092.280464] ll_file_release+0x128/0xc50 [lustre] [13092.283421] __fput+0x10d/0x440 [13092.285353] ____fput+0x12/0x20 [13092.286817] task_work_run+0xc6/0x110 [13092.288894] exit_to_usermode_loop+0x1dd/0x1f0 [13092.291480] do_syscall_64+0x3d6/0x440 [13092.293364] entry_SYSCALL_64_after_hwframe+0x49/0xae [13092.298725] RIP: 0033:0x7ff54af45cd8 [13092.301379] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [13092.324741] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [13092.334910] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [13092.341326] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [13092.343970] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [13092.362807] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [13092.368781] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13096.055029] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [13096.065056] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13096.127908] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13096.145867] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13096.154551] RIP: 0010:native_safe_halt+0x2e/0x50 [13096.160236] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13096.180648] RSP: 0000:ffffaf9fc1b97bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13096.191845] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [13096.203648] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff96848811cae0 [13096.212563] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [13096.217820] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [13096.223481] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13096.226981] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [13096.231943] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13096.235804] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [13096.240125] Call Trace: [13096.241685] [13096.243079] ? show_regs.cold.9+0x22/0x2f [13096.245360] ? watchdog_timer_fn.cold.6+0x62/0x10a [13096.248548] ? __hrtimer_run_queues+0x1c2/0x4d0 [13096.251540] ? watchdog+0x40/0x40 [13096.253663] ? hrtimer_interrupt+0x128/0x2c0 [13096.256493] ? smp_apic_timer_interrupt+0xa6/0x270 [13096.259347] ? apic_timer_interrupt+0xf/0x20 [13096.262290] [13096.263779] ? native_safe_halt+0x2e/0x50 [13096.266366] kvm_wait+0xcd/0xe0 [13096.268109] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13096.270809] do_raw_spin_lock+0x177/0x190 [13096.272967] _raw_spin_lock+0x21/0x30 [13096.274951] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13096.278134] obd_mod_free+0x46/0x140 [ptlrpc] [13096.281471] mdc_commit_open+0x91/0xb0 [mdc] [13096.284054] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13096.287404] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13096.291049] after_reply+0xe2c/0x1780 [ptlrpc] [13096.293954] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13096.297013] ptlrpcd+0xa82/0xe40 [ptlrpc] [13096.300133] ? do_wait_intr+0xf0/0xf0 [13096.302486] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13096.305960] kthread+0x1d1/0x200 [13096.307809] ? set_kthread_struct+0x70/0x70 [13096.310366] ret_from_fork+0x1f/0x30 [13100.513739] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11141s! [13100.532450] Showing busy workqueues and worker pools: [13100.543771] workqueue events: flags=0x0 [13100.547016] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13100.552197] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [13100.561136] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [13100.574796] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [13100.597092] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13100.605253] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [13100.617672] workqueue events_long: flags=0x0 [13100.625532] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13100.635100] pending: lu_objects_flush [obdclass] [13100.637846] workqueue events_unbound: flags=0x2 [13100.642848] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [13100.653514] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [13100.663786] workqueue events_power_efficient: flags=0x80 [13100.672803] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13100.677077] pending: neigh_periodic_work [13100.678095] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13100.689041] pending: do_cache_clean, check_lifetime [13100.692851] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13100.695671] pending: neigh_periodic_work [13100.703976] workqueue writeback: flags=0x4a [13100.706510] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [13100.715084] pending: wb_workfn, wb_workfn [13100.723304] workqueue xprtiod: flags=0xa [13100.724792] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [13100.731822] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [13100.741161] workqueue ipv6_addrconf: flags=0x40008 [13100.746887] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13100.757219] pending: addrconf_verify_work [13100.758913] workqueue knbd0-recv: flags=0x1a [13100.762423] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13100.766025] in-flight: 657:recv_work [13100.772818] workqueue ptlrpc_pinger: flags=0x2 [13100.774594] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13100.778628] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [13100.783912] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11141s workers=4 idle: 20831 12642 28859 [13100.789622] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13091s workers=2 idle: 659 [13108.059041] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [13108.060052] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [13108.068665] Modules linked in: lustre(O) [13108.074754] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13108.074837] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13108.074846] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13108.074849] RIP: 0010:native_safe_halt+0x2e/0x50 [13108.074865] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13108.074871] RSP: 0000:ffffaf9fc15234f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13108.074876] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [13108.074879] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c21b4154 [13108.074881] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [13108.074884] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [13108.074886] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [13108.074890] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [13108.074893] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13108.074895] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [13108.074904] Call Trace: [13108.074912] [13108.074919] ? show_regs.cold.9+0x22/0x2f [13108.074929] ? watchdog_timer_fn.cold.6+0x62/0x10a [13108.074936] ? __hrtimer_run_queues+0x1c2/0x4d0 [13108.074940] ? watchdog+0x40/0x40 [13108.074944] ? hrtimer_interrupt+0x128/0x2c0 [13108.074949] ? smp_apic_timer_interrupt+0xa6/0x270 [13108.074955] ? apic_timer_interrupt+0xf/0x20 [13108.074960] [13108.074963] ? native_safe_halt+0x2e/0x50 [13108.074967] kvm_wait+0xcd/0xe0 [13108.074974] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13108.074984] do_raw_spin_lock+0x177/0x190 [13108.074988] _raw_spin_lock+0x21/0x30 [13108.074992] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13108.075320] ? do_raw_spin_unlock+0x75/0x190 [13108.075326] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13108.075519] ? do_wait_intr+0xf0/0xf0 [13108.075528] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13108.075720] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [13108.075932] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13108.076131] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13108.076260] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13108.077064] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [13108.077095] mdc_intent_lock+0x353/0x7e0 [mdc] [13108.077178] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13108.077792] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13108.078404] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13108.078812] lmv_intent_open+0x338/0x1150 [lmv] [13108.078846] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13108.080482] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13108.080572] lmv_intent_lock+0x1dd/0x4c0 [lmv] [13108.080602] ll_intent_lock+0xc5/0xaa0 [lustre] [13108.080971] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13108.081050] ll_lookup_it+0xe0e/0x2270 [lustre] [13108.081142] ? do_raw_spin_unlock+0x75/0x190 [13108.081151] ? _raw_spin_unlock+0x12/0x30 [13108.081158] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [13108.081185] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [13108.081204] ll_atomic_open+0x36a/0xd70 [lustre] [13108.081307] lookup_open+0x356/0xb40 [13108.081317] do_last+0x313/0xfc0 [13108.081322] ? nd_jump_root+0xe5/0x160 [13108.081326] ? path_init+0x437/0x520 [13108.081331] path_openat+0xf7/0x500 [13108.081336] ? do_raw_spin_unlock+0x75/0x190 [13108.081341] do_filp_open+0x99/0x140 [13108.081347] ? getname_flags+0x6e/0x330 [13108.081351] ? __check_object_size+0xff/0x256 [13108.081360] ? do_raw_spin_unlock+0x75/0x190 [13108.081364] ? _raw_spin_unlock+0x12/0x30 [13108.081369] do_sys_openat2+0x2b4/0x410 [13108.081373] do_sys_open+0x73/0xa0 [13108.081378] __x64_sys_openat+0x24/0x30 [13108.081381] do_syscall_64+0xc1/0x440 [13108.081388] entry_SYSCALL_64_after_hwframe+0x49/0xae [13108.081395] RIP: 0033:0x7ff54af45332 [13108.081403] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [13108.081407] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [13108.081412] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [13108.081415] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [13108.081417] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [13108.081419] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [13108.081420] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [13108.508479] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13108.543888] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13108.552307] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13108.561348] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13108.568328] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13108.575380] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13108.584675] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13108.587183] RAX: 0000000000007b8d RBX: ffff96848811cae0 RCX: 0000000000000001 [13108.592471] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [13108.601649] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [13108.608540] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [13108.613920] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [13108.618344] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [13108.620763] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13108.623819] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [13108.627229] Call Trace: [13108.628311] [13108.629072] ? show_regs.cold.9+0x22/0x2f [13108.630816] ? watchdog_timer_fn.cold.6+0x62/0x10a [13108.636477] ? __hrtimer_run_queues+0x1c2/0x4d0 [13108.640947] ? watchdog+0x40/0x40 [13108.646911] ? hrtimer_interrupt+0x128/0x2c0 [13108.652524] ? smp_apic_timer_interrupt+0xa6/0x270 [13108.658214] ? apic_timer_interrupt+0xf/0x20 [13108.659175] [13108.660814] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13108.663124] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13108.666404] do_raw_spin_lock+0x177/0x190 [13108.668818] _raw_spin_lock+0x21/0x30 [13108.670874] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13108.675375] process_one_work+0x2c8/0x700 [13108.678549] worker_thread+0x296/0x6e0 [13108.687510] ? rescuer_thread+0x570/0x570 [13108.688757] kthread+0x1d1/0x200 [13108.696743] ? set_kthread_struct+0x70/0x70 [13108.698437] ret_from_fork+0x1f/0x30 [13120.060039] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [13120.072836] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13120.118906] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13120.129055] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13120.136874] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13120.142840] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13120.159232] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13120.164340] RAX: 0000000000007e9a RBX: ffff96848811cae0 RCX: 0000000000000001 [13120.168780] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [13120.173451] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [13120.177109] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [13120.183516] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [13120.191353] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [13120.197290] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13120.203044] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [13120.211722] Call Trace: [13120.215916] [13120.221201] ? show_regs.cold.9+0x22/0x2f [13120.227688] ? watchdog_timer_fn.cold.6+0x62/0x10a [13120.232500] ? __hrtimer_run_queues+0x1c2/0x4d0 [13120.239724] ? watchdog+0x40/0x40 [13120.243168] ? hrtimer_interrupt+0x128/0x2c0 [13120.246316] ? smp_apic_timer_interrupt+0xa6/0x270 [13120.250469] ? apic_timer_interrupt+0xf/0x20 [13120.254323] [13120.256071] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13120.260196] do_raw_spin_lock+0x177/0x190 [13120.264112] _raw_spin_lock+0x21/0x30 [13120.265887] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13120.267924] ? do_raw_spin_unlock+0x75/0x190 [13120.280264] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13120.282061] ? do_wait_intr+0xf0/0xf0 [13120.289744] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13120.294497] mdc_close+0x53b/0xd70 [mdc] [13120.297521] lmv_close+0x18d/0x490 [lmv] [13120.300170] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [13120.303513] ll_close_inode_openhandle+0x493/0xea0 [lustre] [13120.307462] ll_md_real_close+0xa5/0x2a0 [lustre] [13120.320008] ll_md_close+0x274/0x800 [lustre] [13120.325188] ll_file_release+0x128/0xc50 [lustre] [13120.333504] __fput+0x10d/0x440 [13120.336968] ____fput+0x12/0x20 [13120.340426] task_work_run+0xc6/0x110 [13120.344979] exit_to_usermode_loop+0x1dd/0x1f0 [13120.348812] do_syscall_64+0x3d6/0x440 [13120.352780] entry_SYSCALL_64_after_hwframe+0x49/0xae [13120.358519] RIP: 0033:0x7ff54af45cd8 [13120.360530] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [13120.376639] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [13120.386284] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [13120.390371] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [13120.396334] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [13120.403414] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [13120.408365] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13128.058490] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [13128.067555] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13128.101597] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13128.112112] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13128.122158] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [13128.128143] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [13128.142371] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13128.148550] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [13128.152186] RDX: 00000000000068ef RSI: 0000000000000003 RDI: ffff96848811cae0 [13128.158180] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [13128.166836] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [13128.174884] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13128.180664] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [13128.188791] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13128.198253] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [13128.209733] Call Trace: [13128.213835] [13128.217249] ? show_regs.cold.9+0x22/0x2f [13128.220199] ? watchdog_timer_fn.cold.6+0x62/0x10a [13128.225023] ? __hrtimer_run_queues+0x1c2/0x4d0 [13128.230094] ? watchdog+0x40/0x40 [13128.236145] ? hrtimer_interrupt+0x128/0x2c0 [13128.240204] ? smp_apic_timer_interrupt+0xa6/0x270 [13128.247829] ? apic_timer_interrupt+0xf/0x20 [13128.250094] [13128.253259] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [13128.257612] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13128.262531] do_raw_spin_lock+0x177/0x190 [13128.264753] _raw_spin_lock+0x21/0x30 [13128.267228] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13128.271518] obd_mod_free+0x46/0x140 [ptlrpc] [13128.275850] mdc_commit_open+0x91/0xb0 [mdc] [13128.279472] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13128.285160] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13128.293531] after_reply+0xe2c/0x1780 [ptlrpc] [13128.297044] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13128.304207] ptlrpcd+0xa82/0xe40 [ptlrpc] [13128.310705] ? do_wait_intr+0xf0/0xf0 [13128.313645] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13128.321742] kthread+0x1d1/0x200 [13128.326764] ? set_kthread_struct+0x70/0x70 [13128.330525] ret_from_fork+0x1f/0x30 [13131.234450] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11172s! [13131.243667] Showing busy workqueues and worker pools: [13131.247150] workqueue events: flags=0x0 [13131.249776] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13131.253433] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [13131.257774] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [13131.261415] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [13131.267081] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13131.271919] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [13131.274641] workqueue events_long: flags=0x0 [13131.276408] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13131.279299] pending: lu_objects_flush [obdclass] [13131.281588] workqueue events_unbound: flags=0x2 [13131.283440] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [13131.286095] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [13131.290160] workqueue events_power_efficient: flags=0x80 [13131.293464] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13131.299520] pending: neigh_periodic_work [13131.302524] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13131.307065] pending: do_cache_clean, check_lifetime [13131.309493] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13131.313029] pending: neigh_periodic_work [13131.315149] workqueue writeback: flags=0x4a [13131.318898] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [13131.322148] pending: wb_workfn, wb_workfn [13131.323826] workqueue xprtiod: flags=0xa [13131.325334] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [13131.327517] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [13131.330508] workqueue ipv6_addrconf: flags=0x40008 [13131.332525] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13131.334563] pending: addrconf_verify_work [13131.336175] workqueue knbd0-recv: flags=0x1a [13131.337587] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13131.340200] in-flight: 657:recv_work [13131.342096] workqueue ptlrpc_pinger: flags=0x2 [13131.344339] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13131.349626] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [13131.352614] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11172s workers=4 idle: 20831 12642 28859 [13131.360415] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13121s workers=2 idle: 659 [13136.060033] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [13136.060888] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [13136.068575] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13136.068672] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13136.068681] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13136.068685] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13136.068703] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13136.068707] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13136.068712] RAX: 0000000000005626 RBX: ffff96848811cae0 RCX: 0000000000000001 [13136.068715] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [13136.068718] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [13136.068720] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [13136.068723] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [13136.068727] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [13136.068730] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13136.068733] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [13136.068741] Call Trace: [13136.068747] [13136.068755] ? show_regs.cold.9+0x22/0x2f [13136.068765] ? watchdog_timer_fn.cold.6+0x62/0x10a [13136.068772] ? __hrtimer_run_queues+0x1c2/0x4d0 [13136.068777] ? watchdog+0x40/0x40 [13136.068781] ? hrtimer_interrupt+0x128/0x2c0 [13136.068785] ? smp_apic_timer_interrupt+0xa6/0x270 [13136.068795] ? apic_timer_interrupt+0xf/0x20 [13136.068800] [13136.068808] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13136.068889] do_raw_spin_lock+0x177/0x190 [13136.068894] _raw_spin_lock+0x21/0x30 [13136.068900] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13136.073862] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13136.073922] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13136.073930] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13136.073936] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13136.074349] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13136.074361] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13136.074365] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13136.074370] RAX: 0000000000007f85 RBX: ffff96848811cae0 RCX: 0000000000000001 [13136.074372] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [13136.074374] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [13136.074376] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [13136.074378] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [13136.074381] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [13136.074384] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13136.074386] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [13136.074394] Call Trace: [13136.074401] [13136.074407] ? show_regs.cold.9+0x22/0x2f [13136.074415] ? watchdog_timer_fn.cold.6+0x62/0x10a [13136.074422] ? __hrtimer_run_queues+0x1c2/0x4d0 [13136.074426] ? watchdog+0x40/0x40 [13136.074430] ? hrtimer_interrupt+0x128/0x2c0 [13136.074434] ? smp_apic_timer_interrupt+0xa6/0x270 [13136.074441] ? apic_timer_interrupt+0xf/0x20 [13136.074446] [13136.074448] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13136.074453] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13136.074458] do_raw_spin_lock+0x177/0x190 [13136.074463] _raw_spin_lock+0x21/0x30 [13136.074468] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13136.074664] process_one_work+0x2c8/0x700 [13136.074671] worker_thread+0x296/0x6e0 [13136.074676] ? rescuer_thread+0x570/0x570 [13136.074679] kthread+0x1d1/0x200 [13136.074683] ? set_kthread_struct+0x70/0x70 [13136.074687] ret_from_fork+0x1f/0x30 [13136.471355] ? do_raw_spin_unlock+0x75/0x190 [13136.473188] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13136.475567] ? do_wait_intr+0xf0/0xf0 [13136.477202] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13136.479417] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [13136.481252] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13136.483675] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13136.485443] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13136.487434] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [13136.489335] mdc_intent_lock+0x353/0x7e0 [mdc] [13136.490793] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13136.493588] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13136.496348] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13136.505522] lmv_intent_open+0x338/0x1150 [lmv] [13136.506545] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13136.509450] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13136.514484] lmv_intent_lock+0x1dd/0x4c0 [lmv] [13136.517570] ll_intent_lock+0xc5/0xaa0 [lustre] [13136.525351] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13136.531498] ll_lookup_it+0xe0e/0x2270 [lustre] [13136.534204] ? do_raw_spin_unlock+0x75/0x190 [13136.537185] ? _raw_spin_unlock+0x12/0x30 [13136.538606] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [13136.547432] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [13136.554355] ll_atomic_open+0x36a/0xd70 [lustre] [13136.558381] lookup_open+0x356/0xb40 [13136.559643] do_last+0x313/0xfc0 [13136.564998] ? nd_jump_root+0xe5/0x160 [13136.566311] ? path_init+0x437/0x520 [13136.569254] path_openat+0xf7/0x500 [13136.575140] ? do_raw_spin_unlock+0x75/0x190 [13136.578331] do_filp_open+0x99/0x140 [13136.581402] ? getname_flags+0x6e/0x330 [13136.583791] ? __check_object_size+0xff/0x256 [13136.588959] ? do_raw_spin_unlock+0x75/0x190 [13136.592972] ? _raw_spin_unlock+0x12/0x30 [13136.597498] do_sys_openat2+0x2b4/0x410 [13136.603681] do_sys_open+0x73/0xa0 [13136.605765] __x64_sys_openat+0x24/0x30 [13136.609446] do_syscall_64+0xc1/0x440 [13136.612988] entry_SYSCALL_64_after_hwframe+0x49/0xae [13136.618069] RIP: 0033:0x7ff54af45332 [13136.619698] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [13136.635152] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [13136.641602] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [13136.649327] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [13136.656105] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [13136.663228] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [13136.668354] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [13148.058602] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [13148.066264] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13148.097214] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13148.105446] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13148.113876] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13148.116968] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13148.129772] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13148.135168] RAX: 0000000000007c82 RBX: ffff96848811cae0 RCX: 0000000000000001 [13148.144020] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [13148.146700] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [13148.153419] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [13148.165106] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [13148.174726] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [13148.183244] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13148.185977] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [13148.192343] Call Trace: [13148.195506] [13148.199584] ? show_regs.cold.9+0x22/0x2f [13148.205517] ? watchdog_timer_fn.cold.6+0x62/0x10a [13148.211546] ? __hrtimer_run_queues+0x1c2/0x4d0 [13148.222103] ? watchdog+0x40/0x40 [13148.229482] ? hrtimer_interrupt+0x128/0x2c0 [13148.237067] ? smp_apic_timer_interrupt+0xa6/0x270 [13148.248205] ? apic_timer_interrupt+0xf/0x20 [13148.254625] [13148.259521] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13148.271769] do_raw_spin_lock+0x177/0x190 [13148.280693] _raw_spin_lock+0x21/0x30 [13148.284810] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13148.298241] ? do_raw_spin_unlock+0x75/0x190 [13148.303178] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13148.315436] ? do_wait_intr+0xf0/0xf0 [13148.324315] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13148.326259] mdc_close+0x53b/0xd70 [mdc] [13148.337992] lmv_close+0x18d/0x490 [lmv] [13148.344352] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [13148.350742] ll_close_inode_openhandle+0x493/0xea0 [lustre] [13148.359856] ll_md_real_close+0xa5/0x2a0 [lustre] [13148.361732] ll_md_close+0x274/0x800 [lustre] [13148.370752] ll_file_release+0x128/0xc50 [lustre] [13148.372247] __fput+0x10d/0x440 [13148.377781] ____fput+0x12/0x20 [13148.379180] task_work_run+0xc6/0x110 [13148.383756] exit_to_usermode_loop+0x1dd/0x1f0 [13148.391460] do_syscall_64+0x3d6/0x440 [13148.400995] entry_SYSCALL_64_after_hwframe+0x49/0xae [13148.406327] RIP: 0033:0x7ff54af45cd8 [13148.417772] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [13148.443668] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [13148.460175] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [13148.471241] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [13148.476836] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [13148.483560] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [13148.497367] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13156.056644] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [13156.064996] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13156.099465] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13156.108371] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13156.115946] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [13156.122607] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [13156.139013] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13156.144582] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [13156.146918] RDX: 000000000000005e RSI: 0000000000000003 RDI: ffff96848811cae0 [13156.155112] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [13156.157746] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [13156.164130] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13156.168628] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [13156.173293] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13156.175637] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [13156.179177] Call Trace: [13156.180595] [13156.181572] ? show_regs.cold.9+0x22/0x2f [13156.184561] ? watchdog_timer_fn.cold.6+0x62/0x10a [13156.188091] ? __hrtimer_run_queues+0x1c2/0x4d0 [13156.190510] ? watchdog+0x40/0x40 [13156.191417] ? hrtimer_interrupt+0x128/0x2c0 [13156.194518] ? smp_apic_timer_interrupt+0xa6/0x270 [13156.201134] ? apic_timer_interrupt+0xf/0x20 [13156.205169] [13156.206803] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [13156.211653] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13156.216928] do_raw_spin_lock+0x177/0x190 [13156.221004] _raw_spin_lock+0x21/0x30 [13156.227622] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13156.237784] obd_mod_free+0x46/0x140 [ptlrpc] [13156.239410] mdc_commit_open+0x91/0xb0 [mdc] [13156.241073] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13156.244008] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13156.247464] after_reply+0xe2c/0x1780 [ptlrpc] [13156.250118] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13156.252943] ptlrpcd+0xa82/0xe40 [ptlrpc] [13156.255446] ? do_wait_intr+0xf0/0xf0 [13156.257105] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13156.259867] kthread+0x1d1/0x200 [13156.261501] ? set_kthread_struct+0x70/0x70 [13156.263357] ret_from_fork+0x1f/0x30 [13161.441959] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11202s! [13161.450493] Showing busy workqueues and worker pools: [13161.457174] workqueue events: flags=0x0 [13161.463099] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13161.470872] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [13161.482953] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [13161.492959] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [13161.499453] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13161.509380] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [13161.513279] workqueue events_long: flags=0x0 [13161.518387] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13161.528360] pending: lu_objects_flush [obdclass] [13161.533939] workqueue events_unbound: flags=0x2 [13161.538142] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [13161.547543] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [13161.559746] workqueue events_power_efficient: flags=0x80 [13161.563933] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13161.577351] pending: neigh_periodic_work [13161.587511] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13161.595549] pending: do_cache_clean, check_lifetime [13161.603438] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13161.613318] pending: neigh_periodic_work [13161.618979] workqueue writeback: flags=0x4a [13161.621741] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [13161.633078] pending: wb_workfn, wb_workfn [13161.636534] workqueue xprtiod: flags=0xa [13161.641930] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [13161.649084] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [13161.664918] workqueue ipv6_addrconf: flags=0x40008 [13161.666701] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13161.676715] pending: addrconf_verify_work [13161.678243] workqueue knbd0-recv: flags=0x1a [13161.694457] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13161.702174] in-flight: 657:recv_work [13161.707684] workqueue ptlrpc_pinger: flags=0x2 [13161.714922] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13161.735526] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [13161.750684] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11202s workers=4 idle: 20831 12642 28859 [13161.758770] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13151s workers=2 idle: 659 [13164.060092] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [13164.064178] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13164.067559] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [13164.085964] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13164.085980] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13164.085984] RIP: 0010:native_safe_halt+0x2e/0x50 [13164.085999] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13164.086003] RSP: 0000:ffffaf9fc15234f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13164.086009] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [13164.086011] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [13164.086013] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [13164.086016] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [13164.086019] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [13164.086022] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [13164.086026] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13164.086028] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [13164.086037] Call Trace: [13164.086043] [13164.086049] ? show_regs.cold.9+0x22/0x2f [13164.086060] ? watchdog_timer_fn.cold.6+0x62/0x10a [13164.086068] ? __hrtimer_run_queues+0x1c2/0x4d0 [13164.086072] ? watchdog+0x40/0x40 [13164.086076] ? hrtimer_interrupt+0x128/0x2c0 [13164.086080] ? smp_apic_timer_interrupt+0xa6/0x270 [13164.086087] ? apic_timer_interrupt+0xf/0x20 [13164.086092] [13164.086095] ? native_safe_halt+0x2e/0x50 [13164.086100] kvm_wait+0xcd/0xe0 [13164.086108] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13164.086118] do_raw_spin_lock+0x177/0x190 [13164.086123] _raw_spin_lock+0x21/0x30 [13164.086127] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13164.101503] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13164.101590] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13164.101598] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13164.101604] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13164.101975] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13164.101987] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13164.101991] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13164.101995] RAX: 0000000000007f92 RBX: ffff96848811cae0 RCX: 0000000000000001 [13164.101997] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [13164.101999] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [13164.102001] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [13164.102003] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [13164.102006] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [13164.102009] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13164.102011] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [13164.102020] Call Trace: [13164.102025] [13164.102030] ? show_regs.cold.9+0x22/0x2f [13164.102039] ? watchdog_timer_fn.cold.6+0x62/0x10a [13164.102046] ? __hrtimer_run_queues+0x1c2/0x4d0 [13164.103783] ? watchdog+0x40/0x40 [13164.103790] ? hrtimer_interrupt+0x128/0x2c0 [13164.103795] ? smp_apic_timer_interrupt+0xa6/0x270 [13164.103803] ? apic_timer_interrupt+0xf/0x20 [13164.103808] [13164.103811] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13164.103818] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13164.103825] do_raw_spin_lock+0x177/0x190 [13164.103831] _raw_spin_lock+0x21/0x30 [13164.103836] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13164.124818] ? do_raw_spin_unlock+0x75/0x190 [13164.124837] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13164.125107] ? do_wait_intr+0xf0/0xf0 [13164.125116] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13164.125807] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [13164.126046] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13164.126265] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13164.139538] process_one_work+0x2c8/0x700 [13164.141157] worker_thread+0x296/0x6e0 [13164.142702] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13164.142755] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [13164.142856] mdc_intent_lock+0x353/0x7e0 [mdc] [13164.142892] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13164.143006] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13164.143398] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13164.143429] lmv_intent_open+0x338/0x1150 [lmv] [13164.143632] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13164.143989] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13164.144078] lmv_intent_lock+0x1dd/0x4c0 [lmv] [13164.144102] ll_intent_lock+0xc5/0xaa0 [lustre] [13164.144193] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13164.144351] ll_lookup_it+0xe0e/0x2270 [lustre] [13164.144441] ? do_raw_spin_unlock+0x75/0x190 [13164.144450] ? _raw_spin_unlock+0x12/0x30 [13164.144457] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [13164.144486] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [13164.144507] ll_atomic_open+0x36a/0xd70 [lustre] [13164.144715] lookup_open+0x356/0xb40 [13164.144724] do_last+0x313/0xfc0 [13164.144730] ? nd_jump_root+0xe5/0x160 [13164.144746] ? path_init+0x437/0x520 [13164.144751] path_openat+0xf7/0x500 [13164.144757] ? do_raw_spin_unlock+0x75/0x190 [13164.144762] do_filp_open+0x99/0x140 [13164.144769] ? getname_flags+0x6e/0x330 [13164.144774] ? __check_object_size+0xff/0x256 [13164.144781] ? do_raw_spin_unlock+0x75/0x190 [13164.144785] ? _raw_spin_unlock+0x12/0x30 [13164.144790] do_sys_openat2+0x2b4/0x410 [13164.144795] do_sys_open+0x73/0xa0 [13164.144799] __x64_sys_openat+0x24/0x30 [13164.144803] do_syscall_64+0xc1/0x440 [13164.144810] entry_SYSCALL_64_after_hwframe+0x49/0xae [13164.144817] RIP: 0033:0x7ff54af45332 [13164.144825] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [13164.144829] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [13164.144834] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [13164.144837] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [13164.144838] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [13164.144841] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [13164.144843] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [13164.777044] ? rescuer_thread+0x570/0x570 [13164.780806] kthread+0x1d1/0x200 [13164.784216] ? set_kthread_struct+0x70/0x70 [13164.785780] ret_from_fork+0x1f/0x30 [13174.145005] rcu: INFO: rcu_sched self-detected stall on CPU [13174.157042] rcu: 0-...!: (5947101 ticks this GP) idle=98a/1/0x4000000000000002 softirq=125154/125154 fqs=959 [13174.172947] (t=11220305 jiffies g=253585 q=6521) [13174.179437] rcu: rcu_sched kthread starved for 11215265 jiffies! g253585 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=3 [13174.189593] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [13174.210104] rcu: RCU grace-period kthread stack dump: [13174.218173] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [13174.230323] Call Trace: [13174.232136] __schedule+0x351/0xcb0 [13174.236976] schedule+0xc0/0x180 [13174.241977] schedule_timeout+0xb4/0x190 [13174.247339] ? do_raw_spin_unlock+0x75/0x190 [13174.252341] ? __next_timer_interrupt+0x160/0x160 [13174.260218] ? prepare_to_swait_event+0xf3/0x140 [13174.263315] rcu_gp_kthread+0x8dd/0x15b0 [13174.276080] ? force_qs_rnp+0x2c0/0x2c0 [13174.278862] kthread+0x1d1/0x200 [13174.282425] ? set_kthread_struct+0x70/0x70 [13174.289966] ret_from_fork+0x1f/0x30 [13174.292211] rcu: Stack dump where RCU GP kthread last ran: [13174.294241] Sending NMI from CPU 0 to CPUs 3: [13174.310776] NMI backtrace for cpu 3 [13174.310782] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13174.310784] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13174.310786] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13174.310787] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13174.310790] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000212 [13174.310792] RAX: 0000000000000483 RBX: ffff96848811cae0 RCX: 0000000000000001 [13174.310793] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [13174.310795] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [13174.310796] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [13174.310797] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [13174.310798] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [13174.310800] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13174.310801] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [13174.310801] Call Trace: [13174.310802] [13174.310803] ? show_regs.cold.9+0x22/0x2f [13174.310804] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [13174.310805] ? nmi_cpu_backtrace_handler+0x15/0x40 [13174.310806] ? nmi_handle+0xbd/0x210 [13174.310807] ? default_do_nmi+0x4b/0x1b0 [13174.310808] ? do_nmi+0x280/0x380 [13174.310809] ? end_repeat_nmi+0x16/0x1f [13174.310810] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13174.310811] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13174.310812] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13174.310813] [13174.310813] do_raw_spin_lock+0x177/0x190 [13174.310814] _raw_spin_lock+0x21/0x30 [13174.310815] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13174.310816] ? do_raw_spin_unlock+0x75/0x190 [13174.310816] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13174.310817] ? do_wait_intr+0xf0/0xf0 [13174.310818] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13174.310819] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [13174.310820] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13174.310821] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13174.310822] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13174.310822] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [13174.310823] mdc_intent_lock+0x353/0x7e0 [mdc] [13174.310824] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13174.310825] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13174.310826] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13174.310827] lmv_intent_open+0x338/0x1150 [lmv] [13174.310828] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13174.310828] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13174.310829] lmv_intent_lock+0x1dd/0x4c0 [lmv] [13174.310830] ll_intent_lock+0xc5/0xaa0 [lustre] [13174.310831] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13174.310832] ll_lookup_it+0xe0e/0x2270 [lustre] [13174.310832] ? do_raw_spin_unlock+0x75/0x190 [13174.310833] ? _raw_spin_unlock+0x12/0x30 [13174.310834] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [13174.310835] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [13174.310836] ll_atomic_open+0x36a/0xd70 [lustre] [13174.310836] lookup_open+0x356/0xb40 [13174.310837] do_last+0x313/0xfc0 [13174.310838] ? nd_jump_root+0xe5/0x160 [13174.310839] ? path_init+0x437/0x520 [13174.310839] path_openat+0xf7/0x500 [13174.310840] ? do_raw_spin_unlock+0x75/0x190 [13174.310841] do_filp_open+0x99/0x140 [13174.310842] ? getname_flags+0x6e/0x330 [13174.310843] ? __check_object_size+0xff/0x256 [13174.310844] ? do_raw_spin_unlock+0x75/0x190 [13174.310845] ? _raw_spin_unlock+0x12/0x30 [13174.310846] do_sys_openat2+0x2b4/0x410 [13174.310847] do_sys_open+0x73/0xa0 [13174.310848] __x64_sys_openat+0x24/0x30 [13174.310849] do_syscall_64+0xc1/0x440 [13174.310851] entry_SYSCALL_64_after_hwframe+0x49/0xae [13174.310852] RIP: 0033:0x7ff54af45332 [13174.310853] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [13174.310855] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [13174.310857] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [13174.310858] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [13174.310860] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [13174.310861] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [13174.310862] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [13174.313379] NMI backtrace for cpu 0 [13174.610916] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13174.616472] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13174.620252] Call Trace: [13174.621447] [13174.622811] ? dump_stack+0xbb/0x10e [13174.624567] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [13174.626861] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [13174.629274] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [13174.632047] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [13174.634362] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [13174.636104] ? rcu_dump_cpu_stacks+0x11b/0x171 [13174.637965] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [13174.640699] ? update_process_times+0x6d/0xc0 [13174.643090] ? tick_sched_handle.isra.18+0x27/0x90 [13174.645535] ? tick_sched_timer+0x53/0xb0 [13174.647644] ? __hrtimer_run_queues+0x1c2/0x4d0 [13174.649965] ? tick_sched_handle.isra.18+0x90/0x90 [13174.652280] ? hrtimer_interrupt+0x128/0x2c0 [13174.654278] ? smp_apic_timer_interrupt+0xa6/0x270 [13174.656891] ? apic_timer_interrupt+0xf/0x20 [13174.659011] [13174.660253] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [13174.662950] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13174.665671] ? do_raw_spin_lock+0x177/0x190 [13174.667621] ? _raw_spin_lock+0x21/0x30 [13174.669047] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13174.671161] ? obd_mod_free+0x46/0x140 [ptlrpc] [13174.673440] ? mdc_commit_open+0x91/0xb0 [mdc] [13174.675311] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13174.678427] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13174.681929] ? after_reply+0xe2c/0x1780 [ptlrpc] [13174.684743] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13174.687221] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [13174.689670] ? do_wait_intr+0xf0/0xf0 [13174.690912] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13174.693149] ? kthread+0x1d1/0x200 [13174.695294] ? set_kthread_struct+0x70/0x70 [13174.697035] ? ret_from_fork+0x1f/0x30 [13176.058056] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [13176.062187] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13176.083785] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13176.089127] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13176.092853] RIP: 0010:native_safe_halt+0x2e/0x50 [13176.094570] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13176.102281] RSP: 0018:ffffaf9fc1543ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13176.108280] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [13176.113097] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c20b4154 [13176.115581] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [13176.119132] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [13176.122171] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [13176.125387] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [13176.129019] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13176.131426] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [13176.134438] Call Trace: [13176.135671] [13176.136456] ? show_regs.cold.9+0x22/0x2f [13176.138525] ? watchdog_timer_fn.cold.6+0x62/0x10a [13176.140764] ? __hrtimer_run_queues+0x1c2/0x4d0 [13176.142841] ? watchdog+0x40/0x40 [13176.144092] ? hrtimer_interrupt+0x128/0x2c0 [13176.145432] ? smp_apic_timer_interrupt+0xa6/0x270 [13176.147276] ? apic_timer_interrupt+0xf/0x20 [13176.148900] [13176.149596] ? native_safe_halt+0x2e/0x50 [13176.152021] kvm_wait+0xcd/0xe0 [13176.156262] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13176.162957] do_raw_spin_lock+0x177/0x190 [13176.165223] _raw_spin_lock+0x21/0x30 [13176.167084] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13176.169179] ? do_raw_spin_unlock+0x75/0x190 [13176.171747] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13176.180989] ? do_wait_intr+0xf0/0xf0 [13176.183673] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13176.185458] mdc_close+0x53b/0xd70 [mdc] [13176.188216] lmv_close+0x18d/0x490 [lmv] [13176.196168] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [13176.200674] ll_close_inode_openhandle+0x493/0xea0 [lustre] [13176.205296] ll_md_real_close+0xa5/0x2a0 [lustre] [13176.209547] ll_md_close+0x274/0x800 [lustre] [13176.216793] ll_file_release+0x128/0xc50 [lustre] [13176.222756] __fput+0x10d/0x440 [13176.228908] ____fput+0x12/0x20 [13176.229915] task_work_run+0xc6/0x110 [13176.231019] exit_to_usermode_loop+0x1dd/0x1f0 [13176.232826] do_syscall_64+0x3d6/0x440 [13176.233927] entry_SYSCALL_64_after_hwframe+0x49/0xae [13176.235446] RIP: 0033:0x7ff54af45cd8 [13176.241875] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [13176.271213] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [13176.278256] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [13176.285684] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [13176.289599] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [13176.302155] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [13176.312910] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13192.059063] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [13192.062195] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13192.084631] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13192.091058] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13192.095649] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13192.101221] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13192.104642] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13192.113062] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13192.116414] RAX: 0000000000004d30 RBX: ffff96848811cae0 RCX: 0000000000000001 [13192.119328] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [13192.122330] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [13192.125265] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [13192.128232] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [13192.131281] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [13192.135017] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13192.138575] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [13192.146816] Call Trace: [13192.147889] [13192.148562] ? show_regs.cold.9+0x22/0x2f [13192.154539] ? watchdog_timer_fn.cold.6+0x62/0x10a [13192.159132] ? __hrtimer_run_queues+0x1c2/0x4d0 [13192.162683] BUG: workqueue lockup - pool [13192.163063] ? watchdog+0x40/0x40 [13192.163082] ? hrtimer_interrupt+0x128/0x2c0 [13192.163088] ? smp_apic_timer_interrupt+0xa6/0x270 [13192.163098] ? apic_timer_interrupt+0xf/0x20 [13192.163103] [13192.163108] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13192.163115] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13192.163123] do_raw_spin_lock+0x177/0x190 [13192.163130] _raw_spin_lock+0x21/0x30 [13192.163135] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13192.163431] process_one_work+0x2c8/0x700 [13192.163438] worker_thread+0x296/0x6e0 [13192.163442] ? rescuer_thread+0x570/0x570 [13192.163447] kthread+0x1d1/0x200 [13192.163451] ? set_kthread_struct+0x70/0x70 [13192.163455] ret_from_fork+0x1f/0x30 [13192.214253] cpus=0-3 flags=0x4 nice=0 stuck for 11233s! [13192.216154] Showing busy workqueues and worker pools: [13192.218371] workqueue events: flags=0x0 [13192.220590] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13192.224389] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [13192.229181] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [13192.232361] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [13192.237919] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13192.241060] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [13192.244485] workqueue events_long: flags=0x0 [13192.246556] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13192.250546] pending: lu_objects_flush [obdclass] [13192.254333] workqueue events_unbound: flags=0x2 [13192.259763] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [13192.266606] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [13192.278604] workqueue events_power_efficient: flags=0x80 [13192.282100] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13192.291397] pending: neigh_periodic_work [13192.292950] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13192.299184] pending: do_cache_clean, check_lifetime [13192.304994] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13192.310113] pending: neigh_periodic_work [13192.314676] workqueue writeback: flags=0x4a [13192.317211] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [13192.322193] pending: wb_workfn, wb_workfn [13192.325224] workqueue xprtiod: flags=0xa [13192.327744] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [13192.330665] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [13192.334487] workqueue ipv6_addrconf: flags=0x40008 [13192.336993] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13192.341591] pending: addrconf_verify_work [13192.346524] workqueue knbd0-recv: flags=0x1a [13192.350196] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13192.353496] in-flight: 657:recv_work [13192.356315] workqueue ptlrpc_pinger: flags=0x2 [13192.358204] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13192.360392] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [13192.362834] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11233s workers=4 idle: 20831 12642 28859 [13192.366079] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13182s workers=2 idle: 659 [13200.066475] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [13200.075608] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13200.122705] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13200.137357] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13200.141214] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13200.151107] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13200.169629] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13200.175200] RAX: 0000000000007f19 RBX: ffff96848811cae0 RCX: 0000000000000001 [13200.180170] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [13200.190463] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [13200.199817] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [13200.205077] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [13200.212856] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [13200.223268] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13200.235635] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [13200.243121] Call Trace: [13200.246556] [13200.248524] ? show_regs.cold.9+0x22/0x2f [13200.253304] ? watchdog_timer_fn.cold.6+0x62/0x10a [13200.255954] ? __hrtimer_run_queues+0x1c2/0x4d0 [13200.258447] ? watchdog+0x40/0x40 [13200.260377] ? hrtimer_interrupt+0x128/0x2c0 [13200.262935] ? smp_apic_timer_interrupt+0xa6/0x270 [13200.271188] ? apic_timer_interrupt+0xf/0x20 [13200.273721] [13200.274458] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13200.279631] do_raw_spin_lock+0x177/0x190 [13200.281870] _raw_spin_lock+0x21/0x30 [13200.286742] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13200.296533] ? do_raw_spin_unlock+0x75/0x190 [13200.307853] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13200.321848] ? do_wait_intr+0xf0/0xf0 [13200.323380] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13200.346170] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [13200.348101] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13200.356826] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13200.375221] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13200.391010] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [13200.392715] mdc_intent_lock+0x353/0x7e0 [mdc] [13200.402929] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13200.404375] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13200.409706] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13200.415837] lmv_intent_open+0x338/0x1150 [lmv] [13200.417768] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13200.435530] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13200.441502] lmv_intent_lock+0x1dd/0x4c0 [lmv] [13200.447427] ll_intent_lock+0xc5/0xaa0 [lustre] [13200.452402] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13200.460292] ll_lookup_it+0xe0e/0x2270 [lustre] [13200.464931] ? do_raw_spin_unlock+0x75/0x190 [13200.466322] ? _raw_spin_unlock+0x12/0x30 [13200.471481] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [13200.473354] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [13200.476982] ll_atomic_open+0x36a/0xd70 [lustre] [13200.483218] lookup_open+0x356/0xb40 [13200.486260] do_last+0x313/0xfc0 [13200.490881] ? nd_jump_root+0xe5/0x160 [13200.492838] ? path_init+0x437/0x520 [13200.495999] path_openat+0xf7/0x500 [13200.498077] ? do_raw_spin_unlock+0x75/0x190 [13200.504017] do_filp_open+0x99/0x140 [13200.508725] ? getname_flags+0x6e/0x330 [13200.510189] ? __check_object_size+0xff/0x256 [13200.523186] ? do_raw_spin_unlock+0x75/0x190 [13200.537086] ? _raw_spin_unlock+0x12/0x30 [13200.540239] do_sys_openat2+0x2b4/0x410 [13200.541625] do_sys_open+0x73/0xa0 [13200.549734] __x64_sys_openat+0x24/0x30 [13200.563486] do_syscall_64+0xc1/0x440 [13200.565302] entry_SYSCALL_64_after_hwframe+0x49/0xae [13200.567036] RIP: 0033:0x7ff54af45332 [13200.579449] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [13200.607917] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [13200.614533] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [13200.633459] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [13200.646973] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [13200.658875] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [13200.677219] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [13204.058242] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [13204.068915] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13204.118383] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13204.136687] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13204.142367] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13204.145536] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13204.176871] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13204.182692] RAX: 0000000000005304 RBX: ffff96848811cae0 RCX: 0000000000000001 [13204.186653] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [13204.195616] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [13204.208312] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [13204.215754] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [13204.224860] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [13204.238421] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13204.251765] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [13204.266312] Call Trace: [13204.269381] [13204.270501] ? show_regs.cold.9+0x22/0x2f [13204.273274] ? watchdog_timer_fn.cold.6+0x62/0x10a [13204.275547] ? __hrtimer_run_queues+0x1c2/0x4d0 [13204.277547] ? watchdog+0x40/0x40 [13204.280443] ? hrtimer_interrupt+0x128/0x2c0 [13204.284381] ? smp_apic_timer_interrupt+0xa6/0x270 [13204.290296] ? apic_timer_interrupt+0xf/0x20 [13204.295422] [13204.298659] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13204.306281] do_raw_spin_lock+0x177/0x190 [13204.308942] _raw_spin_lock+0x21/0x30 [13204.314522] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13204.317943] ? do_raw_spin_unlock+0x75/0x190 [13204.320596] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13204.327705] ? do_wait_intr+0xf0/0xf0 [13204.334055] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13204.342340] mdc_close+0x53b/0xd70 [mdc] [13204.349668] lmv_close+0x18d/0x490 [lmv] [13204.352763] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [13204.361659] ll_close_inode_openhandle+0x493/0xea0 [lustre] [13204.369685] ll_md_real_close+0xa5/0x2a0 [lustre] [13204.377731] ll_md_close+0x274/0x800 [lustre] [13204.384704] ll_file_release+0x128/0xc50 [lustre] [13204.390381] __fput+0x10d/0x440 [13204.399657] ____fput+0x12/0x20 [13204.403944] task_work_run+0xc6/0x110 [13204.406477] exit_to_usermode_loop+0x1dd/0x1f0 [13204.411451] do_syscall_64+0x3d6/0x440 [13204.420387] entry_SYSCALL_64_after_hwframe+0x49/0xae [13204.430373] RIP: 0033:0x7ff54af45cd8 [13204.437486] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [13204.452735] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [13204.457761] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [13204.465364] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [13204.475418] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [13204.480756] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [13204.495708] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13220.060221] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [13220.061790] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [13220.064856] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13220.064941] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13220.065925] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13220.065933] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13220.068637] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13220.068714] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13220.068723] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13220.068725] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [13220.068740] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [13220.068745] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13220.068749] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [13220.068752] RDX: 0000000000000e25 RSI: 0000000000000003 RDI: ffff96848811cae0 [13220.068754] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [13220.068757] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [13220.068759] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13220.068763] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [13220.068766] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13220.068768] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [13220.068777] Call Trace: [13220.068784] [13220.068789] ? show_regs.cold.9+0x22/0x2f [13220.068799] ? watchdog_timer_fn.cold.6+0x62/0x10a [13220.068805] ? __hrtimer_run_queues+0x1c2/0x4d0 [13220.069456] ? watchdog+0x40/0x40 [13220.069466] ? hrtimer_interrupt+0x128/0x2c0 [13220.069471] ? smp_apic_timer_interrupt+0xa6/0x270 [13220.069480] ? apic_timer_interrupt+0xf/0x20 [13220.069485] [13220.069489] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [13220.069495] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13220.069500] do_raw_spin_lock+0x177/0x190 [13220.069506] _raw_spin_lock+0x21/0x30 [13220.069513] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13220.069788] obd_mod_free+0x46/0x140 [ptlrpc] [13220.106158] [13220.106168] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13220.106185] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13220.106199] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13220.106204] RAX: 0000000000006410 RBX: ffff96848811cae0 RCX: 0000000000000001 [13220.106206] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [13220.106208] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [13220.106210] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [13220.106212] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [13220.106215] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [13220.106217] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13220.106220] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [13220.106228] Call Trace: [13220.106233] [13220.106239] ? show_regs.cold.9+0x22/0x2f [13220.106249] ? watchdog_timer_fn.cold.6+0x62/0x10a [13220.106257] ? __hrtimer_run_queues+0x1c2/0x4d0 [13220.106261] ? watchdog+0x40/0x40 [13220.106265] ? hrtimer_interrupt+0x128/0x2c0 [13220.106269] ? smp_apic_timer_interrupt+0xa6/0x270 [13220.106277] ? apic_timer_interrupt+0xf/0x20 [13220.106281] [13220.106284] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13220.106289] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13220.106296] do_raw_spin_lock+0x177/0x190 [13220.106301] _raw_spin_lock+0x21/0x30 [13220.106305] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13220.106811] process_one_work+0x2c8/0x700 [13220.106818] worker_thread+0x296/0x6e0 [13220.106822] ? rescuer_thread+0x570/0x570 [13220.106826] kthread+0x1d1/0x200 [13220.106832] ? set_kthread_struct+0x70/0x70 [13220.106835] ret_from_fork+0x1f/0x30 [13220.593522] mdc_commit_open+0x91/0xb0 [mdc] [13220.595259] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13220.601312] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13220.605380] after_reply+0xe2c/0x1780 [ptlrpc] [13220.609559] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13220.615314] ptlrpcd+0xa82/0xe40 [ptlrpc] [13220.619772] ? do_wait_intr+0xf0/0xf0 [13220.621680] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13220.624713] kthread+0x1d1/0x200 [13220.628617] ? set_kthread_struct+0x70/0x70 [13220.633414] ret_from_fork+0x1f/0x30 [13222.881216] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11264s! [13222.889207] Showing busy workqueues and worker pools: [13222.891311] workqueue events: flags=0x0 [13222.893961] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13222.896040] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [13222.911354] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [13222.914505] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [13222.919346] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13222.921894] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [13222.924815] workqueue events_long: flags=0x0 [13222.926730] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13222.930270] pending: lu_objects_flush [obdclass] [13222.933136] workqueue events_unbound: flags=0x2 [13222.936068] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [13222.939732] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [13222.945660] workqueue events_power_efficient: flags=0x80 [13222.948873] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13222.960723] pending: neigh_periodic_work [13222.963405] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13222.968278] pending: do_cache_clean, check_lifetime [13222.974487] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13222.981214] pending: neigh_periodic_work [13222.983772] workqueue writeback: flags=0x4a [13222.985886] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [13222.989887] pending: wb_workfn, wb_workfn [13222.995818] workqueue xprtiod: flags=0xa [13223.004976] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [13223.010476] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [13223.018329] workqueue ipv6_addrconf: flags=0x40008 [13223.021239] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13223.025508] pending: addrconf_verify_work [13223.027380] workqueue knbd0-recv: flags=0x1a [13223.033573] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13223.043571] in-flight: 657:recv_work [13223.045818] workqueue ptlrpc_pinger: flags=0x2 [13223.048477] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13223.051833] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [13223.060415] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11264s workers=4 idle: 20831 12642 28859 [13223.064974] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13213s workers=2 idle: 659 [13228.062547] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [13228.067303] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13228.101582] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13228.114332] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13228.119363] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13228.126359] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13228.144603] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [13228.151925] RAX: 0000000000007484 RBX: ffff96848811cae0 RCX: 0000000000000001 [13228.159293] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [13228.167174] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [13228.172663] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [13228.176217] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [13228.180638] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [13228.190669] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13228.197376] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [13228.207162] Call Trace: [13228.212617] [13228.216059] ? show_regs.cold.9+0x22/0x2f [13228.221102] ? watchdog_timer_fn.cold.6+0x62/0x10a [13228.227599] ? __hrtimer_run_queues+0x1c2/0x4d0 [13228.231630] ? watchdog+0x40/0x40 [13228.241397] ? hrtimer_interrupt+0x128/0x2c0 [13228.245737] ? smp_apic_timer_interrupt+0xa6/0x270 [13228.255131] ? apic_timer_interrupt+0xf/0x20 [13228.262499] [13228.263223] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13228.274392] do_raw_spin_lock+0x177/0x190 [13228.275677] _raw_spin_lock+0x21/0x30 [13228.276804] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13228.294237] ? do_raw_spin_unlock+0x75/0x190 [13228.300294] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13228.305371] ? do_wait_intr+0xf0/0xf0 [13228.307208] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13228.308770] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [13228.323045] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13228.330736] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13228.337668] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13228.343384] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [13228.346327] mdc_intent_lock+0x353/0x7e0 [mdc] [13228.352552] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13228.357745] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13228.364360] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13228.368165] lmv_intent_open+0x338/0x1150 [lmv] [13228.377163] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13228.382668] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13228.385911] lmv_intent_lock+0x1dd/0x4c0 [lmv] [13228.390596] ll_intent_lock+0xc5/0xaa0 [lustre] [13228.393750] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13228.397173] ll_lookup_it+0xe0e/0x2270 [lustre] [13228.400478] ? do_raw_spin_unlock+0x75/0x190 [13228.402513] ? _raw_spin_unlock+0x12/0x30 [13228.410148] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [13228.417298] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [13228.427431] ll_atomic_open+0x36a/0xd70 [lustre] [13228.428895] lookup_open+0x356/0xb40 [13228.430097] do_last+0x313/0xfc0 [13228.441037] ? nd_jump_root+0xe5/0x160 [13228.447219] ? path_init+0x437/0x520 [13228.454276] path_openat+0xf7/0x500 [13228.456122] ? do_raw_spin_unlock+0x75/0x190 [13228.460191] do_filp_open+0x99/0x140 [13228.462514] ? getname_flags+0x6e/0x330 [13228.464047] ? __check_object_size+0xff/0x256 [13228.465926] ? do_raw_spin_unlock+0x75/0x190 [13228.471609] ? _raw_spin_unlock+0x12/0x30 [13228.475620] do_sys_openat2+0x2b4/0x410 [13228.479188] do_sys_open+0x73/0xa0 [13228.482437] __x64_sys_openat+0x24/0x30 [13228.486505] do_syscall_64+0xc1/0x440 [13228.488288] entry_SYSCALL_64_after_hwframe+0x49/0xae [13228.493854] RIP: 0033:0x7ff54af45332 [13228.497028] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [13228.512885] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [13228.517465] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [13228.529833] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [13228.536552] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [13228.543093] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [13228.548316] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [13232.058019] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [13232.068219] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13232.162478] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13232.169632] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13232.173663] RIP: 0010:native_safe_halt+0x2e/0x50 [13232.175654] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13232.197653] RSP: 0018:ffffaf9fc1543ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13232.206054] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [13232.212585] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c20b4154 [13232.226401] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [13232.228108] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [13232.233476] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [13232.241654] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [13232.251218] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13232.253259] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [13232.260598] Call Trace: [13232.262176] [13232.263435] ? show_regs.cold.9+0x22/0x2f [13232.266176] ? watchdog_timer_fn.cold.6+0x62/0x10a [13232.268476] ? __hrtimer_run_queues+0x1c2/0x4d0 [13232.270475] ? watchdog+0x40/0x40 [13232.272140] ? hrtimer_interrupt+0x128/0x2c0 [13232.275167] ? smp_apic_timer_interrupt+0xa6/0x270 [13232.277376] ? apic_timer_interrupt+0xf/0x20 [13232.279536] [13232.280430] ? native_safe_halt+0x2e/0x50 [13232.281758] kvm_wait+0xcd/0xe0 [13232.283449] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13232.286070] do_raw_spin_lock+0x177/0x190 [13232.287820] _raw_spin_lock+0x21/0x30 [13232.289532] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13232.292412] ? do_raw_spin_unlock+0x75/0x190 [13232.294663] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13232.297204] ? do_wait_intr+0xf0/0xf0 [13232.298471] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13232.300183] mdc_close+0x53b/0xd70 [mdc] [13232.301263] lmv_close+0x18d/0x490 [lmv] [13232.302246] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [13232.305341] ll_close_inode_openhandle+0x493/0xea0 [lustre] [13232.309856] ll_md_real_close+0xa5/0x2a0 [lustre] [13232.314088] ll_md_close+0x274/0x800 [lustre] [13232.315897] ll_file_release+0x128/0xc50 [lustre] [13232.318742] __fput+0x10d/0x440 [13232.320139] ____fput+0x12/0x20 [13232.321371] task_work_run+0xc6/0x110 [13232.322818] exit_to_usermode_loop+0x1dd/0x1f0 [13232.325215] do_syscall_64+0x3d6/0x440 [13232.326746] entry_SYSCALL_64_after_hwframe+0x49/0xae [13232.328594] RIP: 0033:0x7ff54af45cd8 [13232.329923] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [13232.337763] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [13232.340751] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [13232.344101] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [13232.347090] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [13232.350148] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [13232.352882] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13248.055871] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [13248.059231] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [13248.059873] Modules linked in: [13248.067138] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13248.067389] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13248.067402] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13248.067410] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13248.073060] RIP: 0010:__pv_queued_spin_lock_slowpath+0x328/0x6d0 [13248.073080] Code: d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 05 18 d4 dc 02 01 e0 48 83 05 06 d4 dc 02 01 48 83 05 be d5 dc 02 01 8b 45 08 85 [13248.073085] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13248.073506] RAX: 000000000000479d RBX: ffff96848811cae0 RCX: 0000000000000001 [13248.073509] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [13248.073512] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [13248.073515] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [13248.073517] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [13248.073521] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [13248.073524] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13248.073527] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [13248.073536] Call Trace: [13248.073607] [13248.073615] ? show_regs.cold.9+0x22/0x2f [13248.073627] ? watchdog_timer_fn.cold.6+0x62/0x10a [13248.073635] ? __hrtimer_run_queues+0x1c2/0x4d0 [13248.073639] ? watchdog+0x40/0x40 [13248.073644] ? hrtimer_interrupt+0x128/0x2c0 [13248.073648] ? smp_apic_timer_interrupt+0xa6/0x270 [13248.073659] ? apic_timer_interrupt+0xf/0x20 [13248.073958] [13248.073962] ? __pv_queued_spin_lock_slowpath+0x328/0x6d0 [13248.073968] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13248.073975] do_raw_spin_lock+0x177/0x190 [13248.073981] _raw_spin_lock+0x21/0x30 [13248.073986] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13248.075056] process_one_work+0x2c8/0x700 [13248.075066] worker_thread+0x296/0x6e0 [13248.075071] ? rescuer_thread+0x570/0x570 [13248.075075] kthread+0x1d1/0x200 [13248.075080] ? set_kthread_struct+0x70/0x70 [13248.075084] ret_from_fork+0x1f/0x30 [13248.197589] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13248.217728] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13248.222722] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13248.226277] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [13248.228740] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [13248.237310] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13248.240366] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [13248.243240] RDX: 0000000000005de8 RSI: 0000000000000003 RDI: ffff96848811cae0 [13248.246705] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [13248.249634] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [13248.252943] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13248.256255] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [13248.259983] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13248.263043] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [13248.266263] Call Trace: [13248.267359] [13248.268185] ? show_regs.cold.9+0x22/0x2f [13248.270109] ? watchdog_timer_fn.cold.6+0x62/0x10a [13248.272106] ? __hrtimer_run_queues+0x1c2/0x4d0 [13248.274003] ? watchdog+0x40/0x40 [13248.275377] ? hrtimer_interrupt+0x128/0x2c0 [13248.277065] ? smp_apic_timer_interrupt+0xa6/0x270 [13248.279160] ? apic_timer_interrupt+0xf/0x20 [13248.281992] [13248.282667] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [13248.285045] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13248.288081] do_raw_spin_lock+0x177/0x190 [13248.290394] _raw_spin_lock+0x21/0x30 [13248.293382] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13248.297138] obd_mod_free+0x46/0x140 [ptlrpc] [13248.299838] mdc_commit_open+0x91/0xb0 [mdc] [13248.301885] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13248.306358] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13248.309326] after_reply+0xe2c/0x1780 [ptlrpc] [13248.312387] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13248.315651] ptlrpcd+0xa82/0xe40 [ptlrpc] [13248.317718] ? do_wait_intr+0xf0/0xf0 [13248.319358] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13248.322099] kthread+0x1d1/0x200 [13248.323422] ? set_kthread_struct+0x70/0x70 [13248.325110] ret_from_fork+0x1f/0x30 [13253.091921] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11294s! [13253.112768] Showing busy workqueues and worker pools: [13253.117926] workqueue events: flags=0x0 [13253.129217] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13253.140130] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [13253.171684] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [13253.187671] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [13253.221843] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13253.244764] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [13253.262882] workqueue events_long: flags=0x0 [13253.267845] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13253.281192] pending: lu_objects_flush [obdclass] [13253.292182] workqueue events_unbound: flags=0x2 [13253.302120] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [13253.315916] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [13253.336125] workqueue events_power_efficient: flags=0x80 [13253.345916] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13253.356179] pending: neigh_periodic_work [13253.359322] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13253.362857] pending: do_cache_clean, check_lifetime [13253.373488] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13253.379392] pending: neigh_periodic_work [13253.382046] workqueue writeback: flags=0x4a [13253.390075] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [13253.394928] pending: wb_workfn, wb_workfn [13253.397397] workqueue xprtiod: flags=0xa [13253.401304] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [13253.412569] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [13253.417906] workqueue ipv6_addrconf: flags=0x40008 [13253.423866] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13253.431242] pending: addrconf_verify_work [13253.435337] workqueue knbd0-recv: flags=0x1a [13253.438173] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13253.446348] in-flight: 657:recv_work [13253.449044] workqueue ptlrpc_pinger: flags=0x2 [13253.456034] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13253.462134] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [13253.471015] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11294s workers=4 idle: 20831 12642 28859 [13253.475637] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13243s workers=2 idle: 659 [13256.061466] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [13256.077196] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13256.179493] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13256.186460] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13256.194651] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13256.206692] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13256.225594] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13256.230094] RAX: 0000000000007dac RBX: ffff96848811cae0 RCX: 0000000000000001 [13256.238492] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [13256.245421] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [13256.251609] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [13256.267514] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [13256.273495] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [13256.278427] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13256.285525] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [13256.289537] Call Trace: [13256.290753] [13256.293539] ? show_regs.cold.9+0x22/0x2f [13256.295622] ? watchdog_timer_fn.cold.6+0x62/0x10a [13256.300205] ? __hrtimer_run_queues+0x1c2/0x4d0 [13256.306557] ? watchdog+0x40/0x40 [13256.308911] ? hrtimer_interrupt+0x128/0x2c0 [13256.311327] ? smp_apic_timer_interrupt+0xa6/0x270 [13256.313604] ? apic_timer_interrupt+0xf/0x20 [13256.315380] [13256.316636] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13256.322069] do_raw_spin_lock+0x177/0x190 [13256.328764] _raw_spin_lock+0x21/0x30 [13256.331569] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13256.339777] ? do_raw_spin_unlock+0x75/0x190 [13256.342636] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13256.349239] ? do_wait_intr+0xf0/0xf0 [13256.352510] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13256.359480] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [13256.366375] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13256.380118] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13256.388455] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13256.391304] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [13256.393623] mdc_intent_lock+0x353/0x7e0 [mdc] [13256.396428] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13256.401266] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13256.407168] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13256.410013] lmv_intent_open+0x338/0x1150 [lmv] [13256.411790] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13256.421829] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13256.429083] lmv_intent_lock+0x1dd/0x4c0 [lmv] [13256.430599] ll_intent_lock+0xc5/0xaa0 [lustre] [13256.433440] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13256.437177] ll_lookup_it+0xe0e/0x2270 [lustre] [13256.440263] ? do_raw_spin_unlock+0x75/0x190 [13256.442380] ? _raw_spin_unlock+0x12/0x30 [13256.444928] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [13256.446536] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [13256.449683] ll_atomic_open+0x36a/0xd70 [lustre] [13256.455002] lookup_open+0x356/0xb40 [13256.457192] do_last+0x313/0xfc0 [13256.460447] ? nd_jump_root+0xe5/0x160 [13256.462554] ? path_init+0x437/0x520 [13256.467068] path_openat+0xf7/0x500 [13256.471401] ? do_raw_spin_unlock+0x75/0x190 [13256.479531] do_filp_open+0x99/0x140 [13256.481647] ? getname_flags+0x6e/0x330 [13256.484506] ? __check_object_size+0xff/0x256 [13256.490615] ? do_raw_spin_unlock+0x75/0x190 [13256.495417] ? _raw_spin_unlock+0x12/0x30 [13256.501327] do_sys_openat2+0x2b4/0x410 [13256.505587] do_sys_open+0x73/0xa0 [13256.507863] __x64_sys_openat+0x24/0x30 [13256.510456] do_syscall_64+0xc1/0x440 [13256.512187] entry_SYSCALL_64_after_hwframe+0x49/0xae [13256.514586] RIP: 0033:0x7ff54af45332 [13256.519221] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [13256.530295] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [13256.536427] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [13256.542985] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [13256.551353] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [13256.566513] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [13256.579467] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [13260.062431] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [13260.077638] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13260.126488] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13260.147162] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13260.159385] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13260.161969] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13260.186533] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13260.202434] RAX: 0000000000005c12 RBX: ffff96848811cae0 RCX: 0000000000000001 [13260.209234] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [13260.214829] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [13260.231556] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [13260.233967] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [13260.243416] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [13260.254713] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13260.259913] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [13260.266403] Call Trace: [13260.271018] [13260.272312] ? show_regs.cold.9+0x22/0x2f [13260.275724] ? watchdog_timer_fn.cold.6+0x62/0x10a [13260.279577] ? __hrtimer_run_queues+0x1c2/0x4d0 [13260.283414] ? watchdog+0x40/0x40 [13260.285478] ? hrtimer_interrupt+0x128/0x2c0 [13260.291388] ? smp_apic_timer_interrupt+0xa6/0x270 [13260.298443] ? apic_timer_interrupt+0xf/0x20 [13260.301394] [13260.302165] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13260.313102] do_raw_spin_lock+0x177/0x190 [13260.316752] _raw_spin_lock+0x21/0x30 [13260.323585] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13260.326210] ? do_raw_spin_unlock+0x75/0x190 [13260.335675] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13260.341037] ? do_wait_intr+0xf0/0xf0 [13260.344624] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13260.349046] mdc_close+0x53b/0xd70 [mdc] [13260.353957] lmv_close+0x18d/0x490 [lmv] [13260.356387] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [13260.364821] ll_close_inode_openhandle+0x493/0xea0 [lustre] [13260.371665] ll_md_real_close+0xa5/0x2a0 [lustre] [13260.375680] ll_md_close+0x274/0x800 [lustre] [13260.382780] ll_file_release+0x128/0xc50 [lustre] [13260.389274] __fput+0x10d/0x440 [13260.391486] ____fput+0x12/0x20 [13260.393357] task_work_run+0xc6/0x110 [13260.406414] exit_to_usermode_loop+0x1dd/0x1f0 [13260.409542] do_syscall_64+0x3d6/0x440 [13260.411031] entry_SYSCALL_64_after_hwframe+0x49/0xae [13260.420528] RIP: 0033:0x7ff54af45cd8 [13260.423399] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [13260.447238] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [13260.451644] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [13260.472682] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [13260.475081] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [13260.486342] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [13260.488334] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13276.060705] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [13276.065896] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13276.106146] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13276.113586] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13276.132905] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13276.141546] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13276.153962] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13276.170582] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13276.176747] RAX: 0000000000007bab RBX: ffff96848811cae0 RCX: 0000000000000001 [13276.182213] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [13276.188879] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [13276.193240] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [13276.196847] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [13276.200318] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [13276.204034] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13276.206387] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [13276.209349] Call Trace: [13276.213211] [13276.213949] ? show_regs.cold.9+0x22/0x2f [13276.215320] ? watchdog_timer_fn.cold.6+0x62/0x10a [13276.222051] ? __hrtimer_run_queues+0x1c2/0x4d0 [13276.227362] ? watchdog+0x40/0x40 [13276.233782] ? hrtimer_interrupt+0x128/0x2c0 [13276.239778] ? smp_apic_timer_interrupt+0xa6/0x270 [13276.250483] ? apic_timer_interrupt+0xf/0x20 [13276.256361] [13276.258261] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13276.260082] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13276.270442] do_raw_spin_lock+0x177/0x190 [13276.275962] _raw_spin_lock+0x21/0x30 [13276.278861] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13276.281962] process_one_work+0x2c8/0x700 [13276.284117] worker_thread+0x296/0x6e0 [13276.286196] ? rescuer_thread+0x570/0x570 [13276.288340] kthread+0x1d1/0x200 [13276.289751] ? set_kthread_struct+0x70/0x70 [13276.291988] ret_from_fork+0x1f/0x30 [13280.060108] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [13280.066394] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13280.143447] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13280.150321] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13280.164303] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [13280.166794] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [13280.175393] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13280.178153] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [13280.181831] RDX: 0000000000006706 RSI: 0000000000000003 RDI: ffff96848811cae0 [13280.185483] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [13280.188593] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [13280.191350] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13280.195368] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [13280.198649] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13280.201059] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [13280.203871] Call Trace: [13280.204805] [13280.205628] ? show_regs.cold.9+0x22/0x2f [13280.207187] ? watchdog_timer_fn.cold.6+0x62/0x10a [13280.209055] ? __hrtimer_run_queues+0x1c2/0x4d0 [13280.210988] ? watchdog+0x40/0x40 [13280.212341] ? hrtimer_interrupt+0x128/0x2c0 [13280.214136] ? smp_apic_timer_interrupt+0xa6/0x270 [13280.216047] ? apic_timer_interrupt+0xf/0x20 [13280.217784] [13280.218652] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [13280.220771] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13280.223087] do_raw_spin_lock+0x177/0x190 [13280.224621] _raw_spin_lock+0x21/0x30 [13280.225981] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13280.228211] obd_mod_free+0x46/0x140 [ptlrpc] [13280.230255] mdc_commit_open+0x91/0xb0 [mdc] [13280.231950] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13280.234169] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13280.236893] after_reply+0xe2c/0x1780 [ptlrpc] [13280.239054] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13280.241401] ptlrpcd+0xa82/0xe40 [ptlrpc] [13280.243237] ? do_wait_intr+0xf0/0xf0 [13280.244780] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13280.247074] kthread+0x1d1/0x200 [13280.248366] ? set_kthread_struct+0x70/0x70 [13280.250087] ret_from_fork+0x1f/0x30 [13283.808301] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11324s! [13283.818055] Showing busy workqueues and worker pools: [13283.821855] workqueue events: flags=0x0 [13283.827775] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13283.836549] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [13283.843441] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [13283.855034] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [13283.862302] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13283.868275] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [13283.876105] workqueue events_long: flags=0x0 [13283.879279] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13283.885055] pending: lu_objects_flush [obdclass] [13283.893082] workqueue events_unbound: flags=0x2 [13283.895617] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [13283.898903] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [13283.904820] workqueue events_power_efficient: flags=0x80 [13283.907395] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13283.910697] pending: neigh_periodic_work [13283.913432] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13283.916729] pending: do_cache_clean, check_lifetime [13283.919234] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13283.922157] pending: neigh_periodic_work [13283.924151] workqueue writeback: flags=0x4a [13283.926881] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [13283.935038] pending: wb_workfn, wb_workfn [13283.937024] workqueue xprtiod: flags=0xa [13283.939825] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [13283.944910] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [13283.949748] workqueue ipv6_addrconf: flags=0x40008 [13283.952041] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13283.954613] pending: addrconf_verify_work [13283.959371] workqueue knbd0-recv: flags=0x1a [13283.961658] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13283.965670] in-flight: 657:recv_work [13283.967743] workqueue ptlrpc_pinger: flags=0x2 [13283.970189] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13283.973889] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [13283.976204] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11325s workers=4 idle: 20831 12642 28859 [13283.980263] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13274s workers=2 idle: 659 [13284.060981] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [13284.075974] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13284.111245] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13284.117577] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13284.124901] RIP: 0010:native_safe_halt+0x2e/0x50 [13284.129445] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13284.137454] RSP: 0000:ffffaf9fc15234f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13284.142863] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [13284.147649] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [13284.154739] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [13284.159287] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [13284.166111] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [13284.176104] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [13284.189283] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13284.195344] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [13284.200950] Call Trace: [13284.205646] [13284.206833] ? show_regs.cold.9+0x22/0x2f [13284.211859] ? watchdog_timer_fn.cold.6+0x62/0x10a [13284.215909] ? __hrtimer_run_queues+0x1c2/0x4d0 [13284.223275] ? watchdog+0x40/0x40 [13284.225328] ? hrtimer_interrupt+0x128/0x2c0 [13284.227922] ? smp_apic_timer_interrupt+0xa6/0x270 [13284.232539] ? apic_timer_interrupt+0xf/0x20 [13284.235685] [13284.238983] ? native_safe_halt+0x2e/0x50 [13284.241615] kvm_wait+0xcd/0xe0 [13284.243197] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13284.246261] do_raw_spin_lock+0x177/0x190 [13284.251087] _raw_spin_lock+0x21/0x30 [13284.254148] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13284.259401] ? do_raw_spin_unlock+0x75/0x190 [13284.264271] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13284.270118] ? do_wait_intr+0xf0/0xf0 [13284.272143] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13284.274631] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [13284.283105] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13284.287898] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13284.291288] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13284.295141] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [13284.299260] mdc_intent_lock+0x353/0x7e0 [mdc] [13284.301294] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13284.305189] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13284.311652] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13284.315243] lmv_intent_open+0x338/0x1150 [lmv] [13284.317690] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13284.320993] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13284.330368] lmv_intent_lock+0x1dd/0x4c0 [lmv] [13284.333220] ll_intent_lock+0xc5/0xaa0 [lustre] [13284.336206] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13284.340599] ll_lookup_it+0xe0e/0x2270 [lustre] [13284.343279] ? do_raw_spin_unlock+0x75/0x190 [13284.347312] ? _raw_spin_unlock+0x12/0x30 [13284.349787] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [13284.353698] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [13284.357914] ll_atomic_open+0x36a/0xd70 [lustre] [13284.360641] lookup_open+0x356/0xb40 [13284.365159] do_last+0x313/0xfc0 [13284.366950] ? nd_jump_root+0xe5/0x160 [13284.368924] ? path_init+0x437/0x520 [13284.372891] path_openat+0xf7/0x500 [13284.376958] ? do_raw_spin_unlock+0x75/0x190 [13284.379142] do_filp_open+0x99/0x140 [13284.384069] ? getname_flags+0x6e/0x330 [13284.386873] ? __check_object_size+0xff/0x256 [13284.390220] ? do_raw_spin_unlock+0x75/0x190 [13284.393132] ? _raw_spin_unlock+0x12/0x30 [13284.395754] do_sys_openat2+0x2b4/0x410 [13284.397284] do_sys_open+0x73/0xa0 [13284.399923] __x64_sys_openat+0x24/0x30 [13284.404714] do_syscall_64+0xc1/0x440 [13284.407309] entry_SYSCALL_64_after_hwframe+0x49/0xae [13284.411441] RIP: 0033:0x7ff54af45332 [13284.412908] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [13284.422229] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [13284.426523] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [13284.432371] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [13284.438916] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [13284.446677] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [13284.449667] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [13288.059255] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [13288.069580] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13288.108745] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13288.118163] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13288.127296] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13288.133776] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13288.146343] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13288.154961] RAX: 0000000000007f83 RBX: ffff96848811cae0 RCX: 0000000000000001 [13288.159342] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [13288.163767] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [13288.167847] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [13288.171722] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [13288.174262] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [13288.179109] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13288.181435] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [13288.186332] Call Trace: [13288.187820] [13288.188785] ? show_regs.cold.9+0x22/0x2f [13288.191106] ? watchdog_timer_fn.cold.6+0x62/0x10a [13288.193281] ? __hrtimer_run_queues+0x1c2/0x4d0 [13288.195213] ? watchdog+0x40/0x40 [13288.196432] ? hrtimer_interrupt+0x128/0x2c0 [13288.198497] ? smp_apic_timer_interrupt+0xa6/0x270 [13288.200393] ? apic_timer_interrupt+0xf/0x20 [13288.202282] [13288.203349] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13288.205526] do_raw_spin_lock+0x177/0x190 [13288.207310] _raw_spin_lock+0x21/0x30 [13288.213979] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13288.219688] ? do_raw_spin_unlock+0x75/0x190 [13288.222696] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13288.227255] ? do_wait_intr+0xf0/0xf0 [13288.229060] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13288.232265] mdc_close+0x53b/0xd70 [mdc] [13288.234477] lmv_close+0x18d/0x490 [lmv] [13288.236427] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [13288.239112] ll_close_inode_openhandle+0x493/0xea0 [lustre] [13288.243706] ll_md_real_close+0xa5/0x2a0 [lustre] [13288.246884] ll_md_close+0x274/0x800 [lustre] [13288.250159] ll_file_release+0x128/0xc50 [lustre] [13288.253127] __fput+0x10d/0x440 [13288.254882] ____fput+0x12/0x20 [13288.256310] task_work_run+0xc6/0x110 [13288.258216] exit_to_usermode_loop+0x1dd/0x1f0 [13288.262068] do_syscall_64+0x3d6/0x440 [13288.265599] entry_SYSCALL_64_after_hwframe+0x49/0xae [13288.268545] RIP: 0033:0x7ff54af45cd8 [13288.276136] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [13288.289389] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [13288.293849] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [13288.298217] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [13288.303094] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [13288.306868] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [13288.311257] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13304.061923] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [13304.073628] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13304.120913] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13304.131441] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13304.141183] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13304.147922] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13304.154131] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13304.188903] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13304.195994] RAX: 0000000000005718 RBX: ffff96848811cae0 RCX: 0000000000000001 [13304.200189] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [13304.204700] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [13304.208521] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [13304.214779] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [13304.221673] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [13304.228715] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13304.238818] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [13304.254344] Call Trace: [13304.258468] [13304.261556] ? show_regs.cold.9+0x22/0x2f [13304.265704] ? watchdog_timer_fn.cold.6+0x62/0x10a [13304.272901] ? __hrtimer_run_queues+0x1c2/0x4d0 [13304.276884] ? watchdog+0x40/0x40 [13304.280822] ? hrtimer_interrupt+0x128/0x2c0 [13304.292703] ? smp_apic_timer_interrupt+0xa6/0x270 [13304.300584] ? apic_timer_interrupt+0xf/0x20 [13304.304817] [13304.308580] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13304.320813] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13304.331649] do_raw_spin_lock+0x177/0x190 [13304.339042] _raw_spin_lock+0x21/0x30 [13304.342722] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13304.346271] process_one_work+0x2c8/0x700 [13304.349545] worker_thread+0x296/0x6e0 [13304.354187] ? rescuer_thread+0x570/0x570 [13304.363357] kthread+0x1d1/0x200 [13304.369750] ? set_kthread_struct+0x70/0x70 [13304.375466] ret_from_fork+0x1f/0x30 [13308.055245] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [13308.058905] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13308.093948] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13308.098476] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13308.102240] RIP: 0010:native_safe_halt+0x2e/0x50 [13308.104293] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13308.114624] RSP: 0000:ffffaf9fc1b97bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13308.118727] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [13308.122819] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff96848811cae0 [13308.126382] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [13308.129264] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [13308.132279] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13308.135135] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [13308.138239] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13308.140518] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [13308.143607] Call Trace: [13308.145654] [13308.146433] ? show_regs.cold.9+0x22/0x2f [13308.148061] ? watchdog_timer_fn.cold.6+0x62/0x10a [13308.150064] ? __hrtimer_run_queues+0x1c2/0x4d0 [13308.151927] ? watchdog+0x40/0x40 [13308.153299] ? hrtimer_interrupt+0x128/0x2c0 [13308.154980] ? smp_apic_timer_interrupt+0xa6/0x270 [13308.156907] ? apic_timer_interrupt+0xf/0x20 [13308.158594] [13308.159378] ? native_safe_halt+0x2e/0x50 [13308.161001] kvm_wait+0xcd/0xe0 [13308.162301] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13308.164494] do_raw_spin_lock+0x177/0x190 [13308.166105] _raw_spin_lock+0x21/0x30 [13308.167653] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13308.169754] obd_mod_free+0x46/0x140 [ptlrpc] [13308.171819] mdc_commit_open+0x91/0xb0 [mdc] [13308.173668] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13308.176027] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13308.178513] after_reply+0xe2c/0x1780 [ptlrpc] [13308.180680] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13308.182978] ptlrpcd+0xa82/0xe40 [ptlrpc] [13308.184786] ? do_wait_intr+0xf0/0xf0 [13308.186210] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13308.188490] kthread+0x1d1/0x200 [13308.189696] ? set_kthread_struct+0x70/0x70 [13308.191213] ret_from_fork+0x1f/0x30 [13312.060082] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [13312.065478] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13312.120002] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13312.141727] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13312.164709] RIP: 0010:native_safe_halt+0x2e/0x50 [13312.166345] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13312.185120] RSP: 0000:ffffaf9fc15234f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13312.189764] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [13312.197566] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c21b4154 [13312.202745] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [13312.209200] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [13312.215492] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [13312.225534] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [13312.239505] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13312.248644] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [13312.259011] Call Trace: [13312.261525] [13312.265580] ? show_regs.cold.9+0x22/0x2f [13312.268027] ? watchdog_timer_fn.cold.6+0x62/0x10a [13312.273654] ? __hrtimer_run_queues+0x1c2/0x4d0 [13312.277544] ? watchdog+0x40/0x40 [13312.280238] ? hrtimer_interrupt+0x128/0x2c0 [13312.284270] ? smp_apic_timer_interrupt+0xa6/0x270 [13312.289966] ? apic_timer_interrupt+0xf/0x20 [13312.292864] [13312.298932] ? native_safe_halt+0x2e/0x50 [13312.306759] kvm_wait+0xcd/0xe0 [13312.308955] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13312.312272] do_raw_spin_lock+0x177/0x190 [13312.316542] _raw_spin_lock+0x21/0x30 [13312.318151] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13312.330746] ? do_raw_spin_unlock+0x75/0x190 [13312.332215] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13312.349593] ? do_wait_intr+0xf0/0xf0 [13312.356822] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13312.376694] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [13312.381991] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13312.399788] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13312.410525] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13312.418322] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [13312.428477] mdc_intent_lock+0x353/0x7e0 [mdc] [13312.433670] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13312.440504] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13312.447437] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13312.457778] lmv_intent_open+0x338/0x1150 [lmv] [13312.463512] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13312.473326] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13312.481702] lmv_intent_lock+0x1dd/0x4c0 [lmv] [13312.487613] ll_intent_lock+0xc5/0xaa0 [lustre] [13312.493263] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13312.501215] ll_lookup_it+0xe0e/0x2270 [lustre] [13312.505451] ? do_raw_spin_unlock+0x75/0x190 [13312.512366] ? _raw_spin_unlock+0x12/0x30 [13312.513753] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [13312.523512] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [13312.527591] ll_atomic_open+0x36a/0xd70 [lustre] [13312.532706] lookup_open+0x356/0xb40 [13312.539795] do_last+0x313/0xfc0 [13312.544696] ? nd_jump_root+0xe5/0x160 [13312.551827] ? path_init+0x437/0x520 [13312.557397] path_openat+0xf7/0x500 [13312.560277] ? do_raw_spin_unlock+0x75/0x190 [13312.564791] do_filp_open+0x99/0x140 [13312.567494] ? getname_flags+0x6e/0x330 [13312.576652] ? __check_object_size+0xff/0x256 [13312.578851] ? do_raw_spin_unlock+0x75/0x190 [13312.582684] ? _raw_spin_unlock+0x12/0x30 [13312.585327] do_sys_openat2+0x2b4/0x410 [13312.588525] do_sys_open+0x73/0xa0 [13312.593952] __x64_sys_openat+0x24/0x30 [13312.598540] do_syscall_64+0xc1/0x440 [13312.601653] entry_SYSCALL_64_after_hwframe+0x49/0xae [13312.609312] RIP: 0033:0x7ff54af45332 [13312.610551] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [13312.651714] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [13312.665701] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [13312.674670] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [13312.687267] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [13312.700310] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [13312.707418] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [13314.015038] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11355s! [13314.018287] Showing busy workqueues and worker pools: [13314.020187] workqueue events: flags=0x0 [13314.021709] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13314.024207] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [13314.027154] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [13314.029273] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [13314.033444] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13314.035934] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [13314.038711] workqueue events_long: flags=0x0 [13314.040627] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13314.043470] pending: lu_objects_flush [obdclass] [13314.045440] workqueue events_unbound: flags=0x2 [13314.047623] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [13314.050373] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [13314.054451] workqueue events_power_efficient: flags=0x80 [13314.056666] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13314.059011] pending: neigh_periodic_work [13314.060729] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13314.063214] pending: do_cache_clean, check_lifetime [13314.065523] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13314.068085] pending: neigh_periodic_work [13314.070050] workqueue writeback: flags=0x4a [13314.071696] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [13314.074185] pending: wb_workfn, wb_workfn [13314.075861] workqueue xprtiod: flags=0xa [13314.077562] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [13314.079951] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [13314.083234] workqueue ipv6_addrconf: flags=0x40008 [13314.085033] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13314.087443] pending: addrconf_verify_work [13314.089016] workqueue knbd0-recv: flags=0x1a [13314.090865] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13314.093360] in-flight: 657:recv_work [13314.095066] workqueue ptlrpc_pinger: flags=0x2 [13314.096797] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13314.098886] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [13314.101217] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11355s workers=4 idle: 20831 12642 28859 [13314.104230] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13304s workers=2 idle: 659 [13316.058392] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [13316.065775] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13316.104413] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13316.112965] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13316.120013] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13316.125541] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13316.140762] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13316.150749] RAX: 000000000000402a RBX: ffff96848811cae0 RCX: 0000000000000001 [13316.154637] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [13316.162277] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [13316.166657] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [13316.170822] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [13316.177232] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [13316.181459] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13316.182702] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [13316.185858] Call Trace: [13316.187416] [13316.188234] ? show_regs.cold.9+0x22/0x2f [13316.190536] ? watchdog_timer_fn.cold.6+0x62/0x10a [13316.195612] ? __hrtimer_run_queues+0x1c2/0x4d0 [13316.200424] ? watchdog+0x40/0x40 [13316.201860] ? hrtimer_interrupt+0x128/0x2c0 [13316.204331] ? smp_apic_timer_interrupt+0xa6/0x270 [13316.208493] ? apic_timer_interrupt+0xf/0x20 [13316.210651] [13316.211552] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13316.214721] do_raw_spin_lock+0x177/0x190 [13316.218641] _raw_spin_lock+0x21/0x30 [13316.221133] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13316.223558] ? do_raw_spin_unlock+0x75/0x190 [13316.226139] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13316.231419] ? do_wait_intr+0xf0/0xf0 [13316.233579] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13316.236384] mdc_close+0x53b/0xd70 [mdc] [13316.241359] lmv_close+0x18d/0x490 [lmv] [13316.244409] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [13316.247851] ll_close_inode_openhandle+0x493/0xea0 [lustre] [13316.260362] ll_md_real_close+0xa5/0x2a0 [lustre] [13316.268614] ll_md_close+0x274/0x800 [lustre] [13316.275060] ll_file_release+0x128/0xc50 [lustre] [13316.278461] __fput+0x10d/0x440 [13316.279922] ____fput+0x12/0x20 [13316.281426] task_work_run+0xc6/0x110 [13316.282967] exit_to_usermode_loop+0x1dd/0x1f0 [13316.284822] do_syscall_64+0x3d6/0x440 [13316.286680] entry_SYSCALL_64_after_hwframe+0x49/0xae [13316.292236] RIP: 0033:0x7ff54af45cd8 [13316.294422] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [13316.307068] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [13316.311548] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [13316.316694] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [13316.324045] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [13316.332397] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [13316.336742] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13332.068311] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [13332.074646] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13332.125072] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13332.142468] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13332.158664] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13332.162726] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13332.164959] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13332.186442] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13332.189809] RAX: 0000000000005378 RBX: ffff96848811cae0 RCX: 0000000000000001 [13332.193502] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [13332.196827] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [13332.200413] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [13332.203682] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [13332.207169] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [13332.210650] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13332.213192] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [13332.216178] Call Trace: [13332.217276] [13332.217906] ? show_regs.cold.9+0x22/0x2f [13332.219450] ? watchdog_timer_fn.cold.6+0x62/0x10a [13332.221715] ? __hrtimer_run_queues+0x1c2/0x4d0 [13332.223804] ? watchdog+0x40/0x40 [13332.225340] ? hrtimer_interrupt+0x128/0x2c0 [13332.227374] ? smp_apic_timer_interrupt+0xa6/0x270 [13332.229607] ? apic_timer_interrupt+0xf/0x20 [13332.231879] [13332.232701] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13332.235148] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13332.237737] do_raw_spin_lock+0x177/0x190 [13332.239486] _raw_spin_lock+0x21/0x30 [13332.241178] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13332.243657] process_one_work+0x2c8/0x700 [13332.245274] worker_thread+0x296/0x6e0 [13332.247099] ? rescuer_thread+0x570/0x570 [13332.248669] kthread+0x1d1/0x200 [13332.251789] ? set_kthread_struct+0x70/0x70 [13332.253676] ret_from_fork+0x1f/0x30 [13340.057009] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [13340.060989] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [13340.066460] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13340.066548] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13340.066556] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13340.066558] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [13340.066572] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [13340.066576] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13340.066580] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [13340.066583] RDX: 00000000000069a5 RSI: 0000000000000003 RDI: ffff96848811cae0 [13340.066585] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [13340.066587] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [13340.066590] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13340.066592] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [13340.066595] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13340.066597] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [13340.066606] Call Trace: [13340.066612] [13340.066618] ? show_regs.cold.9+0x22/0x2f [13340.066627] ? watchdog_timer_fn.cold.6+0x62/0x10a [13340.066634] ? __hrtimer_run_queues+0x1c2/0x4d0 [13340.066637] ? watchdog+0x40/0x40 [13340.066642] ? hrtimer_interrupt+0x128/0x2c0 [13340.066646] ? smp_apic_timer_interrupt+0xa6/0x270 [13340.066654] ? apic_timer_interrupt+0xf/0x20 [13340.066660] [13340.066663] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [13340.066668] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13340.066673] do_raw_spin_lock+0x177/0x190 [13340.066677] _raw_spin_lock+0x21/0x30 [13340.066682] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13340.066974] obd_mod_free+0x46/0x140 [ptlrpc] [13340.071805] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13340.072103] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13340.072110] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13340.072177] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13340.072193] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13340.072198] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13340.072202] RAX: 0000000000007f0f RBX: ffff96848811cae0 RCX: 0000000000000001 [13340.072205] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [13340.072207] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [13340.072209] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [13340.072590] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [13340.072594] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [13340.072597] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13340.072599] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [13340.072607] Call Trace: [13340.072614] [13340.072619] ? show_regs.cold.9+0x22/0x2f [13340.072631] ? watchdog_timer_fn.cold.6+0x62/0x10a [13340.072639] ? __hrtimer_run_queues+0x1c2/0x4d0 [13340.072642] ? watchdog+0x40/0x40 [13340.072647] ? hrtimer_interrupt+0x128/0x2c0 [13340.072651] ? smp_apic_timer_interrupt+0xa6/0x270 [13340.072661] ? apic_timer_interrupt+0xf/0x20 [13340.072666] [13340.072668] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13340.072676] do_raw_spin_lock+0x177/0x190 [13340.072681] _raw_spin_lock+0x21/0x30 [13340.072687] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13340.073172] ? do_raw_spin_unlock+0x75/0x190 [13340.073180] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13340.073374] ? do_wait_intr+0xf0/0xf0 [13340.073383] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13340.073647] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [13340.073983] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13340.074284] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13340.074409] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13340.074456] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [13340.074484] mdc_intent_lock+0x353/0x7e0 [mdc] [13340.074662] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13340.074749] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13340.075109] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13340.075151] lmv_intent_open+0x338/0x1150 [lmv] [13340.075185] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13340.080224] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13340.080311] lmv_intent_lock+0x1dd/0x4c0 [lmv] [13340.080346] ll_intent_lock+0xc5/0xaa0 [lustre] [13340.080431] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13340.080515] ll_lookup_it+0xe0e/0x2270 [lustre] [13340.080600] ? do_raw_spin_unlock+0x75/0x190 [13340.080609] ? _raw_spin_unlock+0x12/0x30 [13340.080616] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [13340.080642] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [13340.080661] ll_atomic_open+0x36a/0xd70 [lustre] [13340.080746] lookup_open+0x356/0xb40 [13340.080758] do_last+0x313/0xfc0 [13340.080763] ? nd_jump_root+0xe5/0x160 [13340.080768] ? path_init+0x437/0x520 [13340.080773] path_openat+0xf7/0x500 [13340.080777] ? do_raw_spin_unlock+0x75/0x190 [13340.080782] do_filp_open+0x99/0x140 [13340.080788] ? getname_flags+0x6e/0x330 [13340.080793] ? __check_object_size+0xff/0x256 [13340.080800] ? do_raw_spin_unlock+0x75/0x190 [13340.080805] ? _raw_spin_unlock+0x12/0x30 [13340.080809] do_sys_openat2+0x2b4/0x410 [13340.080814] do_sys_open+0x73/0xa0 [13340.080819] __x64_sys_openat+0x24/0x30 [13340.080823] do_syscall_64+0xc1/0x440 [13340.080831] entry_SYSCALL_64_after_hwframe+0x49/0xae [13340.080837] RIP: 0033:0x7ff54af45332 [13340.080845] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [13340.080850] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [13340.082109] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [13340.082114] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [13340.082117] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [13340.082119] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [13340.082122] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [13340.562413] mdc_commit_open+0x91/0xb0 [mdc] [13340.571728] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13340.576642] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13340.578728] after_reply+0xe2c/0x1780 [ptlrpc] [13340.581604] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13340.584539] ptlrpcd+0xa82/0xe40 [ptlrpc] [13340.586493] ? do_wait_intr+0xf0/0xf0 [13340.589375] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13340.595502] kthread+0x1d1/0x200 [13340.598538] ? set_kthread_struct+0x70/0x70 [13340.602207] ret_from_fork+0x1f/0x30 [13344.059112] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [13344.063136] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13344.091669] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13344.104482] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13344.112114] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13344.118339] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13344.135779] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13344.140690] RAX: 0000000000007c38 RBX: ffff96848811cae0 RCX: 0000000000000001 [13344.145869] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [13344.150035] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [13344.153997] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [13344.160877] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [13344.163764] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [13344.168751] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13344.173179] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [13344.177349] Call Trace: [13344.179096] [13344.183159] ? show_regs.cold.9+0x22/0x2f [13344.186271] ? watchdog_timer_fn.cold.6+0x62/0x10a [13344.190557] ? __hrtimer_run_queues+0x1c2/0x4d0 [13344.194145] ? watchdog+0x40/0x40 [13344.197105] ? hrtimer_interrupt+0x128/0x2c0 [13344.200349] ? smp_apic_timer_interrupt+0xa6/0x270 [13344.203338] ? apic_timer_interrupt+0xf/0x20 [13344.206554] [13344.208018] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13344.211075] do_raw_spin_lock+0x177/0x190 [13344.216722] _raw_spin_lock+0x21/0x30 [13344.219626] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13344.223202] ? do_raw_spin_unlock+0x75/0x190 [13344.225226] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11385s! [13344.230962] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13344.243598] Showing busy workqueues and worker pools: [13344.243608] workqueue events: flags=0x0 [13344.243614] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13344.243629] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [13344.244187] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [13344.244198] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [13344.244393] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13344.244400] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [13344.244612] workqueue events_long: flags=0x0 [13344.244615] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13344.244622] pending: lu_objects_flush [obdclass] [13344.244809] workqueue events_unbound: flags=0x2 [13344.244812] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [13344.244819] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [13344.244840] workqueue events_power_efficient: flags=0x80 [13344.244842] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13344.244848] pending: neigh_periodic_work [13344.244855] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13344.244863] pending: do_cache_clean, check_lifetime [13344.244876] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13344.244883] pending: neigh_periodic_work [13344.244893] workqueue writeback: flags=0x4a [13344.244896] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [13344.244902] pending: wb_workfn, wb_workfn [13344.244916] workqueue xprtiod: flags=0xa [13344.244918] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [13344.244924] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [13344.244944] workqueue ipv6_addrconf: flags=0x40008 [13344.244948] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13344.244954] pending: addrconf_verify_work [13344.244966] workqueue knbd0-recv: flags=0x1a [13344.244969] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13344.244976] in-flight: 657:recv_work [13344.245027] workqueue ptlrpc_pinger: flags=0x2 [13344.245031] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13344.245037] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [13344.245258] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11385s workers=4 idle: 20831 12642 28859 [13344.245270] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13334s workers=2 idle: 659 [13344.392346] ? do_wait_intr+0xf0/0xf0 [13344.397419] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13344.402097] mdc_close+0x53b/0xd70 [mdc] [13344.403539] lmv_close+0x18d/0x490 [lmv] [13344.410194] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [13344.413162] ll_close_inode_openhandle+0x493/0xea0 [lustre] [13344.418336] ll_md_real_close+0xa5/0x2a0 [lustre] [13344.427935] ll_md_close+0x274/0x800 [lustre] [13344.430491] ll_file_release+0x128/0xc50 [lustre] [13344.435277] __fput+0x10d/0x440 [13344.443422] ____fput+0x12/0x20 [13344.445177] task_work_run+0xc6/0x110 [13344.448477] exit_to_usermode_loop+0x1dd/0x1f0 [13344.453967] do_syscall_64+0x3d6/0x440 [13344.457227] entry_SYSCALL_64_after_hwframe+0x49/0xae [13344.458753] RIP: 0033:0x7ff54af45cd8 [13344.460254] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [13344.474697] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [13344.484253] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [13344.493672] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [13344.499308] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [13344.514231] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [13344.527618] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13354.148070] rcu: INFO: rcu_sched self-detected stall on CPU [13354.151174] rcu: 0-...!: (6049026 ticks this GP) idle=98a/1/0x4000000000000002 softirq=125154/125154 fqs=959 [13354.156463] (t=11400308 jiffies g=253585 q=6612) [13354.158987] rcu: rcu_sched kthread starved for 11395268 jiffies! g253585 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [13354.164758] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [13354.176713] rcu: RCU grace-period kthread stack dump: [13354.178294] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [13354.190345] Call Trace: [13354.193035] __schedule+0x351/0xcb0 [13354.195300] schedule+0xc0/0x180 [13354.198362] schedule_timeout+0xb4/0x190 [13354.201567] ? do_raw_spin_unlock+0x75/0x190 [13354.204961] ? __next_timer_interrupt+0x160/0x160 [13354.209311] ? prepare_to_swait_event+0xf3/0x140 [13354.212165] rcu_gp_kthread+0x8dd/0x15b0 [13354.214708] ? force_qs_rnp+0x2c0/0x2c0 [13354.217385] kthread+0x1d1/0x200 [13354.219568] ? set_kthread_struct+0x70/0x70 [13354.222389] ret_from_fork+0x1f/0x30 [13354.225443] rcu: Stack dump where RCU GP kthread last ran: [13354.229058] Sending NMI from CPU 0 to CPUs 2: [13354.236575] NMI backtrace for cpu 2 [13354.236584] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13354.236586] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13354.236587] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13354.236590] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13354.236592] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13354.236594] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 [13354.236595] RAX: 0000000000007c93 RBX: ffff96848811cae0 RCX: 0000000000000001 [13354.236597] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [13354.236598] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [13354.236600] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [13354.236601] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [13354.236602] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [13354.236604] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13354.236605] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [13354.236606] Call Trace: [13354.236607] [13354.236608] ? show_regs.cold.9+0x22/0x2f [13354.236609] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [13354.236610] ? nmi_cpu_backtrace_handler+0x15/0x40 [13354.236611] ? nmi_handle+0xbd/0x210 [13354.236612] ? default_do_nmi+0x4b/0x1b0 [13354.236613] ? do_nmi+0x280/0x380 [13354.236614] ? end_repeat_nmi+0x16/0x1f [13354.236615] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13354.236616] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13354.236618] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13354.236619] [13354.236620] do_raw_spin_lock+0x177/0x190 [13354.236621] _raw_spin_lock+0x21/0x30 [13354.236623] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13354.236624] process_one_work+0x2c8/0x700 [13354.236625] worker_thread+0x296/0x6e0 [13354.236626] ? rescuer_thread+0x570/0x570 [13354.236627] kthread+0x1d1/0x200 [13354.236628] ? set_kthread_struct+0x70/0x70 [13354.236629] ret_from_fork+0x1f/0x30 [13354.238654] NMI backtrace for cpu 0 [13354.428030] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13354.436811] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13354.444646] Call Trace: [13354.446017] [13354.449309] ? dump_stack+0xbb/0x10e [13354.452724] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [13354.458587] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [13354.466838] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [13354.471365] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [13354.478667] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [13354.482412] ? rcu_dump_cpu_stacks+0x11b/0x171 [13354.484556] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [13354.486941] ? update_process_times+0x6d/0xc0 [13354.489931] ? tick_sched_handle.isra.18+0x27/0x90 [13354.491755] ? tick_sched_timer+0x53/0xb0 [13354.495386] ? __hrtimer_run_queues+0x1c2/0x4d0 [13354.500189] ? tick_sched_handle.isra.18+0x90/0x90 [13354.503925] ? hrtimer_interrupt+0x128/0x2c0 [13354.509242] ? smp_apic_timer_interrupt+0xa6/0x270 [13354.511704] ? apic_timer_interrupt+0xf/0x20 [13354.516503] [13354.517389] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [13354.519272] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13354.521252] ? do_raw_spin_lock+0x177/0x190 [13354.522950] ? _raw_spin_lock+0x21/0x30 [13354.530386] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13354.536297] ? obd_mod_free+0x46/0x140 [ptlrpc] [13354.538992] ? mdc_commit_open+0x91/0xb0 [mdc] [13354.541531] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13354.545570] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13354.547995] ? after_reply+0xe2c/0x1780 [ptlrpc] [13354.550609] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13354.553374] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [13354.555192] ? do_wait_intr+0xf0/0xf0 [13354.556474] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13354.558419] ? kthread+0x1d1/0x200 [13354.559695] ? set_kthread_struct+0x70/0x70 [13354.561466] ? ret_from_fork+0x1f/0x30 [13368.060332] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [13368.067754] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13368.089752] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13368.094885] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13368.099215] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13368.102200] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13368.110853] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13368.114186] RAX: 00000000000020fd RBX: ffff96848811cae0 RCX: 0000000000000001 [13368.117147] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [13368.120845] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [13368.124164] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [13368.127733] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [13368.130696] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [13368.134336] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13368.137253] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [13368.140171] Call Trace: [13368.141150] [13368.142167] ? show_regs.cold.9+0x22/0x2f [13368.144294] ? watchdog_timer_fn.cold.6+0x62/0x10a [13368.146672] ? __hrtimer_run_queues+0x1c2/0x4d0 [13368.148621] ? watchdog+0x40/0x40 [13368.149779] ? hrtimer_interrupt+0x128/0x2c0 [13368.151921] ? smp_apic_timer_interrupt+0xa6/0x270 [13368.154097] ? apic_timer_interrupt+0xf/0x20 [13368.155796] [13368.156275] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13368.158339] do_raw_spin_lock+0x177/0x190 [13368.160207] _raw_spin_lock+0x21/0x30 [13368.161955] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13368.164693] ? do_raw_spin_unlock+0x75/0x190 [13368.166709] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13368.169245] ? do_wait_intr+0xf0/0xf0 [13368.170918] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13368.173723] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [13368.176301] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13368.179141] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13368.181906] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13368.183952] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [13368.186065] mdc_intent_lock+0x353/0x7e0 [mdc] [13368.188289] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13368.190993] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13368.193732] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13368.197041] lmv_intent_open+0x338/0x1150 [lmv] [13368.199556] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13368.202130] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13368.204407] lmv_intent_lock+0x1dd/0x4c0 [lmv] [13368.206430] ll_intent_lock+0xc5/0xaa0 [lustre] [13368.208817] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13368.211135] ll_lookup_it+0xe0e/0x2270 [lustre] [13368.213318] ? do_raw_spin_unlock+0x75/0x190 [13368.215178] ? _raw_spin_unlock+0x12/0x30 [13368.217279] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [13368.219425] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [13368.222381] ll_atomic_open+0x36a/0xd70 [lustre] [13368.225210] lookup_open+0x356/0xb40 [13368.228789] do_last+0x313/0xfc0 [13368.232119] ? nd_jump_root+0xe5/0x160 [13368.234317] ? path_init+0x437/0x520 [13368.236211] path_openat+0xf7/0x500 [13368.243326] ? do_raw_spin_unlock+0x75/0x190 [13368.245460] do_filp_open+0x99/0x140 [13368.251928] ? getname_flags+0x6e/0x330 [13368.253373] ? __check_object_size+0xff/0x256 [13368.256068] ? do_raw_spin_unlock+0x75/0x190 [13368.259127] ? _raw_spin_unlock+0x12/0x30 [13368.261063] do_sys_openat2+0x2b4/0x410 [13368.262390] do_sys_open+0x73/0xa0 [13368.268260] __x64_sys_openat+0x24/0x30 [13368.270811] do_syscall_64+0xc1/0x440 [13368.273040] entry_SYSCALL_64_after_hwframe+0x49/0xae [13368.275950] RIP: 0033:0x7ff54af45332 [13368.276871] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [13368.287487] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [13368.292748] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [13368.297362] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [13368.303836] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [13368.307285] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [13368.312460] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [13372.058714] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [13372.066019] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13372.101055] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13372.114248] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13372.127639] RIP: 0010:native_safe_halt+0x2e/0x50 [13372.130670] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13372.144272] RSP: 0018:ffffaf9fc1543ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13372.151520] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [13372.156112] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [13372.159723] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [13372.166282] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [13372.173313] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [13372.181215] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [13372.188270] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13372.193704] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [13372.198652] Call Trace: [13372.200095] [13372.206256] ? show_regs.cold.9+0x22/0x2f [13372.208978] ? watchdog_timer_fn.cold.6+0x62/0x10a [13372.215368] ? __hrtimer_run_queues+0x1c2/0x4d0 [13372.221247] ? watchdog+0x40/0x40 [13372.223625] ? hrtimer_interrupt+0x128/0x2c0 [13372.228283] ? smp_apic_timer_interrupt+0xa6/0x270 [13372.231917] ? apic_timer_interrupt+0xf/0x20 [13372.234121] [13372.237950] ? native_safe_halt+0x2e/0x50 [13372.240341] kvm_wait+0xcd/0xe0 [13372.242361] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13372.248987] do_raw_spin_lock+0x177/0x190 [13372.252843] _raw_spin_lock+0x21/0x30 [13372.258434] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13372.265816] ? do_raw_spin_unlock+0x75/0x190 [13372.269996] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13372.275440] ? do_wait_intr+0xf0/0xf0 [13372.280766] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13372.285119] mdc_close+0x53b/0xd70 [mdc] [13372.287189] lmv_close+0x18d/0x490 [lmv] [13372.289441] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [13372.293380] ll_close_inode_openhandle+0x493/0xea0 [lustre] [13372.303106] ll_md_real_close+0xa5/0x2a0 [lustre] [13372.309317] ll_md_close+0x274/0x800 [lustre] [13372.312854] ll_file_release+0x128/0xc50 [lustre] [13372.315679] __fput+0x10d/0x440 [13372.317100] ____fput+0x12/0x20 [13372.319063] task_work_run+0xc6/0x110 [13372.322456] exit_to_usermode_loop+0x1dd/0x1f0 [13372.325511] do_syscall_64+0x3d6/0x440 [13372.328514] entry_SYSCALL_64_after_hwframe+0x49/0xae [13372.331306] RIP: 0033:0x7ff54af45cd8 [13372.333085] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [13372.343408] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [13372.349619] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [13372.352916] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [13372.359518] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [13372.367040] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [13372.373280] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13374.439409] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11415s! [13374.460820] Showing busy workqueues and worker pools: [13374.471398] workqueue events: flags=0x0 [13374.472944] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13374.493116] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [13374.513051] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [13374.523011] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [13374.534872] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13374.547288] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [13374.549157] workqueue events_long: flags=0x0 [13374.553655] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13374.561035] pending: lu_objects_flush [obdclass] [13374.563882] workqueue events_unbound: flags=0x2 [13374.567825] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [13374.574082] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [13374.587963] workqueue events_power_efficient: flags=0x80 [13374.589186] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13374.590674] pending: neigh_periodic_work [13374.599909] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13374.603634] pending: do_cache_clean, check_lifetime [13374.606353] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13374.609835] pending: neigh_periodic_work [13374.612567] workqueue writeback: flags=0x4a [13374.618597] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [13374.631948] pending: wb_workfn, wb_workfn [13374.633944] workqueue xprtiod: flags=0xa [13374.641899] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [13374.647723] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [13374.651897] workqueue ipv6_addrconf: flags=0x40008 [13374.654349] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13374.657794] pending: addrconf_verify_work [13374.660374] workqueue knbd0-recv: flags=0x1a [13374.662336] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13374.665458] in-flight: 657:recv_work [13374.667299] workqueue ptlrpc_pinger: flags=0x2 [13374.669904] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13374.672890] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [13374.675973] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11415s workers=4 idle: 20831 12642 28859 [13374.681393] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13364s workers=2 idle: 659 [13380.059299] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [13380.064080] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13380.087037] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13380.091997] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13380.096925] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13380.100713] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13380.109295] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13380.119771] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13380.124262] RAX: 000000000000650a RBX: ffff96848811cae0 RCX: 0000000000000001 [13380.132772] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [13380.147859] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [13380.153322] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [13380.162559] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [13380.169314] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [13380.177228] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13380.181773] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [13380.189593] Call Trace: [13380.193846] [13380.195247] ? show_regs.cold.9+0x22/0x2f [13380.197253] ? watchdog_timer_fn.cold.6+0x62/0x10a [13380.200011] ? __hrtimer_run_queues+0x1c2/0x4d0 [13380.202351] ? watchdog+0x40/0x40 [13380.205337] ? hrtimer_interrupt+0x128/0x2c0 [13380.210459] ? smp_apic_timer_interrupt+0xa6/0x270 [13380.213411] ? apic_timer_interrupt+0xf/0x20 [13380.216041] [13380.218108] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13380.221777] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13380.224541] do_raw_spin_lock+0x177/0x190 [13380.234103] _raw_spin_lock+0x21/0x30 [13380.237396] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13380.241564] process_one_work+0x2c8/0x700 [13380.244316] worker_thread+0x296/0x6e0 [13380.248159] ? rescuer_thread+0x570/0x570 [13380.251102] kthread+0x1d1/0x200 [13380.253163] ? set_kthread_struct+0x70/0x70 [13380.255364] ret_from_fork+0x1f/0x30 [13396.060039] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [13396.064090] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13396.093170] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13396.103095] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13396.107952] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13396.112404] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13396.133051] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13396.143195] RAX: 0000000000006ed6 RBX: ffff96848811cae0 RCX: 0000000000000001 [13396.147885] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [13396.155152] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [13396.164130] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [13396.170995] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [13396.179024] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [13396.184818] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13396.190371] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [13396.199601] Call Trace: [13396.201811] [13396.203015] ? show_regs.cold.9+0x22/0x2f [13396.206047] ? watchdog_timer_fn.cold.6+0x62/0x10a [13396.211908] ? __hrtimer_run_queues+0x1c2/0x4d0 [13396.214621] ? watchdog+0x40/0x40 [13396.216686] ? hrtimer_interrupt+0x128/0x2c0 [13396.222090] ? smp_apic_timer_interrupt+0xa6/0x270 [13396.225993] ? apic_timer_interrupt+0xf/0x20 [13396.227583] [13396.228537] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13396.230171] do_raw_spin_lock+0x177/0x190 [13396.232182] _raw_spin_lock+0x21/0x30 [13396.239786] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13396.246691] ? do_raw_spin_unlock+0x75/0x190 [13396.249781] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13396.254786] ? do_wait_intr+0xf0/0xf0 [13396.258158] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13396.262054] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [13396.264704] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13396.268100] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13396.272191] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13396.276984] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [13396.283670] mdc_intent_lock+0x353/0x7e0 [mdc] [13396.290083] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13396.295810] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13396.310296] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13396.321269] lmv_intent_open+0x338/0x1150 [lmv] [13396.323182] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13396.333301] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13396.341452] lmv_intent_lock+0x1dd/0x4c0 [lmv] [13396.346127] ll_intent_lock+0xc5/0xaa0 [lustre] [13396.351859] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13396.362263] ll_lookup_it+0xe0e/0x2270 [lustre] [13396.375828] ? do_raw_spin_unlock+0x75/0x190 [13396.377390] ? _raw_spin_unlock+0x12/0x30 [13396.385675] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [13396.387499] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [13396.397085] ll_atomic_open+0x36a/0xd70 [lustre] [13396.404043] lookup_open+0x356/0xb40 [13396.413922] do_last+0x313/0xfc0 [13396.419170] ? nd_jump_root+0xe5/0x160 [13396.425460] ? path_init+0x437/0x520 [13396.428357] path_openat+0xf7/0x500 [13396.429437] ? do_raw_spin_unlock+0x75/0x190 [13396.432462] do_filp_open+0x99/0x140 [13396.435551] ? getname_flags+0x6e/0x330 [13396.437555] ? __check_object_size+0xff/0x256 [13396.439155] ? do_raw_spin_unlock+0x75/0x190 [13396.441812] ? _raw_spin_unlock+0x12/0x30 [13396.444157] do_sys_openat2+0x2b4/0x410 [13396.446181] do_sys_open+0x73/0xa0 [13396.447583] __x64_sys_openat+0x24/0x30 [13396.448584] do_syscall_64+0xc1/0x440 [13396.453116] entry_SYSCALL_64_after_hwframe+0x49/0xae [13396.455487] RIP: 0033:0x7ff54af45332 [13396.457203] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [13396.466136] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [13396.470494] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [13396.472969] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [13396.475711] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [13396.478277] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [13396.481081] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [13400.057545] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [13400.058135] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [13400.063781] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13400.063869] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13400.067246] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13400.067259] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [13400.067288] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [13400.067292] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13400.067298] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [13400.067301] RDX: 0000000000007454 RSI: 0000000000000003 RDI: ffff96848811cae0 [13400.067303] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [13400.067305] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [13400.067308] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13400.067311] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [13400.067314] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13400.067317] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [13400.067325] Call Trace: [13400.067332] [13400.067337] ? show_regs.cold.9+0x22/0x2f [13400.067348] ? watchdog_timer_fn.cold.6+0x62/0x10a [13400.067356] ? __hrtimer_run_queues+0x1c2/0x4d0 [13400.067360] ? watchdog+0x40/0x40 [13400.067365] ? hrtimer_interrupt+0x128/0x2c0 [13400.067369] ? smp_apic_timer_interrupt+0xa6/0x270 [13400.067378] ? apic_timer_interrupt+0xf/0x20 [13400.067382] [13400.067385] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [13400.067391] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13400.067395] do_raw_spin_lock+0x177/0x190 [13400.067400] _raw_spin_lock+0x21/0x30 [13400.067406] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13400.077623] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13400.077713] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13400.077720] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13400.077724] RIP: 0010:native_safe_halt+0x2e/0x50 [13400.077738] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13400.077742] RSP: 0018:ffffaf9fc1543ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13400.077747] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [13400.077749] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [13400.077752] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [13400.077754] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [13400.077756] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [13400.077759] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [13400.077763] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13400.077765] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [13400.077775] Call Trace: [13400.077780] [13400.077786] ? show_regs.cold.9+0x22/0x2f [13400.077797] ? watchdog_timer_fn.cold.6+0x62/0x10a [13400.077803] ? __hrtimer_run_queues+0x1c2/0x4d0 [13400.077807] ? watchdog+0x40/0x40 [13400.077812] ? hrtimer_interrupt+0x128/0x2c0 [13400.077816] ? smp_apic_timer_interrupt+0xa6/0x270 [13400.077823] ? apic_timer_interrupt+0xf/0x20 [13400.077827] [13400.077830] ? native_safe_halt+0x2e/0x50 [13400.077834] kvm_wait+0xcd/0xe0 [13400.077842] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13400.077922] do_raw_spin_lock+0x177/0x190 [13400.077929] _raw_spin_lock+0x21/0x30 [13400.077934] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13400.126853] obd_mod_free+0x46/0x140 [ptlrpc] [13400.127155] mdc_commit_open+0x91/0xb0 [mdc] [13400.127194] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13400.127393] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13400.141844] ? do_raw_spin_unlock+0x75/0x190 [13400.141894] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13400.142508] ? do_wait_intr+0xf0/0xf0 [13400.142516] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13400.142900] mdc_close+0x53b/0xd70 [mdc] [13400.142944] lmv_close+0x18d/0x490 [lmv] [13400.142971] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [13400.143549] ll_close_inode_openhandle+0x493/0xea0 [lustre] [13400.143686] ll_md_real_close+0xa5/0x2a0 [lustre] [13400.143774] ll_md_close+0x274/0x800 [lustre] [13400.143853] ll_file_release+0x128/0xc50 [lustre] [13400.150899] after_reply+0xe2c/0x1780 [ptlrpc] [13400.151241] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13400.151439] ptlrpcd+0xa82/0xe40 [ptlrpc] [13400.151657] ? do_wait_intr+0xf0/0xf0 [13400.151668] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13400.151883] kthread+0x1d1/0x200 [13400.151890] ? set_kthread_struct+0x70/0x70 [13400.151893] ret_from_fork+0x1f/0x30 [13400.551015] __fput+0x10d/0x440 [13400.552136] ____fput+0x12/0x20 [13400.555736] task_work_run+0xc6/0x110 [13400.558528] exit_to_usermode_loop+0x1dd/0x1f0 [13400.560858] do_syscall_64+0x3d6/0x440 [13400.562753] entry_SYSCALL_64_after_hwframe+0x49/0xae [13400.566769] RIP: 0033:0x7ff54af45cd8 [13400.568591] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [13400.596865] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [13400.602675] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [13400.617081] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [13400.629934] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [13400.639666] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [13400.648205] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13405.151457] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11446s! [13405.163190] Showing busy workqueues and worker pools: [13405.167497] workqueue events: flags=0x0 [13405.169659] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13405.178469] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [13405.195694] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [13405.200806] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [13405.217670] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13405.220636] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [13405.224779] workqueue events_long: flags=0x0 [13405.229636] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13405.235782] pending: lu_objects_flush [obdclass] [13405.239714] workqueue events_unbound: flags=0x2 [13405.244972] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [13405.248811] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [13405.265805] workqueue events_power_efficient: flags=0x80 [13405.270108] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13405.273378] pending: neigh_periodic_work [13405.275922] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13405.279878] pending: do_cache_clean, check_lifetime [13405.287616] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13405.294701] pending: neigh_periodic_work [13405.297747] workqueue writeback: flags=0x4a [13405.299718] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [13405.305177] pending: wb_workfn, wb_workfn [13405.307681] workqueue xprtiod: flags=0xa [13405.309689] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [13405.311968] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [13405.316988] workqueue ipv6_addrconf: flags=0x40008 [13405.319534] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13405.321986] pending: addrconf_verify_work [13405.325171] workqueue knbd0-recv: flags=0x1a [13405.327613] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13405.330953] in-flight: 657:recv_work [13405.333011] workqueue ptlrpc_pinger: flags=0x2 [13405.335722] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13405.341321] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [13405.347209] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11446s workers=4 idle: 20831 12642 28859 [13405.352343] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13395s workers=2 idle: 659 [13408.063830] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [13408.078659] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13408.144440] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13408.157738] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13408.167562] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13408.173539] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13408.177656] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13408.191287] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13408.197176] RAX: 0000000000007b96 RBX: ffff96848811cae0 RCX: 0000000000000001 [13408.201357] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [13408.204740] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [13408.207929] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [13408.212147] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [13408.217161] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [13408.222276] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13408.228057] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [13408.233567] Call Trace: [13408.234803] [13408.236411] ? show_regs.cold.9+0x22/0x2f [13408.238791] ? watchdog_timer_fn.cold.6+0x62/0x10a [13408.245529] ? __hrtimer_run_queues+0x1c2/0x4d0 [13408.248973] ? watchdog+0x40/0x40 [13408.254765] ? hrtimer_interrupt+0x128/0x2c0 [13408.256765] ? smp_apic_timer_interrupt+0xa6/0x270 [13408.258718] ? apic_timer_interrupt+0xf/0x20 [13408.260958] [13408.262344] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13408.264806] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13408.267831] do_raw_spin_lock+0x177/0x190 [13408.271530] _raw_spin_lock+0x21/0x30 [13408.275281] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13408.280685] process_one_work+0x2c8/0x700 [13408.282091] worker_thread+0x296/0x6e0 [13408.283350] ? rescuer_thread+0x570/0x570 [13408.287217] kthread+0x1d1/0x200 [13408.288404] ? set_kthread_struct+0x70/0x70 [13408.291129] ret_from_fork+0x1f/0x30 [13424.062610] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [13424.067820] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13424.106015] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13424.115617] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13424.120097] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13424.123998] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13424.148011] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [13424.156771] RAX: 0000000000006774 RBX: ffff96848811cae0 RCX: 0000000000000001 [13424.161604] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [13424.169882] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [13424.176430] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [13424.186337] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [13424.196856] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [13424.205460] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13424.214729] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [13424.221347] Call Trace: [13424.222465] [13424.223027] ? show_regs.cold.9+0x22/0x2f [13424.224812] ? watchdog_timer_fn.cold.6+0x62/0x10a [13424.226903] ? __hrtimer_run_queues+0x1c2/0x4d0 [13424.235030] ? watchdog+0x40/0x40 [13424.239150] ? hrtimer_interrupt+0x128/0x2c0 [13424.245833] ? smp_apic_timer_interrupt+0xa6/0x270 [13424.249798] ? apic_timer_interrupt+0xf/0x20 [13424.251864] [13424.253476] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13424.256781] do_raw_spin_lock+0x177/0x190 [13424.260245] _raw_spin_lock+0x21/0x30 [13424.265267] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13424.267298] ? do_raw_spin_unlock+0x75/0x190 [13424.268793] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13424.273755] ? do_wait_intr+0xf0/0xf0 [13424.275845] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13424.282720] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [13424.288577] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13424.293277] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13424.298483] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13424.302359] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [13424.307586] mdc_intent_lock+0x353/0x7e0 [mdc] [13424.311112] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13424.316506] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13424.319487] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13424.324299] lmv_intent_open+0x338/0x1150 [lmv] [13424.329419] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13424.335415] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13424.338671] lmv_intent_lock+0x1dd/0x4c0 [lmv] [13424.340761] ll_intent_lock+0xc5/0xaa0 [lustre] [13424.343490] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13424.345152] ll_lookup_it+0xe0e/0x2270 [lustre] [13424.354612] ? do_raw_spin_unlock+0x75/0x190 [13424.357823] ? _raw_spin_unlock+0x12/0x30 [13424.359016] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [13424.361696] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [13424.363053] ll_atomic_open+0x36a/0xd70 [lustre] [13424.366951] lookup_open+0x356/0xb40 [13424.368640] do_last+0x313/0xfc0 [13424.374638] ? nd_jump_root+0xe5/0x160 [13424.377386] ? path_init+0x437/0x520 [13424.380387] path_openat+0xf7/0x500 [13424.382084] ? do_raw_spin_unlock+0x75/0x190 [13424.383324] do_filp_open+0x99/0x140 [13424.389228] ? getname_flags+0x6e/0x330 [13424.391525] ? __check_object_size+0xff/0x256 [13424.392847] ? do_raw_spin_unlock+0x75/0x190 [13424.397529] ? _raw_spin_unlock+0x12/0x30 [13424.402262] do_sys_openat2+0x2b4/0x410 [13424.405912] do_sys_open+0x73/0xa0 [13424.407359] __x64_sys_openat+0x24/0x30 [13424.408723] do_syscall_64+0xc1/0x440 [13424.410641] entry_SYSCALL_64_after_hwframe+0x49/0xae [13424.413682] RIP: 0033:0x7ff54af45332 [13424.416035] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [13424.429914] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [13424.433724] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [13424.437596] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [13424.441498] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [13424.444137] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [13424.450049] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [13428.061392] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [13428.064066] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13428.096920] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13428.101508] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13428.104775] RIP: 0010:native_safe_halt+0x2e/0x50 [13428.111635] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13428.129629] RSP: 0018:ffffaf9fc1543ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13428.134453] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [13428.138576] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c20b4154 [13428.142273] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [13428.149696] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [13428.155401] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [13428.160265] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [13428.172133] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13428.179037] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [13428.185219] Call Trace: [13428.187156] [13428.187943] ? show_regs.cold.9+0x22/0x2f [13428.189810] ? watchdog_timer_fn.cold.6+0x62/0x10a [13428.198746] ? __hrtimer_run_queues+0x1c2/0x4d0 [13428.201641] ? watchdog+0x40/0x40 [13428.203898] ? hrtimer_interrupt+0x128/0x2c0 [13428.207669] ? smp_apic_timer_interrupt+0xa6/0x270 [13428.211054] ? apic_timer_interrupt+0xf/0x20 [13428.213864] [13428.215711] ? native_safe_halt+0x2e/0x50 [13428.220560] kvm_wait+0xcd/0xe0 [13428.226590] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13428.236742] do_raw_spin_lock+0x177/0x190 [13428.246548] _raw_spin_lock+0x21/0x30 [13428.256766] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13428.268579] ? do_raw_spin_unlock+0x75/0x190 [13428.274742] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13428.280803] ? do_wait_intr+0xf0/0xf0 [13428.284722] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13428.287564] mdc_close+0x53b/0xd70 [mdc] [13428.289756] lmv_close+0x18d/0x490 [lmv] [13428.292668] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [13428.296921] ll_close_inode_openhandle+0x493/0xea0 [lustre] [13428.304640] ll_md_real_close+0xa5/0x2a0 [lustre] [13428.313657] ll_md_close+0x274/0x800 [lustre] [13428.318354] ll_file_release+0x128/0xc50 [lustre] [13428.322865] __fput+0x10d/0x440 [13428.325480] ____fput+0x12/0x20 [13428.326985] task_work_run+0xc6/0x110 [13428.329371] exit_to_usermode_loop+0x1dd/0x1f0 [13428.331546] do_syscall_64+0x3d6/0x440 [13428.333940] entry_SYSCALL_64_after_hwframe+0x49/0xae [13428.337932] RIP: 0033:0x7ff54af45cd8 [13428.340546] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [13428.358384] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [13428.361967] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [13428.366495] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [13428.370338] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [13428.373286] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [13428.376407] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13432.057142] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [13432.061842] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13432.107770] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13432.113902] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13432.124640] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [13432.128615] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [13432.134732] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13432.136819] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [13432.145233] RDX: 0000000000001f51 RSI: 0000000000000003 RDI: ffff96848811cae0 [13432.149484] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [13432.152270] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [13432.156434] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13432.161344] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [13432.164595] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13432.167894] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [13432.172253] Call Trace: [13432.173605] [13432.174916] ? show_regs.cold.9+0x22/0x2f [13432.176543] ? watchdog_timer_fn.cold.6+0x62/0x10a [13432.178074] ? __hrtimer_run_queues+0x1c2/0x4d0 [13432.183947] ? watchdog+0x40/0x40 [13432.187185] ? hrtimer_interrupt+0x128/0x2c0 [13432.190712] ? smp_apic_timer_interrupt+0xa6/0x270 [13432.192853] ? apic_timer_interrupt+0xf/0x20 [13432.194276] [13432.195067] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [13432.198903] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13432.202353] do_raw_spin_lock+0x177/0x190 [13432.203946] _raw_spin_lock+0x21/0x30 [13432.207400] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13432.212204] obd_mod_free+0x46/0x140 [ptlrpc] [13432.214719] mdc_commit_open+0x91/0xb0 [mdc] [13432.217623] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13432.219952] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13432.222531] after_reply+0xe2c/0x1780 [ptlrpc] [13432.224739] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13432.227175] ptlrpcd+0xa82/0xe40 [ptlrpc] [13432.229734] ? do_wait_intr+0xf0/0xf0 [13432.231381] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13432.233729] kthread+0x1d1/0x200 [13432.235175] ? set_kthread_struct+0x70/0x70 [13432.237253] ret_from_fork+0x1f/0x30 [13435.871375] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11476s! [13435.881475] Showing busy workqueues and worker pools: [13435.884715] workqueue events: flags=0x0 [13435.889862] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13435.892880] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [13435.912828] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [13435.922448] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [13435.933741] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13435.938626] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [13435.949177] workqueue events_long: flags=0x0 [13435.953727] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13435.965425] pending: lu_objects_flush [obdclass] [13435.969476] workqueue events_unbound: flags=0x2 [13435.975396] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [13435.986132] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [13435.998288] workqueue events_power_efficient: flags=0x80 [13436.000794] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13436.006666] pending: neigh_periodic_work [13436.009696] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13436.016310] pending: do_cache_clean, check_lifetime [13436.019994] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13436.024850] pending: neigh_periodic_work [13436.027537] workqueue writeback: flags=0x4a [13436.030887] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [13436.036010] pending: wb_workfn, wb_workfn [13436.038899] workqueue xprtiod: flags=0xa [13436.041203] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [13436.045475] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [13436.052265] workqueue ipv6_addrconf: flags=0x40008 [13436.056605] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13436.059061] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [13436.061751] pending: addrconf_verify_work [13436.061815] workqueue knbd0-recv: flags=0x1a [13436.061821] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13436.061831] in-flight: 657:recv_work [13436.061842] workqueue ptlrpc_pinger: flags=0x2 [13436.061845] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13436.061852] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [13436.068649] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13436.068725] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13436.068732] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13436.068739] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13436.069046] RIP: 0010:native_safe_halt+0x2e/0x50 [13436.069056] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13436.069060] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13436.069065] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [13436.069067] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c2134154 [13436.069069] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [13436.069072] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [13436.069074] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [13436.069078] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [13436.069081] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13436.069083] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [13436.069092] Call Trace: [13436.069107] [13436.069113] ? show_regs.cold.9+0x22/0x2f [13436.069124] ? watchdog_timer_fn.cold.6+0x62/0x10a [13436.069130] ? __hrtimer_run_queues+0x1c2/0x4d0 [13436.069135] ? watchdog+0x40/0x40 [13436.069139] ? hrtimer_interrupt+0x128/0x2c0 [13436.069143] ? smp_apic_timer_interrupt+0xa6/0x270 [13436.069151] ? apic_timer_interrupt+0xf/0x20 [13436.069155] [13436.069158] ? native_safe_halt+0x2e/0x50 [13436.069162] kvm_wait+0xcd/0xe0 [13436.069169] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13436.069307] do_raw_spin_lock+0x177/0x190 [13436.069312] _raw_spin_lock+0x21/0x30 [13436.069317] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13436.069828] process_one_work+0x2c8/0x700 [13436.069836] worker_thread+0x296/0x6e0 [13436.069841] ? rescuer_thread+0x570/0x570 [13436.069845] kthread+0x1d1/0x200 [13436.069849] ? set_kthread_struct+0x70/0x70 [13436.069853] ret_from_fork+0x1f/0x30 [13436.322629] [13436.326116] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11477s workers=4 idle: 20831 12642 28859 [13436.334547] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13426s workers=2 idle: 659 [13452.070400] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [13452.081486] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13452.154530] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13452.171616] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13452.183500] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13452.198395] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13452.222206] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13452.232436] RAX: 0000000000007d88 RBX: ffff96848811cae0 RCX: 0000000000000001 [13452.240598] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [13452.245623] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [13452.254171] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [13452.263926] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [13452.268649] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [13452.273862] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13452.284741] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [13452.289640] Call Trace: [13452.293131] [13452.294269] ? show_regs.cold.9+0x22/0x2f [13452.298849] ? watchdog_timer_fn.cold.6+0x62/0x10a [13452.305431] ? __hrtimer_run_queues+0x1c2/0x4d0 [13452.310111] ? watchdog+0x40/0x40 [13452.314805] ? hrtimer_interrupt+0x128/0x2c0 [13452.317133] ? smp_apic_timer_interrupt+0xa6/0x270 [13452.328159] ? apic_timer_interrupt+0xf/0x20 [13452.333403] [13452.335224] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13452.350220] do_raw_spin_lock+0x177/0x190 [13452.364423] _raw_spin_lock+0x21/0x30 [13452.376355] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13452.390601] ? do_raw_spin_unlock+0x75/0x190 [13452.394933] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13452.403653] ? do_wait_intr+0xf0/0xf0 [13452.412760] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13452.423952] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [13452.430331] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13452.442796] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13452.448259] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13452.450374] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [13452.456324] mdc_intent_lock+0x353/0x7e0 [mdc] [13452.468606] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13452.478087] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13452.488479] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13452.500750] lmv_intent_open+0x338/0x1150 [lmv] [13452.505862] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13452.515947] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13452.518440] lmv_intent_lock+0x1dd/0x4c0 [lmv] [13452.522670] ll_intent_lock+0xc5/0xaa0 [lustre] [13452.526536] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13452.530657] ll_lookup_it+0xe0e/0x2270 [lustre] [13452.532831] ? do_raw_spin_unlock+0x75/0x190 [13452.535986] ? _raw_spin_unlock+0x12/0x30 [13452.541194] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [13452.544255] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [13452.547692] ll_atomic_open+0x36a/0xd70 [lustre] [13452.558102] lookup_open+0x356/0xb40 [13452.560921] do_last+0x313/0xfc0 [13452.563964] ? nd_jump_root+0xe5/0x160 [13452.567747] ? path_init+0x437/0x520 [13452.570090] path_openat+0xf7/0x500 [13452.574630] ? do_raw_spin_unlock+0x75/0x190 [13452.575666] do_filp_open+0x99/0x140 [13452.576694] ? getname_flags+0x6e/0x330 [13452.582348] ? __check_object_size+0xff/0x256 [13452.587632] ? do_raw_spin_unlock+0x75/0x190 [13452.591226] ? _raw_spin_unlock+0x12/0x30 [13452.596139] do_sys_openat2+0x2b4/0x410 [13452.599945] do_sys_open+0x73/0xa0 [13452.605085] __x64_sys_openat+0x24/0x30 [13452.608694] do_syscall_64+0xc1/0x440 [13452.610987] entry_SYSCALL_64_after_hwframe+0x49/0xae [13452.616588] RIP: 0033:0x7ff54af45332 [13452.620973] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [13452.641576] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [13452.645955] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [13452.650485] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [13452.659419] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [13452.672363] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [13452.674518] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [13456.060135] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [13456.069623] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13456.102535] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13456.114762] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13456.126545] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13456.133716] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13456.154861] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13456.161500] RAX: 0000000000007b0f RBX: ffff96848811cae0 RCX: 0000000000000001 [13456.170571] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [13456.175218] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [13456.178514] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [13456.181713] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [13456.185331] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [13456.193119] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13456.196801] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [13456.204638] Call Trace: [13456.208451] [13456.210122] ? show_regs.cold.9+0x22/0x2f [13456.214038] ? watchdog_timer_fn.cold.6+0x62/0x10a [13456.221216] ? __hrtimer_run_queues+0x1c2/0x4d0 [13456.227118] ? watchdog+0x40/0x40 [13456.231183] ? hrtimer_interrupt+0x128/0x2c0 [13456.234875] ? smp_apic_timer_interrupt+0xa6/0x270 [13456.238145] ? apic_timer_interrupt+0xf/0x20 [13456.246570] [13456.249192] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13456.254225] do_raw_spin_lock+0x177/0x190 [13456.257074] _raw_spin_lock+0x21/0x30 [13456.261276] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13456.266019] ? do_raw_spin_unlock+0x75/0x190 [13456.267198] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13456.275302] ? do_wait_intr+0xf0/0xf0 [13456.279893] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13456.282001] mdc_close+0x53b/0xd70 [mdc] [13456.291214] lmv_close+0x18d/0x490 [lmv] [13456.294124] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [13456.298794] ll_close_inode_openhandle+0x493/0xea0 [lustre] [13456.303259] ll_md_real_close+0xa5/0x2a0 [lustre] [13456.308717] ll_md_close+0x274/0x800 [lustre] [13456.327228] ll_file_release+0x128/0xc50 [lustre] [13456.332920] __fput+0x10d/0x440 [13456.342028] ____fput+0x12/0x20 [13456.355098] task_work_run+0xc6/0x110 [13456.370065] exit_to_usermode_loop+0x1dd/0x1f0 [13456.371689] do_syscall_64+0x3d6/0x440 [13456.376080] entry_SYSCALL_64_after_hwframe+0x49/0xae [13456.379184] RIP: 0033:0x7ff54af45cd8 [13456.383952] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [13456.398117] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [13456.407086] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [13456.414131] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [13456.417628] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [13456.422102] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [13456.430939] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13464.058630] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [13464.061032] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [13464.063652] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13464.063741] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13464.063749] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13464.063752] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [13464.063767] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [13464.063771] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13464.063776] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [13464.063779] RDX: 00000000000040c4 RSI: 0000000000000003 RDI: ffff96848811cae0 [13464.063781] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [13464.063783] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [13464.063786] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13464.063789] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [13464.063792] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13464.063794] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [13464.063803] Call Trace: [13464.063811] [13464.063816] ? show_regs.cold.9+0x22/0x2f [13464.063984] ? watchdog_timer_fn.cold.6+0x62/0x10a [13464.063993] ? __hrtimer_run_queues+0x1c2/0x4d0 [13464.063997] ? watchdog+0x40/0x40 [13464.064019] ? hrtimer_interrupt+0x128/0x2c0 [13464.064024] ? smp_apic_timer_interrupt+0xa6/0x270 [13464.064033] ? apic_timer_interrupt+0xf/0x20 [13464.064038] [13464.064042] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [13464.064047] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13464.064053] do_raw_spin_lock+0x177/0x190 [13464.064058] _raw_spin_lock+0x21/0x30 [13464.064065] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13464.064386] obd_mod_free+0x46/0x140 [ptlrpc] [13464.064626] mdc_commit_open+0x91/0xb0 [mdc] [13464.064668] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13464.064903] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13464.065128] after_reply+0xe2c/0x1780 [ptlrpc] [13464.065369] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13464.065578] ptlrpcd+0xa82/0xe40 [ptlrpc] [13464.065818] ? do_wait_intr+0xf0/0xf0 [13464.065843] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13464.066090] kthread+0x1d1/0x200 [13464.066095] ? set_kthread_struct+0x70/0x70 [13464.066100] ret_from_fork+0x1f/0x30 [13464.425070] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13464.501463] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13464.513327] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13464.528062] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13464.537304] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13464.539495] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13464.579973] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13464.596622] RAX: 000000000000770c RBX: ffff96848811cae0 RCX: 0000000000000001 [13464.604531] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [13464.610404] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [13464.616891] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [13464.619378] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [13464.635944] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [13464.647153] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13464.655174] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [13464.666417] Call Trace: [13464.670310] [13464.675422] ? show_regs.cold.9+0x22/0x2f [13464.678578] ? watchdog_timer_fn.cold.6+0x62/0x10a [13464.684282] ? __hrtimer_run_queues+0x1c2/0x4d0 [13464.688477] ? watchdog+0x40/0x40 [13464.694700] ? hrtimer_interrupt+0x128/0x2c0 [13464.698317] ? smp_apic_timer_interrupt+0xa6/0x270 [13464.701173] ? apic_timer_interrupt+0xf/0x20 [13464.703377] [13464.704259] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13464.707069] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13464.709277] do_raw_spin_lock+0x177/0x190 [13464.711141] _raw_spin_lock+0x21/0x30 [13464.712715] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13464.715091] process_one_work+0x2c8/0x700 [13464.717356] worker_thread+0x296/0x6e0 [13464.720540] ? rescuer_thread+0x570/0x570 [13464.722042] kthread+0x1d1/0x200 [13464.723174] ? set_kthread_struct+0x70/0x70 [13464.728133] ret_from_fork+0x1f/0x30 [13466.594763] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11507s! [13466.606272] Showing busy workqueues and worker pools: [13466.610783] workqueue events: flags=0x0 [13466.614110] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13466.624337] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [13466.631952] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [13466.638955] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [13466.648085] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13466.651716] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [13466.655869] workqueue events_long: flags=0x0 [13466.658375] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13466.664182] pending: lu_objects_flush [obdclass] [13466.667193] workqueue events_unbound: flags=0x2 [13466.672360] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [13466.675368] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [13466.682693] workqueue events_power_efficient: flags=0x80 [13466.686127] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13466.690604] pending: neigh_periodic_work [13466.693302] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13466.697687] pending: do_cache_clean, check_lifetime [13466.700622] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13466.705173] pending: neigh_periodic_work [13466.707813] workqueue writeback: flags=0x4a [13466.710659] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [13466.715397] pending: wb_workfn, wb_workfn [13466.718718] workqueue xprtiod: flags=0xa [13466.721306] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [13466.731523] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [13466.741846] workqueue ipv6_addrconf: flags=0x40008 [13466.744011] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13466.755155] pending: addrconf_verify_work [13466.761595] workqueue knbd0-recv: flags=0x1a [13466.765095] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13466.769165] in-flight: 657:recv_work [13466.771979] workqueue ptlrpc_pinger: flags=0x2 [13466.777970] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13466.787749] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [13466.796017] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11507s workers=4 idle: 20831 12642 28859 [13466.804145] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13457s workers=2 idle: 659 [13480.060020] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [13480.063889] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13480.091136] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13480.103331] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13480.110450] RIP: 0010:native_safe_halt+0x2e/0x50 [13480.118467] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13480.131918] RSP: 0000:ffffaf9fc15234f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13480.136715] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [13480.140078] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [13480.143918] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [13480.147035] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [13480.152898] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [13480.157204] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [13480.161113] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13480.164057] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [13480.169184] Call Trace: [13480.170136] [13480.171844] ? show_regs.cold.9+0x22/0x2f [13480.175281] ? watchdog_timer_fn.cold.6+0x62/0x10a [13480.183385] ? __hrtimer_run_queues+0x1c2/0x4d0 [13480.184520] ? watchdog+0x40/0x40 [13480.185933] ? hrtimer_interrupt+0x128/0x2c0 [13480.190280] ? smp_apic_timer_interrupt+0xa6/0x270 [13480.192697] ? apic_timer_interrupt+0xf/0x20 [13480.202370] [13480.203184] ? native_safe_halt+0x2e/0x50 [13480.205301] kvm_wait+0xcd/0xe0 [13480.206598] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13480.209733] do_raw_spin_lock+0x177/0x190 [13480.216222] _raw_spin_lock+0x21/0x30 [13480.217312] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13480.220244] ? do_raw_spin_unlock+0x75/0x190 [13480.222230] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13480.225115] ? do_wait_intr+0xf0/0xf0 [13480.226730] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13480.229375] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [13480.232603] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13480.234547] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13480.241256] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13480.243543] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [13480.245651] mdc_intent_lock+0x353/0x7e0 [mdc] [13480.247641] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13480.252719] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13480.264357] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13480.279701] lmv_intent_open+0x338/0x1150 [lmv] [13480.288713] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13480.299142] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13480.301514] lmv_intent_lock+0x1dd/0x4c0 [lmv] [13480.306304] ll_intent_lock+0xc5/0xaa0 [lustre] [13480.310684] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13480.323815] ll_lookup_it+0xe0e/0x2270 [lustre] [13480.329413] ? do_raw_spin_unlock+0x75/0x190 [13480.341194] ? _raw_spin_unlock+0x12/0x30 [13480.345614] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [13480.349359] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [13480.352258] ll_atomic_open+0x36a/0xd70 [lustre] [13480.354608] lookup_open+0x356/0xb40 [13480.356969] do_last+0x313/0xfc0 [13480.358981] ? nd_jump_root+0xe5/0x160 [13480.360901] ? path_init+0x437/0x520 [13480.362660] path_openat+0xf7/0x500 [13480.364610] ? do_raw_spin_unlock+0x75/0x190 [13480.370150] do_filp_open+0x99/0x140 [13480.375304] ? getname_flags+0x6e/0x330 [13480.380269] ? __check_object_size+0xff/0x256 [13480.384066] ? do_raw_spin_unlock+0x75/0x190 [13480.389653] ? _raw_spin_unlock+0x12/0x30 [13480.392551] do_sys_openat2+0x2b4/0x410 [13480.394902] do_sys_open+0x73/0xa0 [13480.396372] __x64_sys_openat+0x24/0x30 [13480.398166] do_syscall_64+0xc1/0x440 [13480.399993] entry_SYSCALL_64_after_hwframe+0x49/0xae [13480.402554] RIP: 0033:0x7ff54af45332 [13480.404415] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [13480.423509] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [13480.428189] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [13480.435806] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [13480.443566] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [13480.454920] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [13480.464087] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [13484.058460] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [13484.061513] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13484.092687] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13484.096330] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13484.099524] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13484.102792] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13484.113036] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13484.115832] RAX: 0000000000007f16 RBX: ffff96848811cae0 RCX: 0000000000000001 [13484.118777] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [13484.121484] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [13484.124638] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [13484.127939] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [13484.130957] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [13484.134114] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13484.136416] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [13484.139930] Call Trace: [13484.140670] [13484.141370] ? show_regs.cold.9+0x22/0x2f [13484.142353] ? watchdog_timer_fn.cold.6+0x62/0x10a [13484.146332] ? __hrtimer_run_queues+0x1c2/0x4d0 [13484.150262] ? watchdog+0x40/0x40 [13484.151261] ? hrtimer_interrupt+0x128/0x2c0 [13484.152394] ? smp_apic_timer_interrupt+0xa6/0x270 [13484.154100] ? apic_timer_interrupt+0xf/0x20 [13484.157418] [13484.158278] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13484.160366] do_raw_spin_lock+0x177/0x190 [13484.162023] _raw_spin_lock+0x21/0x30 [13484.164706] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13484.171072] ? do_raw_spin_unlock+0x75/0x190 [13484.172563] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13484.178559] ? do_wait_intr+0xf0/0xf0 [13484.180710] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13484.188393] mdc_close+0x53b/0xd70 [mdc] [13484.190382] lmv_close+0x18d/0x490 [lmv] [13484.191770] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [13484.200217] ll_close_inode_openhandle+0x493/0xea0 [lustre] [13484.203374] ll_md_real_close+0xa5/0x2a0 [lustre] [13484.211428] ll_md_close+0x274/0x800 [lustre] [13484.217357] ll_file_release+0x128/0xc50 [lustre] [13484.222898] __fput+0x10d/0x440 [13484.226056] ____fput+0x12/0x20 [13484.229442] task_work_run+0xc6/0x110 [13484.233537] exit_to_usermode_loop+0x1dd/0x1f0 [13484.242166] do_syscall_64+0x3d6/0x440 [13484.251749] entry_SYSCALL_64_after_hwframe+0x49/0xae [13484.253803] RIP: 0033:0x7ff54af45cd8 [13484.255059] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [13484.271196] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [13484.275291] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [13484.278680] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [13484.282060] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [13484.289372] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [13484.296003] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13492.055391] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [13492.059115] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [13492.064689] Modules linked in: [13492.067524] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13492.067619] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13492.067686] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13492.067694] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13492.068159] RIP: 0010:native_safe_halt+0x2e/0x50 [13492.068170] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13492.068175] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13492.068179] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [13492.068182] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [13492.068184] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [13492.068186] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [13492.068188] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [13492.068191] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [13492.068194] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13492.068197] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [13492.068206] Call Trace: [13492.068212] [13492.068218] ? show_regs.cold.9+0x22/0x2f [13492.068227] ? watchdog_timer_fn.cold.6+0x62/0x10a [13492.068233] ? __hrtimer_run_queues+0x1c2/0x4d0 [13492.068237] ? watchdog+0x40/0x40 [13492.068241] ? hrtimer_interrupt+0x128/0x2c0 [13492.068245] ? smp_apic_timer_interrupt+0xa6/0x270 [13492.068252] ? apic_timer_interrupt+0xf/0x20 [13492.068257] [13492.068260] ? native_safe_halt+0x2e/0x50 [13492.068264] kvm_wait+0xcd/0xe0 [13492.068271] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13492.068354] do_raw_spin_lock+0x177/0x190 [13492.068361] _raw_spin_lock+0x21/0x30 [13492.068365] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13492.068709] process_one_work+0x2c8/0x700 [13492.068717] worker_thread+0x296/0x6e0 [13492.068721] ? rescuer_thread+0x570/0x570 [13492.068725] kthread+0x1d1/0x200 [13492.068729] ? set_kthread_struct+0x70/0x70 [13492.068733] ret_from_fork+0x1f/0x30 [13492.210078] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13492.237291] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13492.242786] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13492.246816] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [13492.250141] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [13492.261077] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13492.265436] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [13492.269581] RDX: 00000000000072d6 RSI: 0000000000000003 RDI: ffff96848811cae0 [13492.275770] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [13492.280020] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [13492.284314] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13492.287961] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [13492.291793] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13492.294744] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [13492.297713] Call Trace: [13492.298522] [13492.299799] ? show_regs.cold.9+0x22/0x2f [13492.302040] ? watchdog_timer_fn.cold.6+0x62/0x10a [13492.303377] ? __hrtimer_run_queues+0x1c2/0x4d0 [13492.304712] ? watchdog+0x40/0x40 [13492.306038] ? hrtimer_interrupt+0x128/0x2c0 [13492.307687] ? smp_apic_timer_interrupt+0xa6/0x270 [13492.309763] ? apic_timer_interrupt+0xf/0x20 [13492.311620] [13492.312362] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [13492.314809] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13492.317686] do_raw_spin_lock+0x177/0x190 [13492.319363] _raw_spin_lock+0x21/0x30 [13492.321138] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13492.323179] obd_mod_free+0x46/0x140 [ptlrpc] [13492.325701] mdc_commit_open+0x91/0xb0 [mdc] [13492.327549] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13492.330319] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13492.332981] after_reply+0xe2c/0x1780 [ptlrpc] [13492.335739] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13492.338185] ptlrpcd+0xa82/0xe40 [ptlrpc] [13492.339948] ? do_wait_intr+0xf0/0xf0 [13492.341479] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13492.344843] kthread+0x1d1/0x200 [13492.346162] ? set_kthread_struct+0x70/0x70 [13492.348158] ret_from_fork+0x1f/0x30 [13497.312617] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11538s! [13497.323392] Showing busy workqueues and worker pools: [13497.325218] workqueue events: flags=0x0 [13497.330106] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13497.334570] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [13497.340072] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [13497.346841] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [13497.353725] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13497.359768] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [13497.366433] workqueue events_long: flags=0x0 [13497.368489] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13497.372897] pending: lu_objects_flush [obdclass] [13497.376388] workqueue events_unbound: flags=0x2 [13497.378971] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [13497.381985] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [13497.402540] workqueue events_power_efficient: flags=0x80 [13497.404917] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13497.411482] pending: neigh_periodic_work [13497.412978] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13497.418396] pending: do_cache_clean, check_lifetime [13497.424301] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13497.426387] pending: neigh_periodic_work [13497.432529] workqueue writeback: flags=0x4a [13497.434066] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [13497.442246] pending: wb_workfn, wb_workfn [13497.443833] workqueue xprtiod: flags=0xa [13497.445355] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [13497.447174] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [13497.450108] workqueue ipv6_addrconf: flags=0x40008 [13497.451594] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13497.453952] pending: addrconf_verify_work [13497.455278] workqueue knbd0-recv: flags=0x1a [13497.456687] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13497.458907] in-flight: 657:recv_work [13497.460480] workqueue ptlrpc_pinger: flags=0x2 [13497.461973] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13497.463973] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [13497.466310] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11538s workers=4 idle: 20831 12642 28859 [13497.469011] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13487s workers=2 idle: 659 [13508.062100] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [13508.069219] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13508.113506] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13508.118896] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13508.122930] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13508.136053] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13508.150493] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13508.154307] RAX: 0000000000007f0c RBX: ffff96848811cae0 RCX: 0000000000000001 [13508.157325] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [13508.162598] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [13508.170482] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [13508.176302] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [13508.180531] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [13508.182194] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13508.186792] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [13508.189016] Call Trace: [13508.189961] [13508.190672] ? show_regs.cold.9+0x22/0x2f [13508.193688] ? watchdog_timer_fn.cold.6+0x62/0x10a [13508.196711] ? __hrtimer_run_queues+0x1c2/0x4d0 [13508.198981] ? watchdog+0x40/0x40 [13508.203431] ? hrtimer_interrupt+0x128/0x2c0 [13508.206005] ? smp_apic_timer_interrupt+0xa6/0x270 [13508.210707] ? apic_timer_interrupt+0xf/0x20 [13508.213896] [13508.215584] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13508.220112] do_raw_spin_lock+0x177/0x190 [13508.220988] _raw_spin_lock+0x21/0x30 [13508.221791] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13508.226032] ? do_raw_spin_unlock+0x75/0x190 [13508.227613] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13508.231106] ? do_wait_intr+0xf0/0xf0 [13508.235308] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13508.238909] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [13508.241677] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13508.245356] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13508.248042] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13508.251529] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [13508.253112] mdc_intent_lock+0x353/0x7e0 [mdc] [13508.267456] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13508.271360] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13508.278534] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13508.284553] lmv_intent_open+0x338/0x1150 [lmv] [13508.286864] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13508.288921] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13508.291217] lmv_intent_lock+0x1dd/0x4c0 [lmv] [13508.296433] ll_intent_lock+0xc5/0xaa0 [lustre] [13508.297968] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13508.301053] ll_lookup_it+0xe0e/0x2270 [lustre] [13508.303191] ? do_raw_spin_unlock+0x75/0x190 [13508.305002] ? _raw_spin_unlock+0x12/0x30 [13508.312892] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [13508.315790] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [13508.317115] ll_atomic_open+0x36a/0xd70 [lustre] [13508.318941] lookup_open+0x356/0xb40 [13508.320647] do_last+0x313/0xfc0 [13508.321819] ? nd_jump_root+0xe5/0x160 [13508.323416] ? path_init+0x437/0x520 [13508.324911] path_openat+0xf7/0x500 [13508.326710] ? do_raw_spin_unlock+0x75/0x190 [13508.331640] do_filp_open+0x99/0x140 [13508.333984] ? getname_flags+0x6e/0x330 [13508.337661] ? __check_object_size+0xff/0x256 [13508.340803] ? do_raw_spin_unlock+0x75/0x190 [13508.341858] ? _raw_spin_unlock+0x12/0x30 [13508.342841] do_sys_openat2+0x2b4/0x410 [13508.344984] do_sys_open+0x73/0xa0 [13508.346540] __x64_sys_openat+0x24/0x30 [13508.348391] do_syscall_64+0xc1/0x440 [13508.349662] entry_SYSCALL_64_after_hwframe+0x49/0xae [13508.351938] RIP: 0033:0x7ff54af45332 [13508.353217] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [13508.365922] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [13508.370176] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [13508.374700] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [13508.378151] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [13508.383208] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [13508.389839] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [13512.059178] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [13512.066902] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13512.120083] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13512.144553] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13512.154304] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13512.166817] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13512.185079] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13512.190781] RAX: 0000000000007a8d RBX: ffff96848811cae0 RCX: 0000000000000001 [13512.195426] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [13512.199820] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [13512.205007] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [13512.213058] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [13512.220740] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [13512.229179] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13512.241644] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [13512.249755] Call Trace: [13512.253590] [13512.254871] ? show_regs.cold.9+0x22/0x2f [13512.258512] ? watchdog_timer_fn.cold.6+0x62/0x10a [13512.265979] ? __hrtimer_run_queues+0x1c2/0x4d0 [13512.269430] ? watchdog+0x40/0x40 [13512.272749] ? hrtimer_interrupt+0x128/0x2c0 [13512.277473] ? smp_apic_timer_interrupt+0xa6/0x270 [13512.285786] ? apic_timer_interrupt+0xf/0x20 [13512.290938] [13512.291994] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13512.295925] do_raw_spin_lock+0x177/0x190 [13512.299366] _raw_spin_lock+0x21/0x30 [13512.306560] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13512.312385] ? do_raw_spin_unlock+0x75/0x190 [13512.317732] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13512.323923] ? do_wait_intr+0xf0/0xf0 [13512.327703] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13512.332795] mdc_close+0x53b/0xd70 [mdc] [13512.338763] lmv_close+0x18d/0x490 [lmv] [13512.344759] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [13512.352942] ll_close_inode_openhandle+0x493/0xea0 [lustre] [13512.363796] ll_md_real_close+0xa5/0x2a0 [lustre] [13512.373203] ll_md_close+0x274/0x800 [lustre] [13512.378220] ll_file_release+0x128/0xc50 [lustre] [13512.385399] __fput+0x10d/0x440 [13512.389623] ____fput+0x12/0x20 [13512.394649] task_work_run+0xc6/0x110 [13512.399637] exit_to_usermode_loop+0x1dd/0x1f0 [13512.405063] do_syscall_64+0x3d6/0x440 [13512.408731] entry_SYSCALL_64_after_hwframe+0x49/0xae [13512.412131] RIP: 0033:0x7ff54af45cd8 [13512.414251] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [13512.438264] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [13512.449733] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [13512.454802] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [13512.462352] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [13512.467448] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [13512.472201] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13520.064543] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [13520.073430] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13520.116487] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13520.124461] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13520.130388] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13520.133815] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13520.136684] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13520.145751] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13520.149589] RAX: 0000000000005e8f RBX: ffff96848811cae0 RCX: 0000000000000001 [13520.154300] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [13520.158209] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [13520.163272] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [13520.167290] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [13520.172535] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [13520.179641] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13520.183042] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [13520.187273] Call Trace: [13520.188703] [13520.189878] ? show_regs.cold.9+0x22/0x2f [13520.192501] ? watchdog_timer_fn.cold.6+0x62/0x10a [13520.195296] ? __hrtimer_run_queues+0x1c2/0x4d0 [13520.197206] ? watchdog+0x40/0x40 [13520.199092] ? hrtimer_interrupt+0x128/0x2c0 [13520.202881] ? smp_apic_timer_interrupt+0xa6/0x270 [13520.210423] ? apic_timer_interrupt+0xf/0x20 [13520.213256] [13520.214479] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13520.218444] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13520.222458] do_raw_spin_lock+0x177/0x190 [13520.224906] _raw_spin_lock+0x21/0x30 [13520.227785] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13520.231511] process_one_work+0x2c8/0x700 [13520.234235] worker_thread+0x296/0x6e0 [13520.236447] ? rescuer_thread+0x570/0x570 [13520.238678] kthread+0x1d1/0x200 [13520.240315] ? set_kthread_struct+0x70/0x70 [13520.242782] ret_from_fork+0x1f/0x30 [13524.056523] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [13524.061587] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13524.107874] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13524.119901] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13524.132713] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [13524.140512] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [13524.165050] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13524.173853] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [13524.183880] RDX: 00000000000079f4 RSI: 0000000000000003 RDI: ffff96848811cae0 [13524.188992] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [13524.193944] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [13524.198548] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13524.209044] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [13524.216689] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13524.219905] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [13524.224670] Call Trace: [13524.226104] [13524.227569] ? show_regs.cold.9+0x22/0x2f [13524.229665] ? watchdog_timer_fn.cold.6+0x62/0x10a [13524.233087] ? __hrtimer_run_queues+0x1c2/0x4d0 [13524.235973] ? watchdog+0x40/0x40 [13524.239000] ? hrtimer_interrupt+0x128/0x2c0 [13524.240943] ? smp_apic_timer_interrupt+0xa6/0x270 [13524.244691] ? apic_timer_interrupt+0xf/0x20 [13524.247037] [13524.248938] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [13524.252608] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13524.255850] do_raw_spin_lock+0x177/0x190 [13524.257990] _raw_spin_lock+0x21/0x30 [13524.260028] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13524.263589] obd_mod_free+0x46/0x140 [ptlrpc] [13524.266012] mdc_commit_open+0x91/0xb0 [mdc] [13524.268587] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13524.272996] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13524.277099] after_reply+0xe2c/0x1780 [ptlrpc] [13524.279748] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13524.283320] ptlrpcd+0xa82/0xe40 [ptlrpc] [13524.286307] ? do_wait_intr+0xf0/0xf0 [13524.288606] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13524.291671] kthread+0x1d1/0x200 [13524.293624] ? set_kthread_struct+0x70/0x70 [13524.295919] ret_from_fork+0x1f/0x30 [13527.521649] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11568s! [13527.529384] Showing busy workqueues and worker pools: [13527.532247] workqueue events: flags=0x0 [13527.535026] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13527.545101] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [13527.557078] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [13527.567405] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [13527.579445] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13527.587117] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [13527.591374] workqueue events_long: flags=0x0 [13527.595546] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13527.609540] pending: lu_objects_flush [obdclass] [13527.611844] workqueue events_unbound: flags=0x2 [13527.622292] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [13527.636846] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [13527.647629] workqueue events_power_efficient: flags=0x80 [13527.656205] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13527.661844] pending: neigh_periodic_work [13527.669613] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13527.675799] pending: do_cache_clean, check_lifetime [13527.679507] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13527.684357] pending: neigh_periodic_work [13527.687206] workqueue writeback: flags=0x4a [13527.690487] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [13527.694382] pending: wb_workfn, wb_workfn [13527.696785] workqueue xprtiod: flags=0xa [13527.698078] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [13527.704755] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [13527.713386] workqueue ipv6_addrconf: flags=0x40008 [13527.715639] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13527.723230] pending: addrconf_verify_work [13527.727409] workqueue knbd0-recv: flags=0x1a [13527.730551] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13527.736578] in-flight: 657:recv_work [13527.739535] workqueue ptlrpc_pinger: flags=0x2 [13527.741147] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13527.743091] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [13527.749654] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11568s workers=4 idle: 20831 12642 28859 [13527.757803] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13517s workers=2 idle: 659 [13534.152185] rcu: INFO: rcu_sched self-detected stall on CPU [13534.155831] rcu: 0-...!: (6149281 ticks this GP) idle=98a/1/0x4000000000000002 softirq=125154/125154 fqs=959 [13534.159743] (t=11580312 jiffies g=253585 q=6650) [13534.161732] rcu: rcu_sched kthread starved for 11575272 jiffies! g253585 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [13534.167103] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [13534.172193] rcu: RCU grace-period kthread stack dump: [13534.174300] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [13534.179228] Call Trace: [13534.180415] __schedule+0x351/0xcb0 [13534.182094] schedule+0xc0/0x180 [13534.183984] schedule_timeout+0xb4/0x190 [13534.185673] ? do_raw_spin_unlock+0x75/0x190 [13534.187025] ? __next_timer_interrupt+0x160/0x160 [13534.189767] ? prepare_to_swait_event+0xf3/0x140 [13534.192185] rcu_gp_kthread+0x8dd/0x15b0 [13534.194291] ? force_qs_rnp+0x2c0/0x2c0 [13534.195601] kthread+0x1d1/0x200 [13534.197156] ? set_kthread_struct+0x70/0x70 [13534.199380] ret_from_fork+0x1f/0x30 [13534.201432] rcu: Stack dump where RCU GP kthread last ran: [13534.203751] Sending NMI from CPU 0 to CPUs 2: [13534.206481] NMI backtrace for cpu 2 skipped: idling at native_safe_halt+0x2e/0x50 [13534.208173] NMI backtrace for cpu 0 [13534.217849] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13534.223525] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13534.227475] Call Trace: [13534.229395] [13534.230541] ? dump_stack+0xbb/0x10e [13534.232428] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [13534.239208] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [13534.243762] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [13534.248114] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [13534.251250] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [13534.256805] ? rcu_dump_cpu_stacks+0x11b/0x171 [13534.259690] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [13534.265249] ? update_process_times+0x6d/0xc0 [13534.267357] ? tick_sched_handle.isra.18+0x27/0x90 [13534.268952] ? tick_sched_timer+0x53/0xb0 [13534.276700] ? __hrtimer_run_queues+0x1c2/0x4d0 [13534.282586] ? tick_sched_handle.isra.18+0x90/0x90 [13534.287042] ? hrtimer_interrupt+0x128/0x2c0 [13534.292677] ? smp_apic_timer_interrupt+0xa6/0x270 [13534.296468] ? apic_timer_interrupt+0xf/0x20 [13534.300113] [13534.300922] ? native_safe_halt+0x2e/0x50 [13534.309254] ? kvm_wait+0xcd/0xe0 [13534.312847] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13534.319321] ? do_raw_spin_lock+0x177/0x190 [13534.323312] ? _raw_spin_lock+0x21/0x30 [13534.326753] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13534.334713] ? obd_mod_free+0x46/0x140 [ptlrpc] [13534.338140] ? mdc_commit_open+0x91/0xb0 [mdc] [13534.341726] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13534.348796] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13534.355948] ? after_reply+0xe2c/0x1780 [ptlrpc] [13534.360281] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13534.365754] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [13534.368281] ? do_wait_intr+0xf0/0xf0 [13534.369408] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13534.371169] ? kthread+0x1d1/0x200 [13534.373690] ? set_kthread_struct+0x70/0x70 [13534.380366] ? ret_from_fork+0x1f/0x30 [13536.065242] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [13536.083125] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13536.167070] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13536.182760] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13536.194625] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13536.197950] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13536.221413] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13536.225014] RAX: 000000000000248f RBX: ffff96848811cae0 RCX: 0000000000000001 [13536.231720] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [13536.238856] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [13536.250171] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [13536.258389] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [13536.263778] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [13536.273194] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13536.278435] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [13536.287099] Call Trace: [13536.290699] [13536.292681] ? show_regs.cold.9+0x22/0x2f [13536.295781] ? watchdog_timer_fn.cold.6+0x62/0x10a [13536.301065] ? __hrtimer_run_queues+0x1c2/0x4d0 [13536.304079] ? watchdog+0x40/0x40 [13536.305729] ? hrtimer_interrupt+0x128/0x2c0 [13536.308268] ? smp_apic_timer_interrupt+0xa6/0x270 [13536.311946] ? apic_timer_interrupt+0xf/0x20 [13536.314767] [13536.315949] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13536.318858] do_raw_spin_lock+0x177/0x190 [13536.321561] _raw_spin_lock+0x21/0x30 [13536.323205] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13536.325815] ? do_raw_spin_unlock+0x75/0x190 [13536.327873] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13536.331142] ? do_wait_intr+0xf0/0xf0 [13536.333390] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13536.339782] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [13536.350673] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13536.360794] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13536.366556] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13536.371892] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [13536.373373] mdc_intent_lock+0x353/0x7e0 [mdc] [13536.374924] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13536.381395] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13536.385545] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13536.388604] lmv_intent_open+0x338/0x1150 [lmv] [13536.391391] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13536.397783] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13536.399484] lmv_intent_lock+0x1dd/0x4c0 [lmv] [13536.400914] ll_intent_lock+0xc5/0xaa0 [lustre] [13536.414531] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13536.423648] ll_lookup_it+0xe0e/0x2270 [lustre] [13536.430745] ? do_raw_spin_unlock+0x75/0x190 [13536.432760] ? _raw_spin_unlock+0x12/0x30 [13536.436673] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [13536.438488] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [13536.444260] ll_atomic_open+0x36a/0xd70 [lustre] [13536.445841] lookup_open+0x356/0xb40 [13536.446783] do_last+0x313/0xfc0 [13536.452846] ? nd_jump_root+0xe5/0x160 [13536.458440] ? path_init+0x437/0x520 [13536.464097] path_openat+0xf7/0x500 [13536.469810] ? do_raw_spin_unlock+0x75/0x190 [13536.477093] do_filp_open+0x99/0x140 [13536.479560] ? getname_flags+0x6e/0x330 [13536.481693] ? __check_object_size+0xff/0x256 [13536.490873] ? do_raw_spin_unlock+0x75/0x190 [13536.493950] ? _raw_spin_unlock+0x12/0x30 [13536.504881] do_sys_openat2+0x2b4/0x410 [13536.507914] do_sys_open+0x73/0xa0 [13536.509706] __x64_sys_openat+0x24/0x30 [13536.514084] do_syscall_64+0xc1/0x440 [13536.515871] entry_SYSCALL_64_after_hwframe+0x49/0xae [13536.520768] RIP: 0033:0x7ff54af45332 [13536.522861] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [13536.547453] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [13536.550405] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [13536.556471] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [13536.559803] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [13536.570143] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [13536.577873] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [13540.058374] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [13540.062912] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13540.110264] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13540.123985] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13540.131721] RIP: 0010:native_safe_halt+0x2e/0x50 [13540.134475] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13540.156489] RSP: 0018:ffffaf9fc1543ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13540.166041] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [13540.185481] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c20b4154 [13540.200498] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [13540.207188] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [13540.213259] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [13540.224615] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [13540.237743] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13540.243778] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [13540.252952] Call Trace: [13540.255131] [13540.259961] ? show_regs.cold.9+0x22/0x2f [13540.264043] ? watchdog_timer_fn.cold.6+0x62/0x10a [13540.268356] ? __hrtimer_run_queues+0x1c2/0x4d0 [13540.272350] ? watchdog+0x40/0x40 [13540.277304] ? hrtimer_interrupt+0x128/0x2c0 [13540.281473] ? smp_apic_timer_interrupt+0xa6/0x270 [13540.293260] ? apic_timer_interrupt+0xf/0x20 [13540.302292] [13540.303432] ? native_safe_halt+0x2e/0x50 [13540.305370] kvm_wait+0xcd/0xe0 [13540.308530] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13540.315906] do_raw_spin_lock+0x177/0x190 [13540.321793] _raw_spin_lock+0x21/0x30 [13540.326251] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13540.335637] ? do_raw_spin_unlock+0x75/0x190 [13540.344369] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13540.350645] ? do_wait_intr+0xf0/0xf0 [13540.353891] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13540.361774] mdc_close+0x53b/0xd70 [mdc] [13540.364269] lmv_close+0x18d/0x490 [lmv] [13540.370687] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [13540.381617] ll_close_inode_openhandle+0x493/0xea0 [lustre] [13540.388658] ll_md_real_close+0xa5/0x2a0 [lustre] [13540.391163] ll_md_close+0x274/0x800 [lustre] [13540.396504] ll_file_release+0x128/0xc50 [lustre] [13540.401783] __fput+0x10d/0x440 [13540.404554] ____fput+0x12/0x20 [13540.406920] task_work_run+0xc6/0x110 [13540.410803] exit_to_usermode_loop+0x1dd/0x1f0 [13540.414502] do_syscall_64+0x3d6/0x440 [13540.418256] entry_SYSCALL_64_after_hwframe+0x49/0xae [13540.429109] RIP: 0033:0x7ff54af45cd8 [13540.432204] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [13540.449774] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [13540.455757] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [13540.460009] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [13540.464792] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [13540.470915] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [13540.478770] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13558.242113] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11599s! [13558.250135] Showing busy workqueues and worker pools: [13558.256115] workqueue events: flags=0x0 [13558.264386] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13558.279218] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [13558.298445] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [13558.304740] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [13558.319354] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13558.326579] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [13558.330116] workqueue events_long: flags=0x0 [13558.335141] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13558.339107] pending: lu_objects_flush [obdclass] [13558.353240] workqueue events_unbound: flags=0x2 [13558.358068] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [13558.371364] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [13558.389748] workqueue events_power_efficient: flags=0x80 [13558.400694] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13558.408384] pending: neigh_periodic_work [13558.409956] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13558.419463] pending: do_cache_clean, check_lifetime [13558.425346] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13558.431275] pending: neigh_periodic_work [13558.433978] workqueue writeback: flags=0x4a [13558.437300] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [13558.442302] pending: wb_workfn, wb_workfn [13558.445882] workqueue xprtiod: flags=0xa [13558.449708] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [13558.452105] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [13558.460081] workqueue ipv6_addrconf: flags=0x40008 [13558.470328] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13558.483721] pending: addrconf_verify_work [13558.490496] workqueue knbd0-recv: flags=0x1a [13558.495509] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13558.501295] in-flight: 657:recv_work [13558.505610] workqueue ptlrpc_pinger: flags=0x2 [13558.511059] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13558.516638] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [13558.519531] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11599s workers=4 idle: 20831 12642 28859 [13558.523404] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13548s workers=2 idle: 659 [13560.061071] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [13560.065697] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13560.083446] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13560.087840] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13560.091423] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13560.094283] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13560.096558] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13560.107040] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13560.109453] RAX: 0000000000007f8c RBX: ffff96848811cae0 RCX: 0000000000000001 [13560.115359] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [13560.119252] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [13560.122062] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [13560.125237] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [13560.128487] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [13560.132381] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13560.135117] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [13560.139085] Call Trace: [13560.140347] [13560.141483] ? show_regs.cold.9+0x22/0x2f [13560.147610] ? watchdog_timer_fn.cold.6+0x62/0x10a [13560.154395] ? __hrtimer_run_queues+0x1c2/0x4d0 [13560.157375] ? watchdog+0x40/0x40 [13560.165190] ? hrtimer_interrupt+0x128/0x2c0 [13560.170355] ? smp_apic_timer_interrupt+0xa6/0x270 [13560.172700] ? apic_timer_interrupt+0xf/0x20 [13560.191803] [13560.192580] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13560.203744] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13560.210632] do_raw_spin_lock+0x177/0x190 [13560.224286] _raw_spin_lock+0x21/0x30 [13560.229506] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13560.239452] process_one_work+0x2c8/0x700 [13560.243249] worker_thread+0x296/0x6e0 [13560.245660] ? rescuer_thread+0x570/0x570 [13560.248399] kthread+0x1d1/0x200 [13560.249689] ? set_kthread_struct+0x70/0x70 [13560.251849] ret_from_fork+0x1f/0x30 [13564.060086] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [13564.066691] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13564.109076] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13564.119118] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13564.124347] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13564.127387] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13564.145106] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13564.151223] RAX: 00000000000001a5 RBX: ffff96848811cae0 RCX: 0000000000000001 [13564.162609] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [13564.171351] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [13564.175443] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [13564.178762] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [13564.186221] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [13564.198847] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13564.209350] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [13564.219809] Call Trace: [13564.222231] [13564.224211] ? show_regs.cold.9+0x22/0x2f [13564.226295] ? watchdog_timer_fn.cold.6+0x62/0x10a [13564.234327] ? __hrtimer_run_queues+0x1c2/0x4d0 [13564.238456] ? watchdog+0x40/0x40 [13564.243302] ? hrtimer_interrupt+0x128/0x2c0 [13564.245810] ? smp_apic_timer_interrupt+0xa6/0x270 [13564.253362] ? apic_timer_interrupt+0xf/0x20 [13564.259454] [13564.270565] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13564.272385] do_raw_spin_lock+0x177/0x190 [13564.279735] _raw_spin_lock+0x21/0x30 [13564.283381] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13564.288770] ? do_raw_spin_unlock+0x75/0x190 [13564.291685] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13564.294472] ? do_wait_intr+0xf0/0xf0 [13564.296834] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13564.303940] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [13564.307578] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13564.313151] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13564.323188] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13564.326814] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [13564.330061] mdc_intent_lock+0x353/0x7e0 [mdc] [13564.333115] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13564.339908] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13564.352078] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13564.354222] lmv_intent_open+0x338/0x1150 [lmv] [13564.367562] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13564.369491] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13564.388987] lmv_intent_lock+0x1dd/0x4c0 [lmv] [13564.392880] ll_intent_lock+0xc5/0xaa0 [lustre] [13564.400505] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13564.408961] ll_lookup_it+0xe0e/0x2270 [lustre] [13564.417512] ? do_raw_spin_unlock+0x75/0x190 [13564.420737] ? _raw_spin_unlock+0x12/0x30 [13564.428055] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [13564.432040] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [13564.437132] ll_atomic_open+0x36a/0xd70 [lustre] [13564.445476] lookup_open+0x356/0xb40 [13564.450071] do_last+0x313/0xfc0 [13564.459511] ? nd_jump_root+0xe5/0x160 [13564.462837] ? path_init+0x437/0x520 [13564.466991] path_openat+0xf7/0x500 [13564.472634] ? do_raw_spin_unlock+0x75/0x190 [13564.475394] do_filp_open+0x99/0x140 [13564.478437] ? getname_flags+0x6e/0x330 [13564.489383] ? __check_object_size+0xff/0x256 [13564.490758] ? do_raw_spin_unlock+0x75/0x190 [13564.500537] ? _raw_spin_unlock+0x12/0x30 [13564.501976] do_sys_openat2+0x2b4/0x410 [13564.507166] do_sys_open+0x73/0xa0 [13564.510236] __x64_sys_openat+0x24/0x30 [13564.514173] do_syscall_64+0xc1/0x440 [13564.516464] entry_SYSCALL_64_after_hwframe+0x49/0xae [13564.522469] RIP: 0033:0x7ff54af45332 [13564.526662] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [13564.536865] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [13564.543150] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [13564.548957] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [13564.564639] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [13564.577215] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [13564.583591] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [13568.066821] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [13568.088280] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13568.179466] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13568.198092] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13568.214609] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13568.228800] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13568.270591] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13568.284939] RAX: 000000000000781d RBX: ffff96848811cae0 RCX: 0000000000000001 [13568.294558] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [13568.306971] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [13568.310902] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [13568.314918] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [13568.318967] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [13568.323313] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13568.326328] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [13568.333073] Call Trace: [13568.335602] [13568.343077] ? show_regs.cold.9+0x22/0x2f [13568.348013] ? watchdog_timer_fn.cold.6+0x62/0x10a [13568.358121] ? __hrtimer_run_queues+0x1c2/0x4d0 [13568.365890] ? watchdog+0x40/0x40 [13568.370039] ? hrtimer_interrupt+0x128/0x2c0 [13568.377235] ? smp_apic_timer_interrupt+0xa6/0x270 [13568.388248] ? apic_timer_interrupt+0xf/0x20 [13568.391346] [13568.392885] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13568.400393] do_raw_spin_lock+0x177/0x190 [13568.404256] _raw_spin_lock+0x21/0x30 [13568.413205] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13568.423219] ? do_raw_spin_unlock+0x75/0x190 [13568.428958] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13568.437762] ? do_wait_intr+0xf0/0xf0 [13568.439368] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13568.444998] mdc_close+0x53b/0xd70 [mdc] [13568.448614] lmv_close+0x18d/0x490 [lmv] [13568.451358] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [13568.455096] ll_close_inode_openhandle+0x493/0xea0 [lustre] [13568.457979] ll_md_real_close+0xa5/0x2a0 [lustre] [13568.461050] ll_md_close+0x274/0x800 [lustre] [13568.464393] ll_file_release+0x128/0xc50 [lustre] [13568.469696] __fput+0x10d/0x440 [13568.472052] ____fput+0x12/0x20 [13568.473404] task_work_run+0xc6/0x110 [13568.483422] exit_to_usermode_loop+0x1dd/0x1f0 [13568.489556] do_syscall_64+0x3d6/0x440 [13568.492282] entry_SYSCALL_64_after_hwframe+0x49/0xae [13568.496030] RIP: 0033:0x7ff54af45cd8 [13568.498200] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [13568.513147] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [13568.524488] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [13568.527276] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [13568.539292] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [13568.547953] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [13568.556114] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13584.061620] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [13584.066085] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13584.132566] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13584.162420] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13584.176099] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [13584.185563] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [13584.218267] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13584.227633] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [13584.235448] RDX: 00000000000044ba RSI: 0000000000000003 RDI: ffff96848811cae0 [13584.248470] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [13584.260507] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [13584.271649] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13584.278476] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [13584.287479] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13584.293583] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [13584.296077] Call Trace: [13584.301187] [13584.305263] ? show_regs.cold.9+0x22/0x2f [13584.322207] ? watchdog_timer_fn.cold.6+0x62/0x10a [13584.332033] ? __hrtimer_run_queues+0x1c2/0x4d0 [13584.348390] ? watchdog+0x40/0x40 [13584.351459] ? hrtimer_interrupt+0x128/0x2c0 [13584.362008] ? smp_apic_timer_interrupt+0xa6/0x270 [13584.365783] ? apic_timer_interrupt+0xf/0x20 [13584.374023] [13584.378711] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [13584.392701] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13584.394414] do_raw_spin_lock+0x177/0x190 [13584.405996] _raw_spin_lock+0x21/0x30 [13584.410766] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13584.416488] obd_mod_free+0x46/0x140 [ptlrpc] [13584.422653] mdc_commit_open+0x91/0xb0 [mdc] [13584.429193] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13584.436758] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13584.445057] after_reply+0xe2c/0x1780 [ptlrpc] [13584.458115] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13584.479956] ptlrpcd+0xa82/0xe40 [ptlrpc] [13584.497336] ? do_wait_intr+0xf0/0xf0 [13584.498738] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13584.520399] kthread+0x1d1/0x200 [13584.521251] ? set_kthread_struct+0x70/0x70 [13584.522633] ret_from_fork+0x1f/0x30 [13588.063610] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [13588.069999] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13588.121296] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13588.127836] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13588.140855] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13588.149063] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13588.156847] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13588.172633] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13588.179650] RAX: 000000000000661f RBX: ffff96848811cae0 RCX: 0000000000000001 [13588.193250] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [13588.202754] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [13588.208163] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [13588.219925] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [13588.233145] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [13588.238877] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13588.243283] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [13588.248173] Call Trace: [13588.250888] [13588.251702] ? show_regs.cold.9+0x22/0x2f [13588.260930] ? watchdog_timer_fn.cold.6+0x62/0x10a [13588.265739] ? __hrtimer_run_queues+0x1c2/0x4d0 [13588.277212] ? watchdog+0x40/0x40 [13588.282899] ? hrtimer_interrupt+0x128/0x2c0 [13588.285786] ? smp_apic_timer_interrupt+0xa6/0x270 [13588.291687] ? apic_timer_interrupt+0xf/0x20 [13588.298955] [13588.303951] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13588.310431] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13588.320061] do_raw_spin_lock+0x177/0x190 [13588.322634] _raw_spin_lock+0x21/0x30 [13588.326792] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13588.333222] process_one_work+0x2c8/0x700 [13588.339815] worker_thread+0x296/0x6e0 [13588.342746] ? rescuer_thread+0x570/0x570 [13588.347777] kthread+0x1d1/0x200 [13588.353082] ? set_kthread_struct+0x70/0x70 [13588.359144] ret_from_fork+0x1f/0x30 [13588.959419] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11630s! [13588.975245] Showing busy workqueues and worker pools: [13588.986784] workqueue events: flags=0x0 [13588.997888] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13589.013944] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [13589.031575] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [13589.047428] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [13589.063536] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13589.071893] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [13589.081815] workqueue events_long: flags=0x0 [13589.093422] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13589.100723] pending: lu_objects_flush [obdclass] [13589.105846] workqueue events_unbound: flags=0x2 [13589.108734] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [13589.113428] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [13589.125205] workqueue events_power_efficient: flags=0x80 [13589.135552] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13589.143939] pending: neigh_periodic_work [13589.148365] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13589.152858] pending: do_cache_clean, check_lifetime [13589.155650] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13589.163651] pending: neigh_periodic_work [13589.167655] workqueue writeback: flags=0x4a [13589.170692] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [13589.176841] pending: wb_workfn, wb_workfn [13589.179371] workqueue xprtiod: flags=0xa [13589.181221] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [13589.185653] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [13589.193578] workqueue ipv6_addrconf: flags=0x40008 [13589.196203] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13589.202509] pending: addrconf_verify_work [13589.209775] workqueue knbd0-recv: flags=0x1a [13589.211195] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13589.213492] in-flight: 657:recv_work [13589.215016] workqueue ptlrpc_pinger: flags=0x2 [13589.226455] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13589.229970] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [13589.235193] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11630s workers=4 idle: 20831 12642 28859 [13589.253184] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13579s workers=2 idle: 659 [13592.061644] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [13592.069826] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13592.141017] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13592.165725] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13592.181704] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13592.187106] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13592.207098] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13592.220921] RAX: 0000000000001e87 RBX: ffff96848811cae0 RCX: 0000000000000001 [13592.229774] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [13592.245969] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [13592.265947] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [13592.272119] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [13592.280992] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [13592.297404] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13592.302005] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [13592.306036] Call Trace: [13592.308274] [13592.309049] ? show_regs.cold.9+0x22/0x2f [13592.313025] ? watchdog_timer_fn.cold.6+0x62/0x10a [13592.317811] ? __hrtimer_run_queues+0x1c2/0x4d0 [13592.321742] ? watchdog+0x40/0x40 [13592.326999] ? hrtimer_interrupt+0x128/0x2c0 [13592.330003] ? smp_apic_timer_interrupt+0xa6/0x270 [13592.342608] ? apic_timer_interrupt+0xf/0x20 [13592.345543] [13592.348013] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13592.352016] do_raw_spin_lock+0x177/0x190 [13592.355014] _raw_spin_lock+0x21/0x30 [13592.360486] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13592.366480] ? do_raw_spin_unlock+0x75/0x190 [13592.371059] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13592.374606] ? do_wait_intr+0xf0/0xf0 [13592.378024] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13592.381193] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [13592.385554] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13592.391077] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13592.394839] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13592.399007] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [13592.404056] mdc_intent_lock+0x353/0x7e0 [mdc] [13592.407837] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13592.413919] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13592.418022] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13592.421089] lmv_intent_open+0x338/0x1150 [lmv] [13592.425983] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13592.430921] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13592.434020] lmv_intent_lock+0x1dd/0x4c0 [lmv] [13592.441528] ll_intent_lock+0xc5/0xaa0 [lustre] [13592.447068] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13592.453835] ll_lookup_it+0xe0e/0x2270 [lustre] [13592.458493] ? do_raw_spin_unlock+0x75/0x190 [13592.462660] ? _raw_spin_unlock+0x12/0x30 [13592.467512] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [13592.474730] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [13592.480032] ll_atomic_open+0x36a/0xd70 [lustre] [13592.486020] lookup_open+0x356/0xb40 [13592.490909] do_last+0x313/0xfc0 [13592.495711] ? nd_jump_root+0xe5/0x160 [13592.498810] ? path_init+0x437/0x520 [13592.502727] path_openat+0xf7/0x500 [13592.505063] ? do_raw_spin_unlock+0x75/0x190 [13592.512999] do_filp_open+0x99/0x140 [13592.518705] ? getname_flags+0x6e/0x330 [13592.522859] ? __check_object_size+0xff/0x256 [13592.528038] ? do_raw_spin_unlock+0x75/0x190 [13592.531735] ? _raw_spin_unlock+0x12/0x30 [13592.534824] do_sys_openat2+0x2b4/0x410 [13592.537028] do_sys_open+0x73/0xa0 [13592.539566] __x64_sys_openat+0x24/0x30 [13592.541809] do_syscall_64+0xc1/0x440 [13592.543836] entry_SYSCALL_64_after_hwframe+0x49/0xae [13592.548007] RIP: 0033:0x7ff54af45332 [13592.551356] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [13592.564770] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [13592.572532] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [13592.587024] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [13592.592959] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [13592.602420] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [13592.614631] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [13596.066143] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [13596.089080] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13596.157199] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13596.168566] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13596.181661] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13596.186410] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13596.203080] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13596.207925] RAX: 000000000000798d RBX: ffff96848811cae0 RCX: 0000000000000001 [13596.214108] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [13596.221434] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [13596.224435] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [13596.244006] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [13596.258138] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [13596.273726] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13596.275630] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [13596.294581] Call Trace: [13596.304301] [13596.307199] ? show_regs.cold.9+0x22/0x2f [13596.309404] ? watchdog_timer_fn.cold.6+0x62/0x10a [13596.314834] ? __hrtimer_run_queues+0x1c2/0x4d0 [13596.324012] ? watchdog+0x40/0x40 [13596.328083] ? hrtimer_interrupt+0x128/0x2c0 [13596.333531] ? smp_apic_timer_interrupt+0xa6/0x270 [13596.344146] ? apic_timer_interrupt+0xf/0x20 [13596.350993] [13596.359798] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13596.364578] do_raw_spin_lock+0x177/0x190 [13596.371964] _raw_spin_lock+0x21/0x30 [13596.376531] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13596.382356] ? do_raw_spin_unlock+0x75/0x190 [13596.391032] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13596.401092] ? do_wait_intr+0xf0/0xf0 [13596.405313] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13596.415064] mdc_close+0x53b/0xd70 [mdc] [13596.416904] lmv_close+0x18d/0x490 [lmv] [13596.418119] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [13596.429058] ll_close_inode_openhandle+0x493/0xea0 [lustre] [13596.440529] ll_md_real_close+0xa5/0x2a0 [lustre] [13596.444334] ll_md_close+0x274/0x800 [lustre] [13596.455288] ll_file_release+0x128/0xc50 [lustre] [13596.467190] __fput+0x10d/0x440 [13596.477155] ____fput+0x12/0x20 [13596.487137] task_work_run+0xc6/0x110 [13596.494665] exit_to_usermode_loop+0x1dd/0x1f0 [13596.503290] do_syscall_64+0x3d6/0x440 [13596.509463] entry_SYSCALL_64_after_hwframe+0x49/0xae [13596.517238] RIP: 0033:0x7ff54af45cd8 [13596.518147] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [13596.547312] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [13596.559097] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [13596.569379] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [13596.580273] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [13596.596195] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [13596.610053] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13616.055024] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [13616.057487] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13616.059030] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [13616.074495] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13616.074510] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13616.074513] RIP: 0010:native_safe_halt+0x2e/0x50 [13616.074528] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13616.074533] RSP: 0000:ffffaf9fc1b97bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13616.074540] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [13616.074543] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff96848811cae0 [13616.074545] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [13616.074547] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [13616.074550] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13616.074553] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [13616.074556] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13616.074559] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [13616.074568] Call Trace: [13616.074573] [13616.074579] ? show_regs.cold.9+0x22/0x2f [13616.074589] ? watchdog_timer_fn.cold.6+0x62/0x10a [13616.074595] ? __hrtimer_run_queues+0x1c2/0x4d0 [13616.074599] ? watchdog+0x40/0x40 [13616.074604] ? hrtimer_interrupt+0x128/0x2c0 [13616.074608] ? smp_apic_timer_interrupt+0xa6/0x270 [13616.074615] ? apic_timer_interrupt+0xf/0x20 [13616.074619] [13616.074622] ? native_safe_halt+0x2e/0x50 [13616.074626] kvm_wait+0xcd/0xe0 [13616.074633] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13616.074640] do_raw_spin_lock+0x177/0x190 [13616.074645] _raw_spin_lock+0x21/0x30 [13616.074651] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13616.079868] Modules linked in: [13616.083921] obd_mod_free+0x46/0x140 [ptlrpc] [13616.084332] mdc_commit_open+0x91/0xb0 [mdc] [13616.084386] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13616.084591] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13616.084789] after_reply+0xe2c/0x1780 [ptlrpc] [13616.084994] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13616.085320] ptlrpcd+0xa82/0xe40 [ptlrpc] [13616.085575] ? do_wait_intr+0xf0/0xf0 [13616.085586] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13616.096067] lustre(O) [13616.097963] kthread+0x1d1/0x200 [13616.135126] osp(O) [13616.138107] ? set_kthread_struct+0x70/0x70 [13616.138125] ret_from_fork+0x1f/0x30 [13616.363460] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13616.413113] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13616.428755] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13616.439776] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13616.449312] RIP: 0010:native_safe_halt+0x2e/0x50 [13616.450852] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13616.469927] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13616.485989] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [13616.516313] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c2134154 [13616.535731] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [13616.544881] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [13616.557892] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [13616.564769] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [13616.573856] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13616.582725] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [13616.592534] Call Trace: [13616.596836] [13616.599246] ? show_regs.cold.9+0x22/0x2f [13616.608362] ? watchdog_timer_fn.cold.6+0x62/0x10a [13616.616869] ? __hrtimer_run_queues+0x1c2/0x4d0 [13616.622540] ? watchdog+0x40/0x40 [13616.628868] ? hrtimer_interrupt+0x128/0x2c0 [13616.633530] ? smp_apic_timer_interrupt+0xa6/0x270 [13616.638622] ? apic_timer_interrupt+0xf/0x20 [13616.642736] [13616.645938] ? native_safe_halt+0x2e/0x50 [13616.652426] kvm_wait+0xcd/0xe0 [13616.662737] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13616.669902] do_raw_spin_lock+0x177/0x190 [13616.679765] _raw_spin_lock+0x21/0x30 [13616.686287] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13616.704210] process_one_work+0x2c8/0x700 [13616.707998] worker_thread+0x296/0x6e0 [13616.717890] ? rescuer_thread+0x570/0x570 [13616.722129] kthread+0x1d1/0x200 [13616.730221] ? set_kthread_struct+0x70/0x70 [13616.737503] ret_from_fork+0x1f/0x30 [13619.679041] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11660s! [13619.682769] Showing busy workqueues and worker pools: [13619.685082] workqueue events: flags=0x0 [13619.687265] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13619.690739] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [13619.695771] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [13619.699624] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [13619.705749] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13619.709123] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [13619.711438] workqueue events_long: flags=0x0 [13619.712927] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13619.716263] pending: lu_objects_flush [obdclass] [13619.719034] workqueue events_unbound: flags=0x2 [13619.721349] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [13619.724534] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [13619.729065] workqueue events_power_efficient: flags=0x80 [13619.731743] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13619.734957] pending: neigh_periodic_work [13619.737436] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13619.740920] pending: do_cache_clean, check_lifetime [13619.743635] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13619.746694] pending: neigh_periodic_work [13619.748375] workqueue writeback: flags=0x4a [13619.749906] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [13619.752580] pending: wb_workfn, wb_workfn [13619.754677] workqueue xprtiod: flags=0xa [13619.755822] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [13619.758041] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [13619.761371] workqueue ipv6_addrconf: flags=0x40008 [13619.763650] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13619.766806] pending: addrconf_verify_work [13619.768878] workqueue knbd0-recv: flags=0x1a [13619.770911] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13619.773811] in-flight: 657:recv_work [13619.775858] workqueue ptlrpc_pinger: flags=0x2 [13619.778033] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13619.780765] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [13619.783081] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11660s workers=4 idle: 20831 12642 28859 [13619.787093] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13610s workers=2 idle: 659 [13620.060116] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [13620.068870] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13620.103061] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13620.110524] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13620.116309] RIP: 0010:native_safe_halt+0x2e/0x50 [13620.118083] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13620.127892] RSP: 0000:ffffaf9fc15234f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13620.132302] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [13620.135899] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [13620.139616] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [13620.148649] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [13620.160160] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [13620.168855] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [13620.178213] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13620.188385] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [13620.196490] Call Trace: [13620.198686] [13620.201698] ? show_regs.cold.9+0x22/0x2f [13620.206890] ? watchdog_timer_fn.cold.6+0x62/0x10a [13620.211341] ? __hrtimer_run_queues+0x1c2/0x4d0 [13620.216107] ? watchdog+0x40/0x40 [13620.219454] ? hrtimer_interrupt+0x128/0x2c0 [13620.222366] ? smp_apic_timer_interrupt+0xa6/0x270 [13620.229933] ? apic_timer_interrupt+0xf/0x20 [13620.233206] [13620.234951] ? native_safe_halt+0x2e/0x50 [13620.237673] kvm_wait+0xcd/0xe0 [13620.240574] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13620.247943] do_raw_spin_lock+0x177/0x190 [13620.253553] _raw_spin_lock+0x21/0x30 [13620.257911] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13620.265136] ? do_raw_spin_unlock+0x75/0x190 [13620.269621] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13620.277150] ? do_wait_intr+0xf0/0xf0 [13620.279836] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13620.286701] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [13620.292664] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13620.298925] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13620.302718] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13620.305546] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [13620.308633] mdc_intent_lock+0x353/0x7e0 [mdc] [13620.312036] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13620.319743] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13620.323490] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13620.325564] lmv_intent_open+0x338/0x1150 [lmv] [13620.330037] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13620.335933] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13620.338827] lmv_intent_lock+0x1dd/0x4c0 [lmv] [13620.342836] ll_intent_lock+0xc5/0xaa0 [lustre] [13620.346921] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13620.350766] ll_lookup_it+0xe0e/0x2270 [lustre] [13620.353920] ? do_raw_spin_unlock+0x75/0x190 [13620.358539] ? _raw_spin_unlock+0x12/0x30 [13620.361132] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [13620.368443] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [13620.370259] ll_atomic_open+0x36a/0xd70 [lustre] [13620.379410] lookup_open+0x356/0xb40 [13620.382545] do_last+0x313/0xfc0 [13620.385762] ? nd_jump_root+0xe5/0x160 [13620.387883] ? path_init+0x437/0x520 [13620.390521] path_openat+0xf7/0x500 [13620.394918] ? do_raw_spin_unlock+0x75/0x190 [13620.398593] do_filp_open+0x99/0x140 [13620.400974] ? getname_flags+0x6e/0x330 [13620.405751] ? __check_object_size+0xff/0x256 [13620.408997] ? do_raw_spin_unlock+0x75/0x190 [13620.411905] ? _raw_spin_unlock+0x12/0x30 [13620.414561] do_sys_openat2+0x2b4/0x410 [13620.418959] do_sys_open+0x73/0xa0 [13620.423659] __x64_sys_openat+0x24/0x30 [13620.425220] do_syscall_64+0xc1/0x440 [13620.428695] entry_SYSCALL_64_after_hwframe+0x49/0xae [13620.431230] RIP: 0033:0x7ff54af45332 [13620.432941] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [13620.456579] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [13620.461943] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [13620.466038] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [13620.472534] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [13620.476806] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [13620.482813] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [13624.062774] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [13624.084047] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13624.139664] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13624.170468] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13624.173192] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13624.184184] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13624.215446] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13624.224996] RAX: 00000000000070fa RBX: ffff96848811cae0 RCX: 0000000000000001 [13624.229734] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [13624.242387] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [13624.251489] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [13624.258718] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [13624.267264] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [13624.288313] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13624.294403] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [13624.301019] Call Trace: [13624.301941] [13624.308050] ? show_regs.cold.9+0x22/0x2f [13624.320711] ? watchdog_timer_fn.cold.6+0x62/0x10a [13624.336003] ? __hrtimer_run_queues+0x1c2/0x4d0 [13624.340440] ? watchdog+0x40/0x40 [13624.343506] ? hrtimer_interrupt+0x128/0x2c0 [13624.348978] ? smp_apic_timer_interrupt+0xa6/0x270 [13624.354573] ? apic_timer_interrupt+0xf/0x20 [13624.356098] [13624.356795] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13624.362648] do_raw_spin_lock+0x177/0x190 [13624.368579] _raw_spin_lock+0x21/0x30 [13624.375625] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13624.389954] ? do_raw_spin_unlock+0x75/0x190 [13624.398121] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13624.412735] ? do_wait_intr+0xf0/0xf0 [13624.413826] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13624.420584] mdc_close+0x53b/0xd70 [mdc] [13624.429997] lmv_close+0x18d/0x490 [lmv] [13624.437804] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [13624.441541] ll_close_inode_openhandle+0x493/0xea0 [lustre] [13624.458061] ll_md_real_close+0xa5/0x2a0 [lustre] [13624.464034] ll_md_close+0x274/0x800 [lustre] [13624.474195] ll_file_release+0x128/0xc50 [lustre] [13624.476647] __fput+0x10d/0x440 [13624.479044] ____fput+0x12/0x20 [13624.482991] task_work_run+0xc6/0x110 [13624.486794] exit_to_usermode_loop+0x1dd/0x1f0 [13624.493411] do_syscall_64+0x3d6/0x440 [13624.494792] entry_SYSCALL_64_after_hwframe+0x49/0xae [13624.503671] RIP: 0033:0x7ff54af45cd8 [13624.505613] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [13624.522986] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [13624.531989] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [13624.536978] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [13624.547638] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [13624.554719] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [13624.558967] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13644.055223] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [13644.059117] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [13644.067604] Modules linked in: lustre(O) osp(O) [13644.069833] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13644.069924] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13644.070018] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13644.070027] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13644.076447] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13644.076519] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13644.076525] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13644.076528] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [13644.096038] [13644.096057] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13644.096090] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13644.096096] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13644.096103] RAX: 0000000000007510 RBX: ffff96848811cae0 RCX: 0000000000000001 [13644.096105] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [13644.096108] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [13644.096110] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [13644.096113] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [13644.096117] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [13644.096120] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13644.096123] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [13644.096131] Call Trace: [13644.096138] [13644.096148] ? show_regs.cold.9+0x22/0x2f [13644.096169] ? watchdog_timer_fn.cold.6+0x62/0x10a [13644.096178] ? __hrtimer_run_queues+0x1c2/0x4d0 [13644.096184] ? watchdog+0x40/0x40 [13644.096188] ? hrtimer_interrupt+0x128/0x2c0 [13644.096192] ? smp_apic_timer_interrupt+0xa6/0x270 [13644.096203] ? apic_timer_interrupt+0xf/0x20 [13644.096215] [13644.096218] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13644.096223] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13644.096231] do_raw_spin_lock+0x177/0x190 [13644.096237] _raw_spin_lock+0x21/0x30 [13644.096248] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13644.096614] process_one_work+0x2c8/0x700 [13644.096624] worker_thread+0x296/0x6e0 [13644.096628] ? rescuer_thread+0x570/0x570 [13644.096633] kthread+0x1d1/0x200 [13644.096638] ? set_kthread_struct+0x70/0x70 [13644.096642] ret_from_fork+0x1f/0x30 [13644.407772] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [13644.434457] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13644.454965] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [13644.468120] RDX: 00000000000065ed RSI: 0000000000000003 RDI: ffff96848811cae0 [13644.470634] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [13644.487724] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [13644.497295] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13644.510419] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [13644.520283] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13644.528187] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [13644.534275] Call Trace: [13644.536559] [13644.538801] ? show_regs.cold.9+0x22/0x2f [13644.543527] ? watchdog_timer_fn.cold.6+0x62/0x10a [13644.548621] ? __hrtimer_run_queues+0x1c2/0x4d0 [13644.553271] ? watchdog+0x40/0x40 [13644.554840] ? hrtimer_interrupt+0x128/0x2c0 [13644.559487] ? smp_apic_timer_interrupt+0xa6/0x270 [13644.565383] ? apic_timer_interrupt+0xf/0x20 [13644.569415] [13644.572566] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [13644.576773] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13644.583565] do_raw_spin_lock+0x177/0x190 [13644.590665] _raw_spin_lock+0x21/0x30 [13644.592699] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13644.599368] obd_mod_free+0x46/0x140 [ptlrpc] [13644.604954] mdc_commit_open+0x91/0xb0 [mdc] [13644.608388] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13644.614349] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13644.620779] after_reply+0xe2c/0x1780 [ptlrpc] [13644.627684] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13644.631711] ptlrpcd+0xa82/0xe40 [ptlrpc] [13644.636854] ? do_wait_intr+0xf0/0xf0 [13644.638131] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13644.642541] kthread+0x1d1/0x200 [13644.645342] ? set_kthread_struct+0x70/0x70 [13644.648246] ret_from_fork+0x1f/0x30 [13648.062162] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [13648.066778] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13648.121744] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13648.132265] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13648.139858] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13648.146629] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13648.158164] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [13648.162230] RAX: 000000000000643c RBX: ffff96848811cae0 RCX: 0000000000000001 [13648.165305] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [13648.168598] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [13648.170251] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [13648.173662] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [13648.177535] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [13648.183176] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13648.187190] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [13648.190769] Call Trace: [13648.191749] [13648.192721] ? show_regs.cold.9+0x22/0x2f [13648.194744] ? watchdog_timer_fn.cold.6+0x62/0x10a [13648.197768] ? __hrtimer_run_queues+0x1c2/0x4d0 [13648.201211] ? watchdog+0x40/0x40 [13648.205257] ? hrtimer_interrupt+0x128/0x2c0 [13648.208805] ? smp_apic_timer_interrupt+0xa6/0x270 [13648.215616] ? apic_timer_interrupt+0xf/0x20 [13648.218770] [13648.220468] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13648.223261] do_raw_spin_lock+0x177/0x190 [13648.224725] _raw_spin_lock+0x21/0x30 [13648.226501] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13648.232675] ? do_raw_spin_unlock+0x75/0x190 [13648.234725] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13648.237235] ? do_wait_intr+0xf0/0xf0 [13648.244545] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13648.252571] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [13648.257837] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13648.265775] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13648.270471] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13648.273239] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [13648.275714] mdc_intent_lock+0x353/0x7e0 [mdc] [13648.278321] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13648.281247] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13648.286578] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13648.293143] lmv_intent_open+0x338/0x1150 [lmv] [13648.295997] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13648.302990] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13648.306561] lmv_intent_lock+0x1dd/0x4c0 [lmv] [13648.309323] ll_intent_lock+0xc5/0xaa0 [lustre] [13648.311359] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13648.314310] ll_lookup_it+0xe0e/0x2270 [lustre] [13648.318853] ? do_raw_spin_unlock+0x75/0x190 [13648.322475] ? _raw_spin_unlock+0x12/0x30 [13648.327434] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [13648.330627] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [13648.340617] ll_atomic_open+0x36a/0xd70 [lustre] [13648.350599] lookup_open+0x356/0xb40 [13648.351634] do_last+0x313/0xfc0 [13648.352731] ? nd_jump_root+0xe5/0x160 [13648.362251] ? path_init+0x437/0x520 [13648.364859] path_openat+0xf7/0x500 [13648.365720] ? do_raw_spin_unlock+0x75/0x190 [13648.371435] do_filp_open+0x99/0x140 [13648.375538] ? getname_flags+0x6e/0x330 [13648.378869] ? __check_object_size+0xff/0x256 [13648.386550] ? do_raw_spin_unlock+0x75/0x190 [13648.390049] ? _raw_spin_unlock+0x12/0x30 [13648.392472] do_sys_openat2+0x2b4/0x410 [13648.394601] do_sys_open+0x73/0xa0 [13648.398634] __x64_sys_openat+0x24/0x30 [13648.400848] do_syscall_64+0xc1/0x440 [13648.403693] entry_SYSCALL_64_after_hwframe+0x49/0xae [13648.408330] RIP: 0033:0x7ff54af45332 [13648.412603] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [13648.430096] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [13648.436786] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [13648.441702] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [13648.448650] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [13648.453703] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [13648.460020] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [13649.894136] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11691s! [13649.902738] Showing busy workqueues and worker pools: [13649.905009] workqueue events: flags=0x0 [13649.913120] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13649.923581] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [13649.934367] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [13649.939609] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [13649.949967] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13649.953994] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [13649.956683] workqueue events_long: flags=0x0 [13649.961924] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13649.968111] pending: lu_objects_flush [obdclass] [13649.971147] workqueue events_unbound: flags=0x2 [13649.973402] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [13649.976477] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [13649.980385] workqueue events_power_efficient: flags=0x80 [13649.982760] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13649.985781] pending: neigh_periodic_work [13649.989065] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13649.994827] pending: do_cache_clean, check_lifetime [13650.000494] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13650.008525] pending: neigh_periodic_work [13650.011833] workqueue writeback: flags=0x4a [13650.014759] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [13650.019189] pending: wb_workfn, wb_workfn [13650.021814] workqueue xprtiod: flags=0xa [13650.027194] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [13650.033107] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [13650.045233] workqueue ipv6_addrconf: flags=0x40008 [13650.048209] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13650.053459] pending: addrconf_verify_work [13650.060223] workqueue knbd0-recv: flags=0x1a [13650.064496] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13650.070072] in-flight: 657:recv_work [13650.073806] workqueue ptlrpc_pinger: flags=0x2 [13650.087500] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13650.098636] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [13650.101956] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11691s workers=4 idle: 20831 12642 28859 [13650.115742] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13640s workers=2 idle: 659 [13652.063437] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [13652.071465] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13652.166518] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13652.182196] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13652.192282] RIP: 0010:native_safe_halt+0x2e/0x50 [13652.198846] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13652.221565] RSP: 0018:ffffaf9fc1543ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13652.231384] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [13652.241566] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c20b4154 [13652.247108] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [13652.254550] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [13652.258859] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [13652.262266] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [13652.268675] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13652.276065] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [13652.287380] Call Trace: [13652.290584] [13652.293778] ? show_regs.cold.9+0x22/0x2f [13652.300008] ? watchdog_timer_fn.cold.6+0x62/0x10a [13652.303425] ? __hrtimer_run_queues+0x1c2/0x4d0 [13652.306458] ? watchdog+0x40/0x40 [13652.307631] ? hrtimer_interrupt+0x128/0x2c0 [13652.310538] ? smp_apic_timer_interrupt+0xa6/0x270 [13652.313984] ? apic_timer_interrupt+0xf/0x20 [13652.315660] [13652.317486] ? native_safe_halt+0x2e/0x50 [13652.324357] kvm_wait+0xcd/0xe0 [13652.326946] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13652.330038] do_raw_spin_lock+0x177/0x190 [13652.336017] _raw_spin_lock+0x21/0x30 [13652.338704] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13652.344248] ? do_raw_spin_unlock+0x75/0x190 [13652.351716] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13652.353399] ? do_wait_intr+0xf0/0xf0 [13652.360614] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13652.368023] mdc_close+0x53b/0xd70 [mdc] [13652.374648] lmv_close+0x18d/0x490 [lmv] [13652.378534] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [13652.395344] ll_close_inode_openhandle+0x493/0xea0 [lustre] [13652.397313] ll_md_real_close+0xa5/0x2a0 [lustre] [13652.415689] ll_md_close+0x274/0x800 [lustre] [13652.428694] ll_file_release+0x128/0xc50 [lustre] [13652.439557] __fput+0x10d/0x440 [13652.444545] ____fput+0x12/0x20 [13652.455819] task_work_run+0xc6/0x110 [13652.466197] exit_to_usermode_loop+0x1dd/0x1f0 [13652.471496] do_syscall_64+0x3d6/0x440 [13652.479283] entry_SYSCALL_64_after_hwframe+0x49/0xae [13652.495121] RIP: 0033:0x7ff54af45cd8 [13652.496178] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [13652.523996] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [13652.529202] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [13652.542518] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [13652.555304] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [13652.564252] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [13652.576558] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13672.059025] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [13672.061820] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13672.084190] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13672.093748] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13672.097183] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13672.100230] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13672.102979] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13672.111506] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13672.115208] RAX: 0000000000006b14 RBX: ffff96848811cae0 RCX: 0000000000000001 [13672.118426] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [13672.121467] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [13672.124963] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [13672.127983] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [13672.130615] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [13672.133880] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13672.135777] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [13672.139004] Call Trace: [13672.140155] [13672.140994] ? show_regs.cold.9+0x22/0x2f [13672.142870] ? watchdog_timer_fn.cold.6+0x62/0x10a [13672.144604] ? __hrtimer_run_queues+0x1c2/0x4d0 [13672.146452] ? watchdog+0x40/0x40 [13672.148299] ? hrtimer_interrupt+0x128/0x2c0 [13672.150376] ? smp_apic_timer_interrupt+0xa6/0x270 [13672.152949] ? apic_timer_interrupt+0xf/0x20 [13672.154763] [13672.155461] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13672.158172] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13672.160409] do_raw_spin_lock+0x177/0x190 [13672.162140] _raw_spin_lock+0x21/0x30 [13672.164124] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13672.166602] process_one_work+0x2c8/0x700 [13672.168496] worker_thread+0x296/0x6e0 [13672.170142] ? rescuer_thread+0x570/0x570 [13672.171466] kthread+0x1d1/0x200 [13672.172959] ? set_kthread_struct+0x70/0x70 [13672.174803] ret_from_fork+0x1f/0x30 [13676.058244] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [13676.061241] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [13676.074063] Modules linked in: [13676.091021] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13676.091128] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13676.091136] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13676.091139] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13676.091155] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13676.091160] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13676.091165] RAX: 0000000000004d5c RBX: ffff96848811cae0 RCX: 0000000000000001 [13676.091167] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [13676.091169] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [13676.091181] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [13676.091184] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [13676.091187] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [13676.091190] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13676.091193] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [13676.091201] Call Trace: [13676.091207] [13676.091214] ? show_regs.cold.9+0x22/0x2f [13676.091224] ? watchdog_timer_fn.cold.6+0x62/0x10a [13676.091232] ? __hrtimer_run_queues+0x1c2/0x4d0 [13676.091237] ? watchdog+0x40/0x40 [13676.091242] ? hrtimer_interrupt+0x128/0x2c0 [13676.091247] ? smp_apic_timer_interrupt+0xa6/0x270 [13676.091260] ? apic_timer_interrupt+0xf/0x20 [13676.091264] [13676.091267] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13676.091275] do_raw_spin_lock+0x177/0x190 [13676.091280] _raw_spin_lock+0x21/0x30 [13676.091285] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13676.091627] ? do_raw_spin_unlock+0x75/0x190 [13676.091634] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13676.092341] ? do_wait_intr+0xf0/0xf0 [13676.092350] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13676.092544] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [13676.092994] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13676.094010] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13676.096625] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13676.096696] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13676.096703] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13676.096706] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [13676.096723] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [13676.096727] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13676.096732] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [13676.096734] RDX: 00000000000067d1 RSI: 0000000000000003 RDI: ffff96848811cae0 [13676.096737] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [13676.096739] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [13676.096741] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13676.096745] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [13676.096748] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13676.096750] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [13676.096758] Call Trace: [13676.096764] [13676.096779] ? show_regs.cold.9+0x22/0x2f [13676.096789] ? watchdog_timer_fn.cold.6+0x62/0x10a [13676.096796] ? __hrtimer_run_queues+0x1c2/0x4d0 [13676.096800] ? watchdog+0x40/0x40 [13676.096804] ? hrtimer_interrupt+0x128/0x2c0 [13676.096808] ? smp_apic_timer_interrupt+0xa6/0x270 [13676.096817] ? apic_timer_interrupt+0xf/0x20 [13676.096822] [13676.096826] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [13676.096832] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13676.096837] do_raw_spin_lock+0x177/0x190 [13676.096843] _raw_spin_lock+0x21/0x30 [13676.096849] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13676.097171] obd_mod_free+0x46/0x140 [ptlrpc] [13676.097376] mdc_commit_open+0x91/0xb0 [mdc] [13676.097419] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13676.097624] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13676.097848] after_reply+0xe2c/0x1780 [ptlrpc] [13676.098062] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13676.098275] ptlrpcd+0xa82/0xe40 [ptlrpc] [13676.098486] ? do_wait_intr+0xf0/0xf0 [13676.098496] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13676.098706] kthread+0x1d1/0x200 [13676.098712] ? set_kthread_struct+0x70/0x70 [13676.098716] ret_from_fork+0x1f/0x30 [13676.608639] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13676.622357] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [13676.632056] mdc_intent_lock+0x353/0x7e0 [mdc] [13676.643089] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13676.649305] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13676.656280] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13676.662229] lmv_intent_open+0x338/0x1150 [lmv] [13676.670046] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13676.672015] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13676.677351] lmv_intent_lock+0x1dd/0x4c0 [lmv] [13676.683385] ll_intent_lock+0xc5/0xaa0 [lustre] [13676.689691] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13676.695185] ll_lookup_it+0xe0e/0x2270 [lustre] [13676.700138] ? do_raw_spin_unlock+0x75/0x190 [13676.702835] ? _raw_spin_unlock+0x12/0x30 [13676.705233] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [13676.707714] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [13676.716532] ll_atomic_open+0x36a/0xd70 [lustre] [13676.721088] lookup_open+0x356/0xb40 [13676.724465] do_last+0x313/0xfc0 [13676.728238] ? nd_jump_root+0xe5/0x160 [13676.731539] ? path_init+0x437/0x520 [13676.735541] path_openat+0xf7/0x500 [13676.737470] ? do_raw_spin_unlock+0x75/0x190 [13676.739485] do_filp_open+0x99/0x140 [13676.742797] ? getname_flags+0x6e/0x330 [13676.744220] ? __check_object_size+0xff/0x256 [13676.745956] ? do_raw_spin_unlock+0x75/0x190 [13676.747941] ? _raw_spin_unlock+0x12/0x30 [13676.749881] do_sys_openat2+0x2b4/0x410 [13676.753287] do_sys_open+0x73/0xa0 [13676.755067] __x64_sys_openat+0x24/0x30 [13676.756961] do_syscall_64+0xc1/0x440 [13676.758322] entry_SYSCALL_64_after_hwframe+0x49/0xae [13676.762777] RIP: 0033:0x7ff54af45332 [13676.764024] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [13676.778218] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [13676.785308] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [13676.794431] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [13676.800196] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [13676.804909] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [13676.807567] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [13680.058031] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [13680.068787] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13680.161427] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13680.182363] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13680.197084] RIP: 0010:native_safe_halt+0x2e/0x50 [13680.199031] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13680.225121] RSP: 0018:ffffaf9fc1543ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13680.236273] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [13680.245117] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [13680.260509] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [13680.270865] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [13680.274714] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [13680.307124] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [13680.312139] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13680.333148] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [13680.347074] Call Trace: [13680.351175] [13680.356742] ? show_regs.cold.9+0x22/0x2f [13680.362377] ? watchdog_timer_fn.cold.6+0x62/0x10a [13680.368239] ? __hrtimer_run_queues+0x1c2/0x4d0 [13680.370046] ? watchdog+0x40/0x40 [13680.373305] ? hrtimer_interrupt+0x128/0x2c0 [13680.376261] ? smp_apic_timer_interrupt+0xa6/0x270 [13680.383963] ? apic_timer_interrupt+0xf/0x20 [13680.391588] [13680.392832] ? native_safe_halt+0x2e/0x50 [13680.396117] kvm_wait+0xcd/0xe0 [13680.397393] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13680.401110] do_raw_spin_lock+0x177/0x190 [13680.410943] _raw_spin_lock+0x21/0x30 [13680.416379] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13680.422900] ? do_raw_spin_unlock+0x75/0x190 [13680.426089] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13680.432913] ? do_wait_intr+0xf0/0xf0 [13680.434499] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13680.444191] mdc_close+0x53b/0xd70 [mdc] [13680.448306] lmv_close+0x18d/0x490 [lmv] [13680.452234] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [13680.455935] ll_close_inode_openhandle+0x493/0xea0 [lustre] [13680.465080] ll_md_real_close+0xa5/0x2a0 [lustre] [13680.470351] ll_md_close+0x274/0x800 [lustre] [13680.478003] ll_file_release+0x128/0xc50 [lustre] [13680.482495] __fput+0x10d/0x440 [13680.484999] ____fput+0x12/0x20 [13680.487202] task_work_run+0xc6/0x110 [13680.489382] exit_to_usermode_loop+0x1dd/0x1f0 [13680.496347] do_syscall_64+0x3d6/0x440 [13680.499709] entry_SYSCALL_64_after_hwframe+0x49/0xae [13680.507113] RIP: 0033:0x7ff54af45cd8 [13680.512184] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [13680.542444] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [13680.552171] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [13680.561923] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [13680.571521] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [13680.578496] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [13680.585361] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13680.607094] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11721s! [13680.617654] Showing busy workqueues and worker pools: [13680.620497] workqueue events: flags=0x0 [13680.622641] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13680.626339] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [13680.632521] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [13680.639372] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [13680.646079] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13680.651243] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [13680.655043] workqueue events_long: flags=0x0 [13680.657164] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13680.660873] pending: lu_objects_flush [obdclass] [13680.663164] workqueue events_unbound: flags=0x2 [13680.664890] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [13680.671952] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [13680.684478] workqueue events_power_efficient: flags=0x80 [13680.688150] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13680.691782] pending: neigh_periodic_work [13680.694663] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13680.700124] pending: do_cache_clean, check_lifetime [13680.703190] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13680.707946] pending: neigh_periodic_work [13680.711535] workqueue writeback: flags=0x4a [13680.714094] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [13680.717155] pending: wb_workfn, wb_workfn [13680.721349] workqueue xprtiod: flags=0xa [13680.722832] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [13680.728306] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [13680.735888] workqueue ipv6_addrconf: flags=0x40008 [13680.738449] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13680.745965] pending: addrconf_verify_work [13680.747569] workqueue knbd0-recv: flags=0x1a [13680.750855] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13680.755498] in-flight: 657:recv_work [13680.765512] workqueue ptlrpc_pinger: flags=0x2 [13680.767297] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13680.772146] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [13680.775398] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11721s workers=4 idle: 20831 12642 28859 [13680.782489] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13671s workers=2 idle: 659 [13700.059202] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [13700.062463] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13700.094156] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13700.099123] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13700.116941] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13700.120463] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13700.123960] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13700.129859] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13700.135704] RAX: 0000000000007a84 RBX: ffff96848811cae0 RCX: 0000000000000001 [13700.139226] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [13700.142929] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [13700.145645] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [13700.148680] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [13700.151621] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [13700.155164] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13700.157428] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [13700.163278] Call Trace: [13700.164301] [13700.165043] ? show_regs.cold.9+0x22/0x2f [13700.168025] ? watchdog_timer_fn.cold.6+0x62/0x10a [13700.170226] ? __hrtimer_run_queues+0x1c2/0x4d0 [13700.176896] ? watchdog+0x40/0x40 [13700.178387] ? hrtimer_interrupt+0x128/0x2c0 [13700.180923] ? smp_apic_timer_interrupt+0xa6/0x270 [13700.186151] ? apic_timer_interrupt+0xf/0x20 [13700.193976] [13700.196263] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13700.204648] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13700.208171] do_raw_spin_lock+0x177/0x190 [13700.214337] _raw_spin_lock+0x21/0x30 [13700.217256] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13700.225158] process_one_work+0x2c8/0x700 [13700.228547] worker_thread+0x296/0x6e0 [13700.230245] ? rescuer_thread+0x570/0x570 [13700.231984] kthread+0x1d1/0x200 [13700.235094] ? set_kthread_struct+0x70/0x70 [13700.238280] ret_from_fork+0x1f/0x30 [13704.063326] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [13704.072703] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13704.104865] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13704.113145] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13704.118031] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13704.122947] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13704.139582] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13704.143897] RAX: 0000000000003d89 RBX: ffff96848811cae0 RCX: 0000000000000001 [13704.149887] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [13704.159215] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [13704.168099] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [13704.171716] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [13704.178166] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [13704.184741] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13704.187717] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [13704.193544] Call Trace: [13704.194729] [13704.195683] ? show_regs.cold.9+0x22/0x2f [13704.198609] ? watchdog_timer_fn.cold.6+0x62/0x10a [13704.202664] ? __hrtimer_run_queues+0x1c2/0x4d0 [13704.205733] ? watchdog+0x40/0x40 [13704.208872] ? hrtimer_interrupt+0x128/0x2c0 [13704.210264] ? smp_apic_timer_interrupt+0xa6/0x270 [13704.220353] ? apic_timer_interrupt+0xf/0x20 [13704.221988] [13704.223805] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13704.226834] do_raw_spin_lock+0x177/0x190 [13704.228906] _raw_spin_lock+0x21/0x30 [13704.231683] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13704.237716] ? do_raw_spin_unlock+0x75/0x190 [13704.241681] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13704.244104] ? do_wait_intr+0xf0/0xf0 [13704.246808] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13704.252717] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [13704.261728] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13704.269652] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13704.274927] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13704.277444] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [13704.287389] mdc_intent_lock+0x353/0x7e0 [mdc] [13704.290625] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13704.297063] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13704.309217] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13704.310902] lmv_intent_open+0x338/0x1150 [lmv] [13704.317741] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13704.321490] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13704.325101] lmv_intent_lock+0x1dd/0x4c0 [lmv] [13704.331479] ll_intent_lock+0xc5/0xaa0 [lustre] [13704.336162] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13704.346030] ll_lookup_it+0xe0e/0x2270 [lustre] [13704.353776] ? do_raw_spin_unlock+0x75/0x190 [13704.358813] ? _raw_spin_unlock+0x12/0x30 [13704.364908] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [13704.373018] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [13704.377890] ll_atomic_open+0x36a/0xd70 [lustre] [13704.383636] lookup_open+0x356/0xb40 [13704.388776] do_last+0x313/0xfc0 [13704.391121] ? nd_jump_root+0xe5/0x160 [13704.399653] ? path_init+0x437/0x520 [13704.402635] path_openat+0xf7/0x500 [13704.409816] ? do_raw_spin_unlock+0x75/0x190 [13704.413793] do_filp_open+0x99/0x140 [13704.417537] ? getname_flags+0x6e/0x330 [13704.420967] ? __check_object_size+0xff/0x256 [13704.424995] ? do_raw_spin_unlock+0x75/0x190 [13704.430575] ? _raw_spin_unlock+0x12/0x30 [13704.434886] do_sys_openat2+0x2b4/0x410 [13704.439624] do_sys_open+0x73/0xa0 [13704.440747] __x64_sys_openat+0x24/0x30 [13704.442012] do_syscall_64+0xc1/0x440 [13704.448560] entry_SYSCALL_64_after_hwframe+0x49/0xae [13704.453171] RIP: 0033:0x7ff54af45332 [13704.455507] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [13704.472654] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [13704.483641] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [13704.488765] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [13704.496531] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [13704.500328] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [13704.508449] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [13708.055014] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [13708.059479] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13708.061533] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [13708.105823] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13708.105839] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13708.105842] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [13708.105920] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [13708.105926] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13708.105932] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [13708.105935] RDX: 0000000000005d20 RSI: 0000000000000003 RDI: ffff96848811cae0 [13708.105937] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [13708.105939] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [13708.105942] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13708.105946] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [13708.110266] Modules linked in: [13708.122400] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13708.122412] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [13708.122422] Call Trace: [13708.122429] [13708.122435] ? show_regs.cold.9+0x22/0x2f [13708.122449] ? watchdog_timer_fn.cold.6+0x62/0x10a [13708.122737] ? __hrtimer_run_queues+0x1c2/0x4d0 [13708.122744] ? watchdog+0x40/0x40 [13708.122749] ? hrtimer_interrupt+0x128/0x2c0 [13708.122753] ? smp_apic_timer_interrupt+0xa6/0x270 [13708.122826] ? apic_timer_interrupt+0xf/0x20 [13708.122832] [13708.122835] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [13708.122842] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13708.122847] do_raw_spin_lock+0x177/0x190 [13708.122852] _raw_spin_lock+0x21/0x30 [13708.122927] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13708.128586] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13708.132590] obd_mod_free+0x46/0x140 [ptlrpc] [13708.132899] mdc_commit_open+0x91/0xb0 [mdc] [13708.132935] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13708.133132] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13708.133693] after_reply+0xe2c/0x1780 [ptlrpc] [13708.133952] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13708.134150] ptlrpcd+0xa82/0xe40 [ptlrpc] [13708.134425] ? do_wait_intr+0xf0/0xf0 [13708.134438] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13708.135162] kthread+0x1d1/0x200 [13708.135170] ? set_kthread_struct+0x70/0x70 [13708.135174] ret_from_fork+0x1f/0x30 [13708.260824] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13708.273179] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13708.277355] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13708.280639] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13708.289055] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13708.292755] RAX: 00000000000034d3 RBX: ffff96848811cae0 RCX: 0000000000000001 [13708.296672] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [13708.303687] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [13708.307642] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [13708.323503] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [13708.327761] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [13708.332717] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13708.341767] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [13708.348785] Call Trace: [13708.350634] [13708.353055] ? show_regs.cold.9+0x22/0x2f [13708.358197] ? watchdog_timer_fn.cold.6+0x62/0x10a [13708.362304] ? __hrtimer_run_queues+0x1c2/0x4d0 [13708.364652] ? watchdog+0x40/0x40 [13708.367554] ? hrtimer_interrupt+0x128/0x2c0 [13708.373215] ? smp_apic_timer_interrupt+0xa6/0x270 [13708.376017] ? apic_timer_interrupt+0xf/0x20 [13708.378641] [13708.379700] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13708.382859] do_raw_spin_lock+0x177/0x190 [13708.385274] _raw_spin_lock+0x21/0x30 [13708.387841] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13708.391766] ? do_raw_spin_unlock+0x75/0x190 [13708.394130] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13708.397158] ? do_wait_intr+0xf0/0xf0 [13708.399671] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13708.402684] mdc_close+0x53b/0xd70 [mdc] [13708.405145] lmv_close+0x18d/0x490 [lmv] [13708.407510] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [13708.411233] ll_close_inode_openhandle+0x493/0xea0 [lustre] [13708.414676] ll_md_real_close+0xa5/0x2a0 [lustre] [13708.417686] ll_md_close+0x274/0x800 [lustre] [13708.420482] ll_file_release+0x128/0xc50 [lustre] [13708.427171] __fput+0x10d/0x440 [13708.430656] ____fput+0x12/0x20 [13708.433001] task_work_run+0xc6/0x110 [13708.434753] exit_to_usermode_loop+0x1dd/0x1f0 [13708.438346] do_syscall_64+0x3d6/0x440 [13708.441054] entry_SYSCALL_64_after_hwframe+0x49/0xae [13708.450141] RIP: 0033:0x7ff54af45cd8 [13708.453693] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [13708.474037] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [13708.482708] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [13708.487007] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [13708.491765] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [13708.501184] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [13708.505611] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13710.815212] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11751s! [13710.826736] Showing busy workqueues and worker pools: [13710.838475] workqueue events: flags=0x0 [13710.846251] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13710.861421] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [13710.880809] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [13710.894159] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [13710.914660] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13710.934991] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [13710.947027] workqueue events_long: flags=0x0 [13710.950423] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13710.962512] pending: lu_objects_flush [obdclass] [13710.967690] workqueue events_unbound: flags=0x2 [13710.970627] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [13710.977048] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [13710.988991] workqueue events_power_efficient: flags=0x80 [13710.993712] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13710.999294] pending: neigh_periodic_work [13711.002884] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13711.014127] pending: do_cache_clean, check_lifetime [13711.022744] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13711.036578] pending: neigh_periodic_work [13711.043023] workqueue writeback: flags=0x4a [13711.044397] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [13711.054601] pending: wb_workfn, wb_workfn [13711.060202] workqueue xprtiod: flags=0xa [13711.061723] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [13711.069621] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [13711.079669] workqueue ipv6_addrconf: flags=0x40008 [13711.083284] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13711.089724] pending: addrconf_verify_work [13711.099717] workqueue knbd0-recv: flags=0x1a [13711.106204] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13711.110213] in-flight: 657:recv_work [13711.115202] workqueue ptlrpc_pinger: flags=0x2 [13711.123046] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13711.130648] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [13711.136871] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11752s workers=4 idle: 20831 12642 28859 [13711.143595] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13701s workers=2 idle: 659 [13714.162827] rcu: INFO: rcu_sched self-detected stall on CPU [13714.175351] rcu: 0-...!: (6244395 ticks this GP) idle=98a/1/0x4000000000000002 softirq=125154/125154 fqs=959 [13714.186557] (t=11760322 jiffies g=253585 q=6731) [13714.188249] rcu: rcu_sched kthread starved for 11755282 jiffies! g253585 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [13714.199467] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [13714.214084] rcu: RCU grace-period kthread stack dump: [13714.216652] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [13714.230118] Call Trace: [13714.231140] __schedule+0x351/0xcb0 [13714.234464] schedule+0xc0/0x180 [13714.236734] schedule_timeout+0xb4/0x190 [13714.238169] ? do_raw_spin_unlock+0x75/0x190 [13714.239407] ? __next_timer_interrupt+0x160/0x160 [13714.245032] ? prepare_to_swait_event+0xf3/0x140 [13714.250056] rcu_gp_kthread+0x8dd/0x15b0 [13714.253978] ? force_qs_rnp+0x2c0/0x2c0 [13714.257639] kthread+0x1d1/0x200 [13714.263661] ? set_kthread_struct+0x70/0x70 [13714.265782] ret_from_fork+0x1f/0x30 [13714.267134] rcu: Stack dump where RCU GP kthread last ran: [13714.273941] Sending NMI from CPU 0 to CPUs 2: [13714.281571] NMI backtrace for cpu 2 [13714.281578] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13714.281580] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13714.281582] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13714.281585] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13714.281587] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13714.281589] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 [13714.281591] RAX: 0000000000003414 RBX: ffff96848811cae0 RCX: 0000000000000001 [13714.281593] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [13714.281594] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [13714.281596] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [13714.281597] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [13714.281599] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [13714.281600] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13714.281602] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [13714.281603] Call Trace: [13714.281604] [13714.281605] ? show_regs.cold.9+0x22/0x2f [13714.281606] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [13714.281607] ? nmi_cpu_backtrace_handler+0x15/0x40 [13714.281608] ? nmi_handle+0xbd/0x210 [13714.281609] ? default_do_nmi+0x4b/0x1b0 [13714.281610] ? do_nmi+0x280/0x380 [13714.281611] ? end_repeat_nmi+0x16/0x1f [13714.281612] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13714.281613] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13714.281614] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13714.281615] [13714.281616] do_raw_spin_lock+0x177/0x190 [13714.281617] _raw_spin_lock+0x21/0x30 [13714.281618] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13714.281619] process_one_work+0x2c8/0x700 [13714.281621] worker_thread+0x296/0x6e0 [13714.281622] ? rescuer_thread+0x570/0x570 [13714.281623] kthread+0x1d1/0x200 [13714.281624] ? set_kthread_struct+0x70/0x70 [13714.281625] ret_from_fork+0x1f/0x30 [13714.284278] NMI backtrace for cpu 0 [13714.455185] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13714.464652] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13714.476935] Call Trace: [13714.478825] [13714.480418] ? dump_stack+0xbb/0x10e [13714.489457] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [13714.495630] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [13714.500537] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [13714.502370] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [13714.510431] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [13714.513531] ? rcu_dump_cpu_stacks+0x11b/0x171 [13714.519451] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [13714.521257] ? update_process_times+0x6d/0xc0 [13714.528570] ? tick_sched_handle.isra.18+0x27/0x90 [13714.532845] ? tick_sched_timer+0x53/0xb0 [13714.537376] ? __hrtimer_run_queues+0x1c2/0x4d0 [13714.540367] ? tick_sched_handle.isra.18+0x90/0x90 [13714.547541] ? hrtimer_interrupt+0x128/0x2c0 [13714.551169] ? smp_apic_timer_interrupt+0xa6/0x270 [13714.555257] ? apic_timer_interrupt+0xf/0x20 [13714.566661] [13714.570761] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [13714.577997] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13714.582190] ? do_raw_spin_lock+0x177/0x190 [13714.588538] ? _raw_spin_lock+0x21/0x30 [13714.592830] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13714.597836] ? obd_mod_free+0x46/0x140 [ptlrpc] [13714.607740] ? mdc_commit_open+0x91/0xb0 [mdc] [13714.611825] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13714.618771] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13714.624159] ? after_reply+0xe2c/0x1780 [ptlrpc] [13714.627810] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13714.632651] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [13714.641505] ? do_wait_intr+0xf0/0xf0 [13714.644550] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13714.649727] ? kthread+0x1d1/0x200 [13714.653211] ? set_kthread_struct+0x70/0x70 [13714.654807] ? ret_from_fork+0x1f/0x30 [13732.067335] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [13732.079462] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13732.107219] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13732.111328] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13732.114517] RIP: 0010:native_safe_halt+0x2e/0x50 [13732.118413] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13732.126281] RSP: 0000:ffffaf9fc15234f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13732.128897] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [13732.130759] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c21b4154 [13732.132619] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [13732.135669] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [13732.138163] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [13732.139814] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [13732.141917] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13732.143449] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [13732.145214] Call Trace: [13732.145765] [13732.146481] ? show_regs.cold.9+0x22/0x2f [13732.147556] ? watchdog_timer_fn.cold.6+0x62/0x10a [13732.148781] ? __hrtimer_run_queues+0x1c2/0x4d0 [13732.150214] ? watchdog+0x40/0x40 [13732.151269] ? hrtimer_interrupt+0x128/0x2c0 [13732.152334] ? smp_apic_timer_interrupt+0xa6/0x270 [13732.153504] ? apic_timer_interrupt+0xf/0x20 [13732.154494] [13732.155250] ? native_safe_halt+0x2e/0x50 [13732.156418] kvm_wait+0xcd/0xe0 [13732.157212] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13732.158401] do_raw_spin_lock+0x177/0x190 [13732.159478] _raw_spin_lock+0x21/0x30 [13732.160389] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13732.161788] ? do_raw_spin_unlock+0x75/0x190 [13732.162922] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13732.164556] ? do_wait_intr+0xf0/0xf0 [13732.165737] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13732.168025] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [13732.170478] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13732.173652] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13732.176453] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13732.179371] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [13732.181632] mdc_intent_lock+0x353/0x7e0 [mdc] [13732.183384] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13732.185592] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13732.187693] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13732.189529] lmv_intent_open+0x338/0x1150 [lmv] [13732.190687] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13732.192206] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13732.193505] lmv_intent_lock+0x1dd/0x4c0 [lmv] [13732.194432] ll_intent_lock+0xc5/0xaa0 [lustre] [13732.195499] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13732.196896] ll_lookup_it+0xe0e/0x2270 [lustre] [13732.198433] ? do_raw_spin_unlock+0x75/0x190 [13732.199573] ? _raw_spin_unlock+0x12/0x30 [13732.201470] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [13732.203810] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [13732.205823] ll_atomic_open+0x36a/0xd70 [lustre] [13732.207977] lookup_open+0x356/0xb40 [13732.209527] do_last+0x313/0xfc0 [13732.210778] ? nd_jump_root+0xe5/0x160 [13732.212334] ? path_init+0x437/0x520 [13732.213831] path_openat+0xf7/0x500 [13732.215451] ? do_raw_spin_unlock+0x75/0x190 [13732.217504] do_filp_open+0x99/0x140 [13732.219024] ? getname_flags+0x6e/0x330 [13732.220615] ? __check_object_size+0xff/0x256 [13732.222535] ? do_raw_spin_unlock+0x75/0x190 [13732.224747] ? _raw_spin_unlock+0x12/0x30 [13732.226620] do_sys_openat2+0x2b4/0x410 [13732.227756] do_sys_open+0x73/0xa0 [13732.229009] __x64_sys_openat+0x24/0x30 [13732.230523] do_syscall_64+0xc1/0x440 [13732.231666] entry_SYSCALL_64_after_hwframe+0x49/0xae [13732.233079] RIP: 0033:0x7ff54af45332 [13732.234370] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [13732.241875] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [13732.245922] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [13732.249274] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [13732.253323] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [13732.256822] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [13732.263491] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [13736.068876] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [13736.075963] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13736.115697] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13736.132961] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13736.141903] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13736.145945] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13736.158650] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13736.162986] RAX: 0000000000007f8c RBX: ffff96848811cae0 RCX: 0000000000000001 [13736.169354] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [13736.173943] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [13736.178784] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [13736.182922] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [13736.187594] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [13736.192760] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13736.196918] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [13736.200688] Call Trace: [13736.201546] [13736.203430] ? show_regs.cold.9+0x22/0x2f [13736.204928] ? watchdog_timer_fn.cold.6+0x62/0x10a [13736.206972] ? __hrtimer_run_queues+0x1c2/0x4d0 [13736.209519] ? watchdog+0x40/0x40 [13736.211952] ? hrtimer_interrupt+0x128/0x2c0 [13736.216606] ? smp_apic_timer_interrupt+0xa6/0x270 [13736.219936] ? apic_timer_interrupt+0xf/0x20 [13736.226459] [13736.228591] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13736.232721] do_raw_spin_lock+0x177/0x190 [13736.233792] _raw_spin_lock+0x21/0x30 [13736.237988] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13736.240135] ? do_raw_spin_unlock+0x75/0x190 [13736.244871] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13736.253777] ? do_wait_intr+0xf0/0xf0 [13736.257461] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13736.266540] mdc_close+0x53b/0xd70 [mdc] [13736.269339] lmv_close+0x18d/0x490 [lmv] [13736.272569] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [13736.281079] ll_close_inode_openhandle+0x493/0xea0 [lustre] [13736.288157] ll_md_real_close+0xa5/0x2a0 [lustre] [13736.295928] ll_md_close+0x274/0x800 [lustre] [13736.297818] ll_file_release+0x128/0xc50 [lustre] [13736.310831] __fput+0x10d/0x440 [13736.312964] ____fput+0x12/0x20 [13736.322971] task_work_run+0xc6/0x110 [13736.324533] exit_to_usermode_loop+0x1dd/0x1f0 [13736.326665] do_syscall_64+0x3d6/0x440 [13736.329275] entry_SYSCALL_64_after_hwframe+0x49/0xae [13736.333279] RIP: 0033:0x7ff54af45cd8 [13736.335393] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [13736.352233] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [13736.354521] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [13736.362999] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [13736.365625] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [13736.372882] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [13736.381933] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13740.059090] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [13740.061287] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [13740.069580] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13740.070784] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13740.070792] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13740.070799] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13740.072281] RIP: 0010:native_safe_halt+0x2e/0x50 [13740.072294] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13740.072298] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13740.072303] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [13740.072305] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c2134154 [13740.072307] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [13740.072309] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [13740.072312] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [13740.072316] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [13740.072318] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13740.072320] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [13740.072329] Call Trace: [13740.072337] [13740.085553] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13740.085636] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13740.085646] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13740.085650] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [13740.085666] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [13740.085670] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13740.085675] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [13740.085677] RDX: 00000000000055b0 RSI: 0000000000000003 RDI: ffff96848811cae0 [13740.085680] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [13740.085682] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [13740.085685] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13740.085688] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [13740.085691] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13740.085693] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [13740.085702] Call Trace: [13740.085709] [13740.085715] ? show_regs.cold.9+0x22/0x2f [13740.085737] ? watchdog_timer_fn.cold.6+0x62/0x10a [13740.085746] ? __hrtimer_run_queues+0x1c2/0x4d0 [13740.085751] ? watchdog+0x40/0x40 [13740.085759] ? hrtimer_interrupt+0x128/0x2c0 [13740.085763] ? smp_apic_timer_interrupt+0xa6/0x270 [13740.085772] ? apic_timer_interrupt+0xf/0x20 [13740.085776] [13740.085779] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [13740.085783] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13740.085788] do_raw_spin_lock+0x177/0x190 [13740.085792] _raw_spin_lock+0x21/0x30 [13740.085799] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13740.086068] obd_mod_free+0x46/0x140 [ptlrpc] [13740.086274] mdc_commit_open+0x91/0xb0 [mdc] [13740.086313] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13740.086522] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13740.086739] after_reply+0xe2c/0x1780 [ptlrpc] [13740.086944] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13740.087139] ptlrpcd+0xa82/0xe40 [ptlrpc] [13740.087354] ? do_wait_intr+0xf0/0xf0 [13740.087364] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13740.147480] ? show_regs.cold.9+0x22/0x2f [13740.147505] ? watchdog_timer_fn.cold.6+0x62/0x10a [13740.147512] ? __hrtimer_run_queues+0x1c2/0x4d0 [13740.147517] ? watchdog+0x40/0x40 [13740.147521] ? hrtimer_interrupt+0x128/0x2c0 [13740.147526] ? smp_apic_timer_interrupt+0xa6/0x270 [13740.147535] ? apic_timer_interrupt+0xf/0x20 [13740.147540] [13740.147545] ? native_safe_halt+0x2e/0x50 [13740.147550] kvm_wait+0xcd/0xe0 [13740.147560] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13740.147571] do_raw_spin_lock+0x177/0x190 [13740.147576] _raw_spin_lock+0x21/0x30 [13740.147581] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13740.150141] process_one_work+0x2c8/0x700 [13740.150160] worker_thread+0x296/0x6e0 [13740.150165] ? rescuer_thread+0x570/0x570 [13740.150170] kthread+0x1d1/0x200 [13740.150175] ? set_kthread_struct+0x70/0x70 [13740.150180] ret_from_fork+0x1f/0x30 [13740.653893] kthread+0x1d1/0x200 [13740.663499] ? set_kthread_struct+0x70/0x70 [13740.682127] ret_from_fork+0x1f/0x30 [13741.535743] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11782s! [13741.540943] Showing busy workqueues and worker pools: [13741.544429] workqueue events: flags=0x0 [13741.547172] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13741.552055] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [13741.557431] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [13741.560721] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [13741.567862] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13741.571599] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [13741.576280] workqueue events_long: flags=0x0 [13741.579444] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13741.585246] pending: lu_objects_flush [obdclass] [13741.597119] workqueue events_unbound: flags=0x2 [13741.603870] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [13741.617096] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [13741.627509] workqueue events_power_efficient: flags=0x80 [13741.633408] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13741.646738] pending: neigh_periodic_work [13741.654546] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13741.660916] pending: do_cache_clean, check_lifetime [13741.668759] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13741.673208] pending: neigh_periodic_work [13741.679150] workqueue writeback: flags=0x4a [13741.687548] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [13741.695871] pending: wb_workfn, wb_workfn [13741.697432] workqueue xprtiod: flags=0xa [13741.703401] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [13741.710532] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [13741.717836] workqueue ipv6_addrconf: flags=0x40008 [13741.720727] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13741.725039] pending: addrconf_verify_work [13741.727414] workqueue knbd0-recv: flags=0x1a [13741.730230] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13741.735665] in-flight: 657:recv_work [13741.739235] workqueue ptlrpc_pinger: flags=0x2 [13741.742428] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13741.757083] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [13741.768704] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11782s workers=4 idle: 20831 12642 28859 [13741.782448] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13732s workers=2 idle: 659 [13760.060027] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [13760.063481] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13760.085503] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13760.093483] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13760.097781] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13760.101114] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13760.114879] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13760.123772] RAX: 0000000000005ac9 RBX: ffff96848811cae0 RCX: 0000000000000001 [13760.132268] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [13760.139576] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [13760.144453] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [13760.152060] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [13760.157936] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [13760.161796] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13760.164621] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [13760.167597] Call Trace: [13760.168738] [13760.170089] ? show_regs.cold.9+0x22/0x2f [13760.172390] ? watchdog_timer_fn.cold.6+0x62/0x10a [13760.174494] ? __hrtimer_run_queues+0x1c2/0x4d0 [13760.176510] ? watchdog+0x40/0x40 [13760.183181] ? hrtimer_interrupt+0x128/0x2c0 [13760.185462] ? smp_apic_timer_interrupt+0xa6/0x270 [13760.188611] ? apic_timer_interrupt+0xf/0x20 [13760.190121] [13760.190605] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13760.193301] do_raw_spin_lock+0x177/0x190 [13760.195199] _raw_spin_lock+0x21/0x30 [13760.197085] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13760.198572] ? do_raw_spin_unlock+0x75/0x190 [13760.199899] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13760.205303] ? do_wait_intr+0xf0/0xf0 [13760.207256] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13760.209319] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [13760.211365] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13760.213526] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13760.216227] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13760.218628] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [13760.220576] mdc_intent_lock+0x353/0x7e0 [mdc] [13760.221741] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13760.223069] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13760.230613] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13760.235483] lmv_intent_open+0x338/0x1150 [lmv] [13760.238110] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13760.240596] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13760.243707] lmv_intent_lock+0x1dd/0x4c0 [lmv] [13760.248351] ll_intent_lock+0xc5/0xaa0 [lustre] [13760.251266] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13760.257624] ll_lookup_it+0xe0e/0x2270 [lustre] [13760.259776] ? do_raw_spin_unlock+0x75/0x190 [13760.268444] ? _raw_spin_unlock+0x12/0x30 [13760.273340] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [13760.279231] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [13760.289630] ll_atomic_open+0x36a/0xd70 [lustre] [13760.295580] lookup_open+0x356/0xb40 [13760.297072] do_last+0x313/0xfc0 [13760.298091] ? nd_jump_root+0xe5/0x160 [13760.300628] ? path_init+0x437/0x520 [13760.302640] path_openat+0xf7/0x500 [13760.305354] ? do_raw_spin_unlock+0x75/0x190 [13760.307472] do_filp_open+0x99/0x140 [13760.312432] ? getname_flags+0x6e/0x330 [13760.315066] ? __check_object_size+0xff/0x256 [13760.317688] ? do_raw_spin_unlock+0x75/0x190 [13760.320429] ? _raw_spin_unlock+0x12/0x30 [13760.322190] do_sys_openat2+0x2b4/0x410 [13760.327311] do_sys_open+0x73/0xa0 [13760.329108] __x64_sys_openat+0x24/0x30 [13760.331423] do_syscall_64+0xc1/0x440 [13760.333441] entry_SYSCALL_64_after_hwframe+0x49/0xae [13760.335776] RIP: 0033:0x7ff54af45332 [13760.338424] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [13760.347425] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [13760.352772] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [13760.357272] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [13760.361764] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [13760.366763] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [13760.371129] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [13764.059508] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [13764.068207] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13764.129854] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13764.144511] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13764.156822] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13764.164362] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13764.190536] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13764.206659] RAX: 0000000000005612 RBX: ffff96848811cae0 RCX: 0000000000000001 [13764.216104] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [13764.226038] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [13764.233453] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [13764.242056] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [13764.249700] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [13764.258028] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13764.262776] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [13764.274620] Call Trace: [13764.278258] [13764.279344] ? show_regs.cold.9+0x22/0x2f [13764.281586] ? watchdog_timer_fn.cold.6+0x62/0x10a [13764.288187] ? __hrtimer_run_queues+0x1c2/0x4d0 [13764.292753] ? watchdog+0x40/0x40 [13764.299576] ? hrtimer_interrupt+0x128/0x2c0 [13764.302866] ? smp_apic_timer_interrupt+0xa6/0x270 [13764.311312] ? apic_timer_interrupt+0xf/0x20 [13764.318710] [13764.322129] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13764.328859] do_raw_spin_lock+0x177/0x190 [13764.331564] _raw_spin_lock+0x21/0x30 [13764.334729] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13764.343928] ? do_raw_spin_unlock+0x75/0x190 [13764.348212] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13764.357231] ? do_wait_intr+0xf0/0xf0 [13764.369464] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13764.372819] mdc_close+0x53b/0xd70 [mdc] [13764.377238] lmv_close+0x18d/0x490 [lmv] [13764.378342] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [13764.381687] ll_close_inode_openhandle+0x493/0xea0 [lustre] [13764.386637] ll_md_real_close+0xa5/0x2a0 [lustre] [13764.388720] ll_md_close+0x274/0x800 [lustre] [13764.390904] ll_file_release+0x128/0xc50 [lustre] [13764.394162] __fput+0x10d/0x440 [13764.397479] ____fput+0x12/0x20 [13764.398600] task_work_run+0xc6/0x110 [13764.399886] exit_to_usermode_loop+0x1dd/0x1f0 [13764.404478] do_syscall_64+0x3d6/0x440 [13764.409395] entry_SYSCALL_64_after_hwframe+0x49/0xae [13764.415076] RIP: 0033:0x7ff54af45cd8 [13764.416144] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [13764.426491] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [13764.430971] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [13764.437585] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [13764.444511] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [13764.452606] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [13764.457625] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13768.055012] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [13768.062777] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [13768.067013] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13768.067086] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13768.067103] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13768.067105] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [13768.067120] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [13768.067124] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13768.067128] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [13768.067130] RDX: 0000000000000100 RSI: 0000000000000003 RDI: ffff96848811cae0 [13768.067132] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [13768.067134] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [13768.067136] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13768.067139] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [13768.067141] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13768.067143] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [13768.067150] Call Trace: [13768.067157] [13768.067163] ? show_regs.cold.9+0x22/0x2f [13768.067173] ? watchdog_timer_fn.cold.6+0x62/0x10a [13768.067180] ? __hrtimer_run_queues+0x1c2/0x4d0 [13768.067183] ? watchdog+0x40/0x40 [13768.067188] ? hrtimer_interrupt+0x128/0x2c0 [13768.067192] ? smp_apic_timer_interrupt+0xa6/0x270 [13768.067199] ? apic_timer_interrupt+0xf/0x20 [13768.067203] [13768.067206] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [13768.067209] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13768.067214] do_raw_spin_lock+0x177/0x190 [13768.067219] _raw_spin_lock+0x21/0x30 [13768.067225] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13768.067495] obd_mod_free+0x46/0x140 [ptlrpc] [13768.067722] mdc_commit_open+0x91/0xb0 [mdc] [13768.067760] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13768.067958] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13768.068153] after_reply+0xe2c/0x1780 [ptlrpc] [13768.068350] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13768.068548] ptlrpcd+0xa82/0xe40 [ptlrpc] [13768.068765] ? do_wait_intr+0xf0/0xf0 [13768.068776] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13768.069004] kthread+0x1d1/0x200 [13768.069010] ? set_kthread_struct+0x70/0x70 [13768.069014] ret_from_fork+0x1f/0x30 [13768.236180] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13768.260542] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13768.265334] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13768.272945] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13768.276169] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13768.278923] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13768.288181] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13768.291814] RAX: 000000000000700b RBX: ffff96848811cae0 RCX: 0000000000000001 [13768.294685] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [13768.299091] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [13768.301715] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [13768.305402] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [13768.308553] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [13768.312293] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13768.315147] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [13768.318675] Call Trace: [13768.319772] [13768.320805] ? show_regs.cold.9+0x22/0x2f [13768.322639] ? watchdog_timer_fn.cold.6+0x62/0x10a [13768.324950] ? __hrtimer_run_queues+0x1c2/0x4d0 [13768.326656] ? watchdog+0x40/0x40 [13768.327887] ? hrtimer_interrupt+0x128/0x2c0 [13768.329676] ? smp_apic_timer_interrupt+0xa6/0x270 [13768.331936] ? apic_timer_interrupt+0xf/0x20 [13768.334320] [13768.335333] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13768.337643] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13768.340270] do_raw_spin_lock+0x177/0x190 [13768.342276] _raw_spin_lock+0x21/0x30 [13768.344159] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13768.346890] process_one_work+0x2c8/0x700 [13768.349222] worker_thread+0x296/0x6e0 [13768.351471] ? rescuer_thread+0x570/0x570 [13768.353384] kthread+0x1d1/0x200 [13768.355138] ? set_kthread_struct+0x70/0x70 [13768.357086] ret_from_fork+0x1f/0x30 [13772.257872] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11813s! [13772.262070] Showing busy workqueues and worker pools: [13772.264701] workqueue events: flags=0x0 [13772.266220] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13772.270549] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [13772.276892] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [13772.280706] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [13772.289230] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13772.292984] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [13772.296795] workqueue events_long: flags=0x0 [13772.299452] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13772.305017] pending: lu_objects_flush [obdclass] [13772.310412] workqueue events_unbound: flags=0x2 [13772.312349] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [13772.322863] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [13772.334031] workqueue events_power_efficient: flags=0x80 [13772.336590] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13772.340620] pending: neigh_periodic_work [13772.343100] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13772.347272] pending: do_cache_clean, check_lifetime [13772.350865] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13772.354441] pending: neigh_periodic_work [13772.356533] workqueue writeback: flags=0x4a [13772.358728] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [13772.362044] pending: wb_workfn, wb_workfn [13772.364908] workqueue xprtiod: flags=0xa [13772.371903] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [13772.378458] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [13772.384990] workqueue ipv6_addrconf: flags=0x40008 [13772.387821] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13772.391722] pending: addrconf_verify_work [13772.394675] workqueue knbd0-recv: flags=0x1a [13772.397391] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13772.401456] in-flight: 657:recv_work [13772.403250] workqueue ptlrpc_pinger: flags=0x2 [13772.405594] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13772.408237] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [13772.411694] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11813s workers=4 idle: 20831 12642 28859 [13772.416295] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13762s workers=2 idle: 659 [13788.060010] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [13788.065667] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13788.106064] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13788.113438] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13788.119179] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13788.125638] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13788.155125] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13788.164170] RAX: 0000000000007586 RBX: ffff96848811cae0 RCX: 0000000000000001 [13788.171463] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [13788.184365] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [13788.186832] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [13788.197880] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [13788.210128] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [13788.221699] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13788.228220] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [13788.236596] Call Trace: [13788.237787] [13788.242999] ? show_regs.cold.9+0x22/0x2f [13788.250529] ? watchdog_timer_fn.cold.6+0x62/0x10a [13788.256066] ? __hrtimer_run_queues+0x1c2/0x4d0 [13788.260219] ? watchdog+0x40/0x40 [13788.264515] ? hrtimer_interrupt+0x128/0x2c0 [13788.267192] ? smp_apic_timer_interrupt+0xa6/0x270 [13788.271443] ? apic_timer_interrupt+0xf/0x20 [13788.275236] [13788.277630] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13788.282924] do_raw_spin_lock+0x177/0x190 [13788.286213] _raw_spin_lock+0x21/0x30 [13788.293110] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13788.298194] ? do_raw_spin_unlock+0x75/0x190 [13788.301296] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13788.305396] ? do_wait_intr+0xf0/0xf0 [13788.307658] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13788.310914] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [13788.314080] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13788.318259] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13788.321910] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13788.324399] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [13788.326221] mdc_intent_lock+0x353/0x7e0 [mdc] [13788.328468] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13788.331384] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13788.334485] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13788.337218] lmv_intent_open+0x338/0x1150 [lmv] [13788.341258] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13788.344251] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13788.348214] lmv_intent_lock+0x1dd/0x4c0 [lmv] [13788.354227] ll_intent_lock+0xc5/0xaa0 [lustre] [13788.360000] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13788.364638] ll_lookup_it+0xe0e/0x2270 [lustre] [13788.369213] ? do_raw_spin_unlock+0x75/0x190 [13788.372521] ? _raw_spin_unlock+0x12/0x30 [13788.375054] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [13788.378444] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [13788.385926] ll_atomic_open+0x36a/0xd70 [lustre] [13788.390769] lookup_open+0x356/0xb40 [13788.392282] do_last+0x313/0xfc0 [13788.394247] ? nd_jump_root+0xe5/0x160 [13788.396940] ? path_init+0x437/0x520 [13788.402494] path_openat+0xf7/0x500 [13788.407418] ? do_raw_spin_unlock+0x75/0x190 [13788.411219] do_filp_open+0x99/0x140 [13788.414004] ? getname_flags+0x6e/0x330 [13788.416428] ? __check_object_size+0xff/0x256 [13788.423037] ? do_raw_spin_unlock+0x75/0x190 [13788.427145] ? _raw_spin_unlock+0x12/0x30 [13788.433393] do_sys_openat2+0x2b4/0x410 [13788.439033] do_sys_open+0x73/0xa0 [13788.449850] __x64_sys_openat+0x24/0x30 [13788.451469] do_syscall_64+0xc1/0x440 [13788.457371] entry_SYSCALL_64_after_hwframe+0x49/0xae [13788.464951] RIP: 0033:0x7ff54af45332 [13788.468237] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [13788.487213] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [13788.495804] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [13788.500050] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [13788.504576] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [13788.507999] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [13788.510665] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [13792.059304] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [13792.065035] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13792.090028] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13792.096120] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13792.101810] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13792.105214] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13792.118407] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13792.123100] RAX: 0000000000007d0e RBX: ffff96848811cae0 RCX: 0000000000000001 [13792.127430] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [13792.131808] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [13792.135825] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [13792.139132] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [13792.145069] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [13792.165491] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13792.175269] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [13792.180284] Call Trace: [13792.182194] [13792.183465] ? show_regs.cold.9+0x22/0x2f [13792.187264] ? watchdog_timer_fn.cold.6+0x62/0x10a [13792.190082] ? __hrtimer_run_queues+0x1c2/0x4d0 [13792.192181] ? watchdog+0x40/0x40 [13792.193522] ? hrtimer_interrupt+0x128/0x2c0 [13792.195407] ? smp_apic_timer_interrupt+0xa6/0x270 [13792.197685] ? apic_timer_interrupt+0xf/0x20 [13792.199680] [13792.200674] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13792.203757] do_raw_spin_lock+0x177/0x190 [13792.205702] _raw_spin_lock+0x21/0x30 [13792.207850] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13792.210090] ? do_raw_spin_unlock+0x75/0x190 [13792.215090] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13792.219387] ? do_wait_intr+0xf0/0xf0 [13792.222392] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13792.227691] mdc_close+0x53b/0xd70 [mdc] [13792.230135] lmv_close+0x18d/0x490 [lmv] [13792.232783] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [13792.239091] ll_close_inode_openhandle+0x493/0xea0 [lustre] [13792.250041] ll_md_real_close+0xa5/0x2a0 [lustre] [13792.261259] ll_md_close+0x274/0x800 [lustre] [13792.265138] ll_file_release+0x128/0xc50 [lustre] [13792.268728] __fput+0x10d/0x440 [13792.271343] ____fput+0x12/0x20 [13792.273337] task_work_run+0xc6/0x110 [13792.277202] exit_to_usermode_loop+0x1dd/0x1f0 [13792.280276] do_syscall_64+0x3d6/0x440 [13792.284039] entry_SYSCALL_64_after_hwframe+0x49/0xae [13792.287566] RIP: 0033:0x7ff54af45cd8 [13792.290262] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [13792.322283] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [13792.340302] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [13792.354509] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [13792.364929] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [13792.377067] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [13792.379625] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13796.069594] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [13796.077752] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13796.117565] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13796.128146] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13796.138069] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13796.147074] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13796.161203] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13796.194088] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13796.203711] RAX: 0000000000004186 RBX: ffff96848811cae0 RCX: 0000000000000001 [13796.222426] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [13796.229213] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [13796.234153] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [13796.237391] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [13796.263302] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [13796.273852] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13796.276321] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [13796.301987] Call Trace: [13796.303328] [13796.304084] ? show_regs.cold.9+0x22/0x2f [13796.316480] ? watchdog_timer_fn.cold.6+0x62/0x10a [13796.318114] ? __hrtimer_run_queues+0x1c2/0x4d0 [13796.334068] ? watchdog+0x40/0x40 [13796.340914] ? hrtimer_interrupt+0x128/0x2c0 [13796.350281] ? smp_apic_timer_interrupt+0xa6/0x270 [13796.351686] ? apic_timer_interrupt+0xf/0x20 [13796.358378] [13796.361299] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13796.364328] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13796.367302] do_raw_spin_lock+0x177/0x190 [13796.371034] _raw_spin_lock+0x21/0x30 [13796.376028] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13796.378904] process_one_work+0x2c8/0x700 [13796.380353] worker_thread+0x296/0x6e0 [13796.385165] ? rescuer_thread+0x570/0x570 [13796.387636] kthread+0x1d1/0x200 [13796.390168] ? set_kthread_struct+0x70/0x70 [13796.392230] ret_from_fork+0x1f/0x30 [13800.055155] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [13800.063662] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13800.083431] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13800.088316] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13800.092059] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [13800.095038] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [13800.104343] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13800.108026] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [13800.111307] RDX: 0000000000006042 RSI: 0000000000000003 RDI: ffff96848811cae0 [13800.114233] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [13800.116694] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [13800.119524] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13800.122098] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [13800.125194] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13800.127584] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [13800.130812] Call Trace: [13800.131806] [13800.132733] ? show_regs.cold.9+0x22/0x2f [13800.134378] ? watchdog_timer_fn.cold.6+0x62/0x10a [13800.136349] ? __hrtimer_run_queues+0x1c2/0x4d0 [13800.138192] ? watchdog+0x40/0x40 [13800.139499] ? hrtimer_interrupt+0x128/0x2c0 [13800.141069] ? smp_apic_timer_interrupt+0xa6/0x270 [13800.143082] ? apic_timer_interrupt+0xf/0x20 [13800.145075] [13800.145988] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [13800.148798] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13800.151866] do_raw_spin_lock+0x177/0x190 [13800.153470] _raw_spin_lock+0x21/0x30 [13800.155023] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13800.157828] obd_mod_free+0x46/0x140 [ptlrpc] [13800.160135] mdc_commit_open+0x91/0xb0 [mdc] [13800.162135] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13800.164099] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13800.166858] after_reply+0xe2c/0x1780 [ptlrpc] [13800.168940] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13800.171243] ptlrpcd+0xa82/0xe40 [ptlrpc] [13800.173094] ? do_wait_intr+0xf0/0xf0 [13800.174659] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13800.177077] kthread+0x1d1/0x200 [13800.178305] ? set_kthread_struct+0x70/0x70 [13800.180487] ret_from_fork+0x1f/0x30 [13802.463293] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11843s! [13802.467849] Showing busy workqueues and worker pools: [13802.470073] workqueue events: flags=0x0 [13802.472183] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13802.475193] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [13802.478971] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [13802.482078] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [13802.487679] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13802.492127] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [13802.496696] workqueue events_long: flags=0x0 [13802.505926] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13802.513060] pending: lu_objects_flush [obdclass] [13802.520247] workqueue events_unbound: flags=0x2 [13802.528105] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [13802.538320] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [13802.548315] workqueue events_power_efficient: flags=0x80 [13802.552196] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13802.555954] pending: neigh_periodic_work [13802.558489] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13802.563995] pending: do_cache_clean, check_lifetime [13802.572914] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13802.577315] pending: neigh_periodic_work [13802.587102] workqueue writeback: flags=0x4a [13802.592615] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [13802.603196] pending: wb_workfn, wb_workfn [13802.605925] workqueue xprtiod: flags=0xa [13802.608090] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [13802.612143] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [13802.616754] workqueue ipv6_addrconf: flags=0x40008 [13802.623833] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13802.636457] pending: addrconf_verify_work [13802.641860] workqueue knbd0-recv: flags=0x1a [13802.645931] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13802.651137] in-flight: 657:recv_work [13802.652919] workqueue ptlrpc_pinger: flags=0x2 [13802.654768] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13802.661415] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [13802.666336] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11843s workers=4 idle: 20831 12642 28859 [13802.672978] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13792s workers=2 idle: 659 [13816.062699] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [13816.074229] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13816.131422] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13816.139654] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13816.148145] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13816.157210] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13816.168775] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13816.182538] RAX: 0000000000007e8a RBX: ffff96848811cae0 RCX: 0000000000000001 [13816.190765] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [13816.197717] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [13816.205067] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [13816.211564] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [13816.222513] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [13816.229441] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13816.236639] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [13816.243621] Call Trace: [13816.246618] [13816.249212] ? show_regs.cold.9+0x22/0x2f [13816.254041] ? watchdog_timer_fn.cold.6+0x62/0x10a [13816.259668] ? __hrtimer_run_queues+0x1c2/0x4d0 [13816.266735] ? watchdog+0x40/0x40 [13816.271217] ? hrtimer_interrupt+0x128/0x2c0 [13816.276161] ? smp_apic_timer_interrupt+0xa6/0x270 [13816.279857] ? apic_timer_interrupt+0xf/0x20 [13816.284749] [13816.286081] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13816.298019] do_raw_spin_lock+0x177/0x190 [13816.302786] _raw_spin_lock+0x21/0x30 [13816.306984] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13816.313109] ? do_raw_spin_unlock+0x75/0x190 [13816.320793] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13816.327542] ? do_wait_intr+0xf0/0xf0 [13816.331660] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13816.336610] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [13816.339788] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13816.343748] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13816.348085] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13816.351175] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [13816.354754] mdc_intent_lock+0x353/0x7e0 [mdc] [13816.358764] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13816.365073] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13816.372626] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13816.378700] lmv_intent_open+0x338/0x1150 [lmv] [13816.383863] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13816.386704] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13816.388986] lmv_intent_lock+0x1dd/0x4c0 [lmv] [13816.392063] ll_intent_lock+0xc5/0xaa0 [lustre] [13816.394720] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13816.401229] ll_lookup_it+0xe0e/0x2270 [lustre] [13816.405731] ? do_raw_spin_unlock+0x75/0x190 [13816.411038] ? _raw_spin_unlock+0x12/0x30 [13816.415422] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [13816.420474] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [13816.424460] ll_atomic_open+0x36a/0xd70 [lustre] [13816.428568] lookup_open+0x356/0xb40 [13816.432153] do_last+0x313/0xfc0 [13816.434759] ? nd_jump_root+0xe5/0x160 [13816.439717] ? path_init+0x437/0x520 [13816.442226] path_openat+0xf7/0x500 [13816.445697] ? do_raw_spin_unlock+0x75/0x190 [13816.452479] do_filp_open+0x99/0x140 [13816.455137] ? getname_flags+0x6e/0x330 [13816.460012] ? __check_object_size+0xff/0x256 [13816.465122] ? do_raw_spin_unlock+0x75/0x190 [13816.470110] ? _raw_spin_unlock+0x12/0x30 [13816.475005] do_sys_openat2+0x2b4/0x410 [13816.478588] do_sys_open+0x73/0xa0 [13816.480451] __x64_sys_openat+0x24/0x30 [13816.483049] do_syscall_64+0xc1/0x440 [13816.485109] entry_SYSCALL_64_after_hwframe+0x49/0xae [13816.489106] RIP: 0033:0x7ff54af45332 [13816.492651] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [13816.519415] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [13816.525677] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [13816.532706] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [13816.536297] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [13816.556138] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [13816.574591] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [13820.058038] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [13820.068443] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13820.124137] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13820.138312] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13820.145721] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13820.151262] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13820.167509] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13820.171868] RAX: 0000000000007387 RBX: ffff96848811cae0 RCX: 0000000000000001 [13820.177158] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [13820.184723] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [13820.192487] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [13820.199038] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [13820.207110] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [13820.213509] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13820.215963] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [13820.225207] Call Trace: [13820.226161] [13820.226945] ? show_regs.cold.9+0x22/0x2f [13820.233997] ? watchdog_timer_fn.cold.6+0x62/0x10a [13820.238170] ? __hrtimer_run_queues+0x1c2/0x4d0 [13820.244019] ? watchdog+0x40/0x40 [13820.246648] ? hrtimer_interrupt+0x128/0x2c0 [13820.258657] ? smp_apic_timer_interrupt+0xa6/0x270 [13820.261029] ? apic_timer_interrupt+0xf/0x20 [13820.264224] [13820.268312] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13820.276036] do_raw_spin_lock+0x177/0x190 [13820.281719] _raw_spin_lock+0x21/0x30 [13820.285707] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13820.289758] ? do_raw_spin_unlock+0x75/0x190 [13820.300608] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13820.308042] ? do_wait_intr+0xf0/0xf0 [13820.315360] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13820.324955] mdc_close+0x53b/0xd70 [mdc] [13820.328937] lmv_close+0x18d/0x490 [lmv] [13820.335160] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [13820.343982] ll_close_inode_openhandle+0x493/0xea0 [lustre] [13820.346114] ll_md_real_close+0xa5/0x2a0 [lustre] [13820.349366] ll_md_close+0x274/0x800 [lustre] [13820.354090] ll_file_release+0x128/0xc50 [lustre] [13820.359533] __fput+0x10d/0x440 [13820.364174] ____fput+0x12/0x20 [13820.365288] task_work_run+0xc6/0x110 [13820.368934] exit_to_usermode_loop+0x1dd/0x1f0 [13820.371178] do_syscall_64+0x3d6/0x440 [13820.377380] entry_SYSCALL_64_after_hwframe+0x49/0xae [13820.381362] RIP: 0033:0x7ff54af45cd8 [13820.385701] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [13820.399560] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [13820.405088] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [13820.411290] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [13820.417156] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [13820.423824] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [13820.431265] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13824.066387] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [13824.076820] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13824.115398] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13824.120805] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13824.127565] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13824.131828] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13824.147673] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13824.165467] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13824.176575] RAX: 0000000000007916 RBX: ffff96848811cae0 RCX: 0000000000000001 [13824.183558] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [13824.188789] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [13824.194723] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [13824.202497] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [13824.205847] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [13824.213128] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13824.220895] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [13824.225860] Call Trace: [13824.229711] [13824.231795] ? show_regs.cold.9+0x22/0x2f [13824.237652] ? watchdog_timer_fn.cold.6+0x62/0x10a [13824.240298] ? __hrtimer_run_queues+0x1c2/0x4d0 [13824.241845] ? watchdog+0x40/0x40 [13824.249376] ? hrtimer_interrupt+0x128/0x2c0 [13824.254813] ? smp_apic_timer_interrupt+0xa6/0x270 [13824.258504] ? apic_timer_interrupt+0xf/0x20 [13824.260877] [13824.262740] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13824.270830] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13824.273298] do_raw_spin_lock+0x177/0x190 [13824.275677] _raw_spin_lock+0x21/0x30 [13824.279644] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13824.289492] process_one_work+0x2c8/0x700 [13824.290985] worker_thread+0x296/0x6e0 [13824.298798] ? rescuer_thread+0x570/0x570 [13824.309130] kthread+0x1d1/0x200 [13824.316757] ? set_kthread_struct+0x70/0x70 [13824.318254] ret_from_fork+0x1f/0x30 [13828.055420] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [13828.063368] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13828.108381] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13828.122109] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13828.133505] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [13828.136740] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [13828.149200] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13828.155767] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [13828.161109] RDX: 0000000000001467 RSI: 0000000000000003 RDI: ffff96848811cae0 [13828.164747] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [13828.168626] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [13828.171619] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13828.174408] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [13828.177593] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13828.182312] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [13828.186687] Call Trace: [13828.187918] [13828.189158] ? show_regs.cold.9+0x22/0x2f [13828.191067] ? watchdog_timer_fn.cold.6+0x62/0x10a [13828.193605] ? __hrtimer_run_queues+0x1c2/0x4d0 [13828.196312] ? watchdog+0x40/0x40 [13828.198774] ? hrtimer_interrupt+0x128/0x2c0 [13828.201041] ? smp_apic_timer_interrupt+0xa6/0x270 [13828.205839] ? apic_timer_interrupt+0xf/0x20 [13828.215554] [13828.216517] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [13828.219831] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13828.222218] do_raw_spin_lock+0x177/0x190 [13828.223782] _raw_spin_lock+0x21/0x30 [13828.225083] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13828.227611] obd_mod_free+0x46/0x140 [ptlrpc] [13828.229884] mdc_commit_open+0x91/0xb0 [mdc] [13828.231486] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13828.234258] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13828.237190] after_reply+0xe2c/0x1780 [ptlrpc] [13828.239547] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13828.242327] ptlrpcd+0xa82/0xe40 [ptlrpc] [13828.244674] ? do_wait_intr+0xf0/0xf0 [13828.245805] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13828.248357] kthread+0x1d1/0x200 [13828.249771] ? set_kthread_struct+0x70/0x70 [13828.251044] ret_from_fork+0x1f/0x30 [13833.184754] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11874s! [13833.196918] Showing busy workqueues and worker pools: [13833.206649] workqueue events: flags=0x0 [13833.208159] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13833.216428] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [13833.233651] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [13833.241529] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [13833.249533] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13833.257732] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [13833.261060] workqueue events_long: flags=0x0 [13833.264949] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13833.267571] pending: lu_objects_flush [obdclass] [13833.276896] workqueue events_unbound: flags=0x2 [13833.280458] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [13833.282764] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [13833.295760] workqueue events_power_efficient: flags=0x80 [13833.302060] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13833.309268] pending: neigh_periodic_work [13833.314388] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13833.318808] pending: do_cache_clean, check_lifetime [13833.324735] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13833.331017] pending: neigh_periodic_work [13833.337966] workqueue writeback: flags=0x4a [13833.345500] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [13833.354762] pending: wb_workfn, wb_workfn [13833.357508] workqueue xprtiod: flags=0xa [13833.362305] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [13833.377832] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [13833.396190] workqueue ipv6_addrconf: flags=0x40008 [13833.408866] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13833.414679] pending: addrconf_verify_work [13833.422610] workqueue knbd0-recv: flags=0x1a [13833.425670] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13833.435144] in-flight: 657:recv_work [13833.443844] workqueue ptlrpc_pinger: flags=0x2 [13833.448995] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13833.456017] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [13833.464414] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11874s workers=4 idle: 20831 12642 28859 [13833.473441] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13823s workers=2 idle: 659 [13844.061402] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [13844.066605] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13844.096186] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13844.102149] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13844.106049] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13844.109263] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13844.118608] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13844.122560] RAX: 0000000000007713 RBX: ffff96848811cae0 RCX: 0000000000000001 [13844.126504] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [13844.130330] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [13844.134815] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [13844.139212] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [13844.144724] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [13844.148834] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13844.152184] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [13844.156198] Call Trace: [13844.157609] [13844.158791] ? show_regs.cold.9+0x22/0x2f [13844.160688] ? watchdog_timer_fn.cold.6+0x62/0x10a [13844.163782] ? __hrtimer_run_queues+0x1c2/0x4d0 [13844.166551] ? watchdog+0x40/0x40 [13844.168099] ? hrtimer_interrupt+0x128/0x2c0 [13844.170446] ? smp_apic_timer_interrupt+0xa6/0x270 [13844.172770] ? apic_timer_interrupt+0xf/0x20 [13844.175203] [13844.176306] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13844.179425] do_raw_spin_lock+0x177/0x190 [13844.181336] _raw_spin_lock+0x21/0x30 [13844.183125] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13844.185742] ? do_raw_spin_unlock+0x75/0x190 [13844.187469] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13844.190026] ? do_wait_intr+0xf0/0xf0 [13844.191372] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13844.193800] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [13844.196459] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13844.199412] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13844.202348] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13844.204930] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [13844.207302] mdc_intent_lock+0x353/0x7e0 [mdc] [13844.208997] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13844.211722] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13844.214534] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13844.216804] lmv_intent_open+0x338/0x1150 [lmv] [13844.219090] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13844.221711] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13844.224355] lmv_intent_lock+0x1dd/0x4c0 [lmv] [13844.226365] ll_intent_lock+0xc5/0xaa0 [lustre] [13844.228680] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13844.230911] ll_lookup_it+0xe0e/0x2270 [lustre] [13844.233289] ? do_raw_spin_unlock+0x75/0x190 [13844.234625] ? _raw_spin_unlock+0x12/0x30 [13844.236281] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [13844.238518] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [13844.240915] ll_atomic_open+0x36a/0xd70 [lustre] [13844.242856] lookup_open+0x356/0xb40 [13844.244279] do_last+0x313/0xfc0 [13844.245719] ? nd_jump_root+0xe5/0x160 [13844.247485] ? path_init+0x437/0x520 [13844.249214] path_openat+0xf7/0x500 [13844.250709] ? do_raw_spin_unlock+0x75/0x190 [13844.255683] do_filp_open+0x99/0x140 [13844.260118] ? getname_flags+0x6e/0x330 [13844.262211] ? __check_object_size+0xff/0x256 [13844.264592] ? do_raw_spin_unlock+0x75/0x190 [13844.267627] ? _raw_spin_unlock+0x12/0x30 [13844.269760] do_sys_openat2+0x2b4/0x410 [13844.272359] do_sys_open+0x73/0xa0 [13844.273897] __x64_sys_openat+0x24/0x30 [13844.275787] do_syscall_64+0xc1/0x440 [13844.277504] entry_SYSCALL_64_after_hwframe+0x49/0xae [13844.280474] RIP: 0033:0x7ff54af45332 [13844.282518] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [13844.297708] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [13844.308783] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [13844.324301] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [13844.331456] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [13844.335401] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [13844.342676] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [13848.058814] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [13848.066914] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13848.127097] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13848.139361] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13848.156094] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13848.161711] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13848.183108] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13848.190745] RAX: 00000000000062a4 RBX: ffff96848811cae0 RCX: 0000000000000001 [13848.206153] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [13848.209202] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [13848.220278] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [13848.230508] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [13848.247332] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [13848.262555] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13848.268319] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [13848.278742] Call Trace: [13848.281487] [13848.282457] ? show_regs.cold.9+0x22/0x2f [13848.292709] ? watchdog_timer_fn.cold.6+0x62/0x10a [13848.300173] ? __hrtimer_run_queues+0x1c2/0x4d0 [13848.304371] ? watchdog+0x40/0x40 [13848.311028] ? hrtimer_interrupt+0x128/0x2c0 [13848.323787] ? smp_apic_timer_interrupt+0xa6/0x270 [13848.325427] ? apic_timer_interrupt+0xf/0x20 [13848.326991] [13848.332384] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13848.334260] do_raw_spin_lock+0x177/0x190 [13848.342635] _raw_spin_lock+0x21/0x30 [13848.347921] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13848.354598] ? do_raw_spin_unlock+0x75/0x190 [13848.359914] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13848.364177] ? do_wait_intr+0xf0/0xf0 [13848.367763] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13848.377316] mdc_close+0x53b/0xd70 [mdc] [13848.384252] lmv_close+0x18d/0x490 [lmv] [13848.387377] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [13848.389722] ll_close_inode_openhandle+0x493/0xea0 [lustre] [13848.403051] ll_md_real_close+0xa5/0x2a0 [lustre] [13848.407854] ll_md_close+0x274/0x800 [lustre] [13848.413634] ll_file_release+0x128/0xc50 [lustre] [13848.417658] __fput+0x10d/0x440 [13848.423690] ____fput+0x12/0x20 [13848.426821] task_work_run+0xc6/0x110 [13848.428093] exit_to_usermode_loop+0x1dd/0x1f0 [13848.439238] do_syscall_64+0x3d6/0x440 [13848.446539] entry_SYSCALL_64_after_hwframe+0x49/0xae [13848.454608] RIP: 0033:0x7ff54af45cd8 [13848.457741] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [13848.488676] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [13848.502422] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [13848.516748] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [13848.525629] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [13848.535163] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [13848.544786] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13852.063376] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [13852.067864] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13852.136194] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13852.150936] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13852.158721] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13852.181325] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13852.196025] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13852.226530] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13852.232135] RAX: 0000000000005306 RBX: ffff96848811cae0 RCX: 0000000000000001 [13852.238326] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [13852.248112] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [13852.256440] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [13852.262421] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [13852.268462] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [13852.278504] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13852.282432] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [13852.298760] Call Trace: [13852.303037] [13852.306611] ? show_regs.cold.9+0x22/0x2f [13852.311918] ? watchdog_timer_fn.cold.6+0x62/0x10a [13852.322501] ? __hrtimer_run_queues+0x1c2/0x4d0 [13852.328117] ? watchdog+0x40/0x40 [13852.329347] ? hrtimer_interrupt+0x128/0x2c0 [13852.336185] ? smp_apic_timer_interrupt+0xa6/0x270 [13852.340465] ? apic_timer_interrupt+0xf/0x20 [13852.342285] [13852.343091] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13852.349488] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13852.353944] do_raw_spin_lock+0x177/0x190 [13852.356624] _raw_spin_lock+0x21/0x30 [13852.360366] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13852.369231] process_one_work+0x2c8/0x700 [13852.372333] worker_thread+0x296/0x6e0 [13852.373958] ? rescuer_thread+0x570/0x570 [13852.376581] kthread+0x1d1/0x200 [13852.380190] ? set_kthread_struct+0x70/0x70 [13852.384070] ret_from_fork+0x1f/0x30 [13860.060041] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [13860.076594] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13860.152989] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13860.173428] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13860.181373] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [13860.185225] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [13860.204822] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13860.219319] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [13860.232264] RDX: 000000000000285e RSI: 0000000000000003 RDI: ffff96848811cae0 [13860.249763] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [13860.264270] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [13860.273466] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13860.293476] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [13860.310807] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13860.322025] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [13860.334117] Call Trace: [13860.337528] [13860.342579] ? show_regs.cold.9+0x22/0x2f [13860.348699] ? watchdog_timer_fn.cold.6+0x62/0x10a [13860.356581] ? __hrtimer_run_queues+0x1c2/0x4d0 [13860.363400] ? watchdog+0x40/0x40 [13860.371771] ? hrtimer_interrupt+0x128/0x2c0 [13860.379910] ? smp_apic_timer_interrupt+0xa6/0x270 [13860.381504] ? apic_timer_interrupt+0xf/0x20 [13860.396381] [13860.401657] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [13860.414793] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13860.420644] do_raw_spin_lock+0x177/0x190 [13860.426552] _raw_spin_lock+0x21/0x30 [13860.431707] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13860.437546] obd_mod_free+0x46/0x140 [ptlrpc] [13860.443208] mdc_commit_open+0x91/0xb0 [mdc] [13860.447053] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13860.458524] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13860.466411] after_reply+0xe2c/0x1780 [ptlrpc] [13860.470788] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13860.478409] ptlrpcd+0xa82/0xe40 [ptlrpc] [13860.488524] ? do_wait_intr+0xf0/0xf0 [13860.493606] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13860.505517] kthread+0x1d1/0x200 [13860.506766] ? set_kthread_struct+0x70/0x70 [13860.510745] ret_from_fork+0x1f/0x30 [13863.909303] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11905s! [13863.918403] Showing busy workqueues and worker pools: [13863.921647] workqueue events: flags=0x0 [13863.928789] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13863.935452] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [13863.939832] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [13863.944096] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [13863.957216] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13863.965277] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [13863.972435] workqueue events_long: flags=0x0 [13863.976300] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13863.981611] pending: lu_objects_flush [obdclass] [13863.987095] workqueue events_unbound: flags=0x2 [13863.991266] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [13863.996438] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [13864.002970] workqueue events_power_efficient: flags=0x80 [13864.004591] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13864.007182] pending: neigh_periodic_work [13864.008594] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13864.011076] pending: do_cache_clean, check_lifetime [13864.012827] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13864.015021] pending: neigh_periodic_work [13864.016537] workqueue writeback: flags=0x4a [13864.017777] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [13864.019883] pending: wb_workfn, wb_workfn [13864.021201] workqueue xprtiod: flags=0xa [13864.022529] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [13864.024322] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [13864.027144] workqueue ipv6_addrconf: flags=0x40008 [13864.028574] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13864.030607] pending: addrconf_verify_work [13864.031880] workqueue knbd0-recv: flags=0x1a [13864.033039] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13864.034719] in-flight: 657:recv_work [13864.035880] workqueue ptlrpc_pinger: flags=0x2 [13864.037114] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13864.038621] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [13864.040942] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11905s workers=4 idle: 20831 12642 28859 [13864.044082] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13854s workers=2 idle: 659 [13872.061214] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [13872.079950] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13872.151508] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13872.187060] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13872.214136] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13872.227499] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13872.272517] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13872.285398] RAX: 0000000000007d3f RBX: ffff96848811cae0 RCX: 0000000000000001 [13872.300753] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [13872.332332] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [13872.344853] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [13872.368742] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [13872.388919] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [13872.414832] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13872.421516] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [13872.430236] Call Trace: [13872.435644] [13872.437848] ? show_regs.cold.9+0x22/0x2f [13872.451533] ? watchdog_timer_fn.cold.6+0x62/0x10a [13872.460629] ? __hrtimer_run_queues+0x1c2/0x4d0 [13872.473843] ? watchdog+0x40/0x40 [13872.476683] ? hrtimer_interrupt+0x128/0x2c0 [13872.481550] ? smp_apic_timer_interrupt+0xa6/0x270 [13872.493429] ? apic_timer_interrupt+0xf/0x20 [13872.502468] [13872.504365] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13872.512944] do_raw_spin_lock+0x177/0x190 [13872.514399] _raw_spin_lock+0x21/0x30 [13872.515632] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13872.526955] ? do_raw_spin_unlock+0x75/0x190 [13872.543225] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13872.558351] ? do_wait_intr+0xf0/0xf0 [13872.562560] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13872.570146] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [13872.574491] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13872.586550] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13872.594543] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13872.606859] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [13872.616252] mdc_intent_lock+0x353/0x7e0 [mdc] [13872.630644] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13872.639391] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13872.655205] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13872.660172] lmv_intent_open+0x338/0x1150 [lmv] [13872.662776] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13872.671434] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13872.673374] lmv_intent_lock+0x1dd/0x4c0 [lmv] [13872.685655] ll_intent_lock+0xc5/0xaa0 [lustre] [13872.690410] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13872.694478] ll_lookup_it+0xe0e/0x2270 [lustre] [13872.697246] ? do_raw_spin_unlock+0x75/0x190 [13872.700300] ? _raw_spin_unlock+0x12/0x30 [13872.702384] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [13872.705300] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [13872.708182] ll_atomic_open+0x36a/0xd70 [lustre] [13872.711049] lookup_open+0x356/0xb40 [13872.713123] do_last+0x313/0xfc0 [13872.715032] ? nd_jump_root+0xe5/0x160 [13872.717342] ? path_init+0x437/0x520 [13872.720606] path_openat+0xf7/0x500 [13872.723704] ? do_raw_spin_unlock+0x75/0x190 [13872.728918] do_filp_open+0x99/0x140 [13872.734245] ? getname_flags+0x6e/0x330 [13872.736336] ? __check_object_size+0xff/0x256 [13872.739305] ? do_raw_spin_unlock+0x75/0x190 [13872.749015] ? _raw_spin_unlock+0x12/0x30 [13872.751423] do_sys_openat2+0x2b4/0x410 [13872.752721] do_sys_open+0x73/0xa0 [13872.756272] __x64_sys_openat+0x24/0x30 [13872.760053] do_syscall_64+0xc1/0x440 [13872.764391] entry_SYSCALL_64_after_hwframe+0x49/0xae [13872.770911] RIP: 0033:0x7ff54af45332 [13872.779829] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [13872.790927] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [13872.794567] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [13872.801226] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [13872.807480] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [13872.810792] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [13872.822311] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [13876.058030] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [13876.062186] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13876.081422] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13876.086226] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13876.089877] RIP: 0010:native_safe_halt+0x2e/0x50 [13876.091533] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13876.099502] RSP: 0018:ffffaf9fc1543ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13876.101947] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [13876.104365] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [13876.107592] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [13876.110777] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [13876.113644] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [13876.116598] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [13876.120109] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13876.122498] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [13876.125365] Call Trace: [13876.126350] [13876.127037] ? show_regs.cold.9+0x22/0x2f [13876.128507] ? watchdog_timer_fn.cold.6+0x62/0x10a [13876.130079] ? __hrtimer_run_queues+0x1c2/0x4d0 [13876.132137] ? watchdog+0x40/0x40 [13876.137596] ? hrtimer_interrupt+0x128/0x2c0 [13876.139706] ? smp_apic_timer_interrupt+0xa6/0x270 [13876.142836] ? apic_timer_interrupt+0xf/0x20 [13876.145159] [13876.146351] ? native_safe_halt+0x2e/0x50 [13876.148293] kvm_wait+0xcd/0xe0 [13876.150011] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13876.152562] do_raw_spin_lock+0x177/0x190 [13876.154434] _raw_spin_lock+0x21/0x30 [13876.156162] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13876.159187] ? do_raw_spin_unlock+0x75/0x190 [13876.161176] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13876.163637] ? do_wait_intr+0xf0/0xf0 [13876.165447] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13876.168563] mdc_close+0x53b/0xd70 [mdc] [13876.176357] lmv_close+0x18d/0x490 [lmv] [13876.179645] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [13876.185311] ll_close_inode_openhandle+0x493/0xea0 [lustre] [13876.188145] ll_md_real_close+0xa5/0x2a0 [lustre] [13876.191418] ll_md_close+0x274/0x800 [lustre] [13876.198447] ll_file_release+0x128/0xc50 [lustre] [13876.201629] __fput+0x10d/0x440 [13876.204007] ____fput+0x12/0x20 [13876.205907] task_work_run+0xc6/0x110 [13876.208538] exit_to_usermode_loop+0x1dd/0x1f0 [13876.210961] do_syscall_64+0x3d6/0x440 [13876.212756] entry_SYSCALL_64_after_hwframe+0x49/0xae [13876.215446] RIP: 0033:0x7ff54af45cd8 [13876.217484] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [13876.227446] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [13876.231381] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [13876.234429] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [13876.238042] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [13876.240967] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [13876.245249] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13880.061777] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [13880.066927] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13880.096851] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13880.105572] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13880.110416] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13880.114712] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13880.118144] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13880.129563] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13880.134052] RAX: 0000000000005e50 RBX: ffff96848811cae0 RCX: 0000000000000001 [13880.138124] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [13880.142056] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [13880.146083] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [13880.151150] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [13880.158158] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [13880.160574] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13880.165798] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [13880.171757] Call Trace: [13880.175880] [13880.177301] ? show_regs.cold.9+0x22/0x2f [13880.180149] ? watchdog_timer_fn.cold.6+0x62/0x10a [13880.183712] ? __hrtimer_run_queues+0x1c2/0x4d0 [13880.187005] ? watchdog+0x40/0x40 [13880.188982] ? hrtimer_interrupt+0x128/0x2c0 [13880.193329] ? smp_apic_timer_interrupt+0xa6/0x270 [13880.197182] ? apic_timer_interrupt+0xf/0x20 [13880.200549] [13880.202739] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13880.207266] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13880.210491] do_raw_spin_lock+0x177/0x190 [13880.214141] _raw_spin_lock+0x21/0x30 [13880.216508] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13880.220085] process_one_work+0x2c8/0x700 [13880.222593] worker_thread+0x296/0x6e0 [13880.225583] ? rescuer_thread+0x570/0x570 [13880.228654] kthread+0x1d1/0x200 [13880.231001] ? set_kthread_struct+0x70/0x70 [13880.235541] ret_from_fork+0x1f/0x30 [13888.056272] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [13888.061761] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13888.098087] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13888.103523] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13888.107845] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [13888.112334] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [13888.126924] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13888.130670] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [13888.135775] RDX: 00000000000039a4 RSI: 0000000000000003 RDI: ffff96848811cae0 [13888.138377] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [13888.140843] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [13888.143102] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13888.145917] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [13888.148776] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13888.150979] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [13888.153652] Call Trace: [13888.154448] [13888.155059] ? show_regs.cold.9+0x22/0x2f [13888.156194] ? watchdog_timer_fn.cold.6+0x62/0x10a [13888.157653] ? __hrtimer_run_queues+0x1c2/0x4d0 [13888.159108] ? watchdog+0x40/0x40 [13888.160054] ? hrtimer_interrupt+0x128/0x2c0 [13888.161723] ? smp_apic_timer_interrupt+0xa6/0x270 [13888.163199] ? apic_timer_interrupt+0xf/0x20 [13888.164799] [13888.165465] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [13888.167153] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13888.169071] do_raw_spin_lock+0x177/0x190 [13888.171073] _raw_spin_lock+0x21/0x30 [13888.172827] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13888.175083] obd_mod_free+0x46/0x140 [ptlrpc] [13888.178350] mdc_commit_open+0x91/0xb0 [mdc] [13888.180613] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13888.185280] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13888.188677] after_reply+0xe2c/0x1780 [ptlrpc] [13888.191103] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13888.193860] ptlrpcd+0xa82/0xe40 [ptlrpc] [13888.196474] ? do_wait_intr+0xf0/0xf0 [13888.197886] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13888.201027] kthread+0x1d1/0x200 [13888.202463] ? set_kthread_struct+0x70/0x70 [13888.204890] ret_from_fork+0x1f/0x30 [13894.114266] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11935s! [13894.118434] Showing busy workqueues and worker pools: [13894.123375] workqueue events: flags=0x0 [13894.125354] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13894.133011] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [13894.146907] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [13894.150519] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [13894.155702] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13894.159779] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [13894.163919] workqueue events_long: flags=0x0 [13894.166463] rcu: INFO: rcu_sched self-detected stall on CPU [13894.166477] rcu: 0-...!: (6342109 ticks this GP) idle=98a/1/0x4000000000000004 softirq=125154/125154 fqs=959 [13894.166546] (t=11940326 jiffies g=253585 q=6778) [13894.166550] rcu: rcu_sched kthread starved for 11935286 jiffies! g253585 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [13894.166555] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [13894.166556] rcu: RCU grace-period kthread stack dump: [13894.166559] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [13894.166690] Call Trace: [13894.166700] __schedule+0x351/0xcb0 [13894.166818] schedule+0xc0/0x180 [13894.166825] schedule_timeout+0xb4/0x190 [13894.166830] ? do_raw_spin_unlock+0x75/0x190 [13894.166838] ? __next_timer_interrupt+0x160/0x160 [13894.166848] ? prepare_to_swait_event+0xf3/0x140 [13894.166855] rcu_gp_kthread+0x8dd/0x15b0 [13894.166861] ? force_qs_rnp+0x2c0/0x2c0 [13894.166864] kthread+0x1d1/0x200 [13894.166869] ? set_kthread_struct+0x70/0x70 [13894.166873] ret_from_fork+0x1f/0x30 [13894.166881] rcu: Stack dump where RCU GP kthread last ran: [13894.166884] Sending NMI from CPU 0 to CPUs 1: [13894.166955] NMI backtrace for cpu 1 skipped: idling at native_safe_halt+0x2e/0x50 [13894.167913] NMI backtrace for cpu 0 [13894.167923] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13894.167928] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13894.167931] Call Trace: [13894.167936] [13894.167939] ? dump_stack+0xbb/0x10e [13894.167947] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [13894.167955] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [13894.167961] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [13894.167965] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [13894.167970] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [13894.167974] ? rcu_dump_cpu_stacks+0x11b/0x171 [13894.167980] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [13894.167985] ? update_process_times+0x6d/0xc0 [13894.167991] ? tick_sched_handle.isra.18+0x27/0x90 [13894.168130] ? tick_sched_timer+0x53/0xb0 [13894.168135] ? __hrtimer_run_queues+0x1c2/0x4d0 [13894.168138] ? tick_sched_handle.isra.18+0x90/0x90 [13894.168143] ? hrtimer_interrupt+0x128/0x2c0 [13894.168148] ? smp_apic_timer_interrupt+0xa6/0x270 [13894.168155] ? apic_timer_interrupt+0xf/0x20 [13894.168160] ? apic_timer_interrupt+0xa/0x20 [13894.168166] ? console_unlock+0x62d/0x940 [13894.168173] ? vprintk_emit+0x274/0x540 [13894.168177] ? apic_timer_interrupt+0xa/0x20 [13894.168181] ? vprintk_default+0x21/0x30 [13894.168185] ? vprintk+0x5b/0x1d0 [13894.168189] ? printk+0x5f/0x7d [13894.168194] ? show_workqueue_state.cold.41+0xec/0x1ee [13894.168201] ? wq_watchdog_timer_fn+0x17c/0x1a0 [13894.168205] ? show_workqueue_state+0x1d0/0x1d0 [13894.168209] ? call_timer_fn+0x53/0x270 [13894.168214] ? run_timer_softirq+0x6da/0x830 [13894.168218] ? show_workqueue_state+0x1d0/0x1d0 [13894.168222] ? kvm_clock_get_cycles+0x2c/0x50 [13894.168230] ? kvm_sched_clock_read+0x2c/0x50 [13894.168235] ? sched_clock+0xd/0x20 [13894.168240] ? __do_softirq+0x134/0x4e8 [13894.168245] ? irq_exit_rcu+0x1c6/0x1e0 [13894.168249] ? irq_exit+0x12/0x30 [13894.168252] ? smp_apic_timer_interrupt+0xe0/0x270 [13894.168257] ? apic_timer_interrupt+0xf/0x20 [13894.168261] [13894.168264] ? native_safe_halt+0x2e/0x50 [13894.168268] ? kvm_wait+0xcd/0xe0 [13894.168271] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13894.168276] ? do_raw_spin_lock+0x177/0x190 [13894.168280] ? _raw_spin_lock+0x21/0x30 [13894.168286] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13894.168938] ? obd_mod_free+0x46/0x140 [ptlrpc] [13894.169169] ? mdc_commit_open+0x91/0xb0 [mdc] [13894.169227] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13894.169437] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13894.169814] ? after_reply+0xe2c/0x1780 [ptlrpc] [13894.170007] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13894.170208] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [13894.170601] ? do_wait_intr+0xf0/0xf0 [13894.170615] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13894.170969] ? kthread+0x1d1/0x200 [13894.170976] ? set_kthread_struct+0x70/0x70 [13894.170980] ? ret_from_fork+0x1f/0x30 [13894.380213] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13894.384381] pending: lu_objects_flush [obdclass] [13894.386945] workqueue events_unbound: flags=0x2 [13894.389183] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [13894.391860] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [13894.395172] workqueue events_power_efficient: flags=0x80 [13894.397266] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13894.400700] pending: neigh_periodic_work [13894.408469] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13894.413234] pending: do_cache_clean, check_lifetime [13894.419429] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13894.427205] pending: neigh_periodic_work [13894.429893] workqueue writeback: flags=0x4a [13894.437845] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [13894.441410] pending: wb_workfn, wb_workfn [13894.445012] workqueue xprtiod: flags=0xa [13894.447869] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [13894.454249] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [13894.458804] workqueue ipv6_addrconf: flags=0x40008 [13894.461453] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13894.464818] pending: addrconf_verify_work [13894.467195] workqueue knbd0-recv: flags=0x1a [13894.471648] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13894.475390] in-flight: 657:recv_work [13894.480436] workqueue ptlrpc_pinger: flags=0x2 [13894.485624] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13894.490247] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [13894.494238] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11935s workers=4 idle: 20831 12642 28859 [13894.499846] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13884s workers=2 idle: 659 [13900.060425] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [13900.065194] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13900.085382] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13900.090093] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13900.094014] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13900.097017] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13900.105991] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13900.109334] RAX: 0000000000002271 RBX: ffff96848811cae0 RCX: 0000000000000001 [13900.112937] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [13900.115289] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [13900.118957] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [13900.122641] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [13900.126224] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [13900.128616] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13900.130363] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [13900.132206] Call Trace: [13900.132999] [13900.133413] ? show_regs.cold.9+0x22/0x2f [13900.134559] ? watchdog_timer_fn.cold.6+0x62/0x10a [13900.137138] ? __hrtimer_run_queues+0x1c2/0x4d0 [13900.139342] ? watchdog+0x40/0x40 [13900.141637] ? hrtimer_interrupt+0x128/0x2c0 [13900.143416] ? smp_apic_timer_interrupt+0xa6/0x270 [13900.145690] ? apic_timer_interrupt+0xf/0x20 [13900.147543] [13900.148364] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13900.151284] do_raw_spin_lock+0x177/0x190 [13900.153336] _raw_spin_lock+0x21/0x30 [13900.155103] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13900.157955] ? do_raw_spin_unlock+0x75/0x190 [13900.160000] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13900.161905] ? do_wait_intr+0xf0/0xf0 [13900.163062] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13900.165052] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [13900.166984] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13900.169432] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13900.171375] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13900.173513] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [13900.175348] mdc_intent_lock+0x353/0x7e0 [mdc] [13900.176924] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13900.178955] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13900.181161] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13900.183211] lmv_intent_open+0x338/0x1150 [lmv] [13900.185171] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13900.187974] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13900.191060] lmv_intent_lock+0x1dd/0x4c0 [lmv] [13900.193075] ll_intent_lock+0xc5/0xaa0 [lustre] [13900.195164] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13900.197848] ll_lookup_it+0xe0e/0x2270 [lustre] [13900.200413] ? do_raw_spin_unlock+0x75/0x190 [13900.202941] ? _raw_spin_unlock+0x12/0x30 [13900.204831] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [13900.207138] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [13900.210016] ll_atomic_open+0x36a/0xd70 [lustre] [13900.212721] lookup_open+0x356/0xb40 [13900.214700] do_last+0x313/0xfc0 [13900.216732] ? nd_jump_root+0xe5/0x160 [13900.218570] ? path_init+0x437/0x520 [13900.220340] path_openat+0xf7/0x500 [13900.222106] ? do_raw_spin_unlock+0x75/0x190 [13900.224147] do_filp_open+0x99/0x140 [13900.225899] ? getname_flags+0x6e/0x330 [13900.227477] ? __check_object_size+0xff/0x256 [13900.229579] ? do_raw_spin_unlock+0x75/0x190 [13900.231481] ? _raw_spin_unlock+0x12/0x30 [13900.233396] do_sys_openat2+0x2b4/0x410 [13900.235520] do_sys_open+0x73/0xa0 [13900.237340] __x64_sys_openat+0x24/0x30 [13900.239442] do_syscall_64+0xc1/0x440 [13900.241407] entry_SYSCALL_64_after_hwframe+0x49/0xae [13900.244516] RIP: 0033:0x7ff54af45332 [13900.246273] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [13900.255170] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [13900.258908] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [13900.262680] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [13900.266223] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [13900.269924] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [13900.273078] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [13904.061068] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [13904.067593] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13904.110626] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13904.117723] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13904.123429] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13904.127741] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13904.140269] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13904.148670] RAX: 00000000000044e4 RBX: ffff96848811cae0 RCX: 0000000000000001 [13904.156758] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [13904.161425] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [13904.163990] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [13904.171012] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [13904.174939] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [13904.180135] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13904.184356] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [13904.191918] Call Trace: [13904.193190] [13904.194290] ? show_regs.cold.9+0x22/0x2f [13904.199931] ? watchdog_timer_fn.cold.6+0x62/0x10a [13904.203008] ? __hrtimer_run_queues+0x1c2/0x4d0 [13904.205696] ? watchdog+0x40/0x40 [13904.207680] ? hrtimer_interrupt+0x128/0x2c0 [13904.211162] ? smp_apic_timer_interrupt+0xa6/0x270 [13904.213750] ? apic_timer_interrupt+0xf/0x20 [13904.215781] [13904.217197] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13904.220326] do_raw_spin_lock+0x177/0x190 [13904.222626] _raw_spin_lock+0x21/0x30 [13904.224531] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13904.227442] ? do_raw_spin_unlock+0x75/0x190 [13904.229707] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13904.232443] ? do_wait_intr+0xf0/0xf0 [13904.233998] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13904.236785] mdc_close+0x53b/0xd70 [mdc] [13904.238198] lmv_close+0x18d/0x490 [lmv] [13904.243310] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [13904.249412] ll_close_inode_openhandle+0x493/0xea0 [lustre] [13904.253545] ll_md_real_close+0xa5/0x2a0 [lustre] [13904.261713] ll_md_close+0x274/0x800 [lustre] [13904.268836] ll_file_release+0x128/0xc50 [lustre] [13904.274324] __fput+0x10d/0x440 [13904.277775] ____fput+0x12/0x20 [13904.285665] task_work_run+0xc6/0x110 [13904.294442] exit_to_usermode_loop+0x1dd/0x1f0 [13904.300925] do_syscall_64+0x3d6/0x440 [13904.307882] entry_SYSCALL_64_after_hwframe+0x49/0xae [13904.313208] RIP: 0033:0x7ff54af45cd8 [13904.318751] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [13904.328609] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [13904.333189] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [13904.337246] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [13904.342449] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [13904.345577] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [13904.352734] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13908.061376] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [13908.067303] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13908.099318] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13908.107630] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13908.117336] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13908.125279] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13908.132873] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13908.154172] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13908.160091] RAX: 0000000000006711 RBX: ffff96848811cae0 RCX: 0000000000000001 [13908.168186] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [13908.172033] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [13908.176454] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [13908.181982] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [13908.185421] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [13908.190234] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13908.194323] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [13908.199255] Call Trace: [13908.203032] [13908.204171] ? show_regs.cold.9+0x22/0x2f [13908.206630] ? watchdog_timer_fn.cold.6+0x62/0x10a [13908.209435] ? __hrtimer_run_queues+0x1c2/0x4d0 [13908.213169] ? watchdog+0x40/0x40 [13908.215191] ? hrtimer_interrupt+0x128/0x2c0 [13908.219322] ? smp_apic_timer_interrupt+0xa6/0x270 [13908.225107] ? apic_timer_interrupt+0xf/0x20 [13908.229587] [13908.233150] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13908.235988] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13908.239235] do_raw_spin_lock+0x177/0x190 [13908.243029] _raw_spin_lock+0x21/0x30 [13908.244749] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13908.246958] process_one_work+0x2c8/0x700 [13908.255032] worker_thread+0x296/0x6e0 [13908.260196] ? rescuer_thread+0x570/0x570 [13908.263131] kthread+0x1d1/0x200 [13908.264611] ? set_kthread_struct+0x70/0x70 [13908.266879] ret_from_fork+0x1f/0x30 [13920.060306] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [13920.066120] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13920.146174] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13920.165881] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13920.176915] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [13920.186521] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [13920.215507] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13920.223783] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [13920.233390] RDX: 00000000000000ab RSI: 0000000000000003 RDI: ffff96848811cae0 [13920.239694] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [13920.248504] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [13920.258020] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13920.262952] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [13920.271165] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13920.283881] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [13920.306053] Call Trace: [13920.313423] [13920.319027] ? show_regs.cold.9+0x22/0x2f [13920.324578] ? watchdog_timer_fn.cold.6+0x62/0x10a [13920.328395] ? __hrtimer_run_queues+0x1c2/0x4d0 [13920.329995] ? watchdog+0x40/0x40 [13920.334499] ? hrtimer_interrupt+0x128/0x2c0 [13920.338126] ? smp_apic_timer_interrupt+0xa6/0x270 [13920.342522] ? apic_timer_interrupt+0xf/0x20 [13920.347166] [13920.348226] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [13920.353129] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13920.356552] do_raw_spin_lock+0x177/0x190 [13920.359609] _raw_spin_lock+0x21/0x30 [13920.362163] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13920.367736] obd_mod_free+0x46/0x140 [ptlrpc] [13920.372972] mdc_commit_open+0x91/0xb0 [mdc] [13920.383933] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13920.392677] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13920.404059] after_reply+0xe2c/0x1780 [ptlrpc] [13920.407309] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13920.410275] ptlrpcd+0xa82/0xe40 [ptlrpc] [13920.413524] ? do_wait_intr+0xf0/0xf0 [13920.415455] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13920.418253] kthread+0x1d1/0x200 [13920.420395] ? set_kthread_struct+0x70/0x70 [13920.422839] ret_from_fork+0x1f/0x30 [13924.831030] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11965s! [13924.834318] Showing busy workqueues and worker pools: [13924.839108] workqueue events: flags=0x0 [13924.841165] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13924.844196] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [13924.848246] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [13924.851591] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [13924.856632] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13924.859944] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [13924.863145] workqueue events_long: flags=0x0 [13924.865602] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13924.868786] pending: lu_objects_flush [obdclass] [13924.871249] workqueue events_unbound: flags=0x2 [13924.873484] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [13924.877019] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [13924.882471] workqueue events_power_efficient: flags=0x80 [13924.888603] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13924.895053] pending: neigh_periodic_work [13924.899601] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13924.902910] pending: do_cache_clean, check_lifetime [13924.908022] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13924.915142] pending: neigh_periodic_work [13924.921111] workqueue writeback: flags=0x4a [13924.922936] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [13924.929914] pending: wb_workfn, wb_workfn [13924.935036] workqueue xprtiod: flags=0xa [13924.936914] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [13924.942921] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [13924.949822] workqueue ipv6_addrconf: flags=0x40008 [13924.952075] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13924.954978] pending: addrconf_verify_work [13924.956875] workqueue knbd0-recv: flags=0x1a [13924.958835] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13924.961555] in-flight: 657:recv_work [13924.963175] workqueue ptlrpc_pinger: flags=0x2 [13924.965073] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13924.968859] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [13924.973393] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11966s workers=4 idle: 20831 12642 28859 [13924.987065] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13915s workers=2 idle: 659 [13928.064595] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [13928.075759] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13928.117643] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13928.129645] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13928.141400] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13928.149676] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13928.188692] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13928.201940] RAX: 0000000000007186 RBX: ffff96848811cae0 RCX: 0000000000000001 [13928.212639] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [13928.217713] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [13928.222594] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [13928.227695] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [13928.232040] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [13928.237105] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13928.240357] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [13928.246118] Call Trace: [13928.247807] [13928.249393] ? show_regs.cold.9+0x22/0x2f [13928.251741] ? watchdog_timer_fn.cold.6+0x62/0x10a [13928.254127] ? __hrtimer_run_queues+0x1c2/0x4d0 [13928.256345] ? watchdog+0x40/0x40 [13928.257993] ? hrtimer_interrupt+0x128/0x2c0 [13928.260193] ? smp_apic_timer_interrupt+0xa6/0x270 [13928.263063] ? apic_timer_interrupt+0xf/0x20 [13928.264998] [13928.266181] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13928.268742] do_raw_spin_lock+0x177/0x190 [13928.270907] _raw_spin_lock+0x21/0x30 [13928.273081] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13928.275834] ? do_raw_spin_unlock+0x75/0x190 [13928.278076] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13928.281736] ? do_wait_intr+0xf0/0xf0 [13928.283729] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13928.286930] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [13928.290019] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13928.293136] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13928.301661] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13928.307077] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [13928.309754] mdc_intent_lock+0x353/0x7e0 [mdc] [13928.316254] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13928.323132] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13928.327300] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13928.332075] lmv_intent_open+0x338/0x1150 [lmv] [13928.335216] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13928.339376] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13928.343302] lmv_intent_lock+0x1dd/0x4c0 [lmv] [13928.349576] ll_intent_lock+0xc5/0xaa0 [lustre] [13928.353603] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13928.356048] ll_lookup_it+0xe0e/0x2270 [lustre] [13928.357478] ? do_raw_spin_unlock+0x75/0x190 [13928.359045] ? _raw_spin_unlock+0x12/0x30 [13928.360350] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [13928.364950] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [13928.369076] ll_atomic_open+0x36a/0xd70 [lustre] [13928.375567] lookup_open+0x356/0xb40 [13928.380617] do_last+0x313/0xfc0 [13928.382746] ? nd_jump_root+0xe5/0x160 [13928.385792] ? path_init+0x437/0x520 [13928.390681] path_openat+0xf7/0x500 [13928.392662] ? do_raw_spin_unlock+0x75/0x190 [13928.395053] do_filp_open+0x99/0x140 [13928.396843] ? getname_flags+0x6e/0x330 [13928.399710] ? __check_object_size+0xff/0x256 [13928.402632] ? do_raw_spin_unlock+0x75/0x190 [13928.406611] ? _raw_spin_unlock+0x12/0x30 [13928.407981] do_sys_openat2+0x2b4/0x410 [13928.416890] do_sys_open+0x73/0xa0 [13928.418686] __x64_sys_openat+0x24/0x30 [13928.420503] do_syscall_64+0xc1/0x440 [13928.421610] entry_SYSCALL_64_after_hwframe+0x49/0xae [13928.424919] RIP: 0033:0x7ff54af45332 [13928.430585] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [13928.455028] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [13928.458757] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [13928.462210] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [13928.465625] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [13928.468930] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [13928.473707] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [13932.066449] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [13932.076978] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13932.133514] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13932.148093] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13932.157769] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13932.159746] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13932.174952] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13932.183766] RAX: 0000000000006b1d RBX: ffff96848811cae0 RCX: 0000000000000001 [13932.202408] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [13932.204926] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [13932.222452] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [13932.230060] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [13932.236009] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [13932.242847] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13932.248910] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [13932.258179] Call Trace: [13932.259136] [13932.259882] ? show_regs.cold.9+0x22/0x2f [13932.261025] ? watchdog_timer_fn.cold.6+0x62/0x10a [13932.266690] ? __hrtimer_run_queues+0x1c2/0x4d0 [13932.269592] ? watchdog+0x40/0x40 [13932.274651] ? hrtimer_interrupt+0x128/0x2c0 [13932.276710] ? smp_apic_timer_interrupt+0xa6/0x270 [13932.278437] ? apic_timer_interrupt+0xf/0x20 [13932.280013] [13932.280622] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13932.282840] do_raw_spin_lock+0x177/0x190 [13932.285738] _raw_spin_lock+0x21/0x30 [13932.289420] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13932.305095] ? do_raw_spin_unlock+0x75/0x190 [13932.307397] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13932.310613] ? do_wait_intr+0xf0/0xf0 [13932.312982] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13932.320442] mdc_close+0x53b/0xd70 [mdc] [13932.322495] lmv_close+0x18d/0x490 [lmv] [13932.325245] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [13932.326926] ll_close_inode_openhandle+0x493/0xea0 [lustre] [13932.329273] ll_md_real_close+0xa5/0x2a0 [lustre] [13932.331478] ll_md_close+0x274/0x800 [lustre] [13932.336244] ll_file_release+0x128/0xc50 [lustre] [13932.338012] __fput+0x10d/0x440 [13932.346056] ____fput+0x12/0x20 [13932.349089] task_work_run+0xc6/0x110 [13932.356941] exit_to_usermode_loop+0x1dd/0x1f0 [13932.359045] do_syscall_64+0x3d6/0x440 [13932.360862] entry_SYSCALL_64_after_hwframe+0x49/0xae [13932.363048] RIP: 0033:0x7ff54af45cd8 [13932.365895] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [13932.400902] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [13932.405816] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [13932.410064] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [13932.412793] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [13932.432092] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [13932.443063] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13936.063234] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [13936.082815] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13936.148369] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13936.173760] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13936.184972] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13936.189747] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13936.201267] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13936.227635] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13936.237631] RAX: 00000000000051aa RBX: ffff96848811cae0 RCX: 0000000000000001 [13936.247816] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [13936.255190] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [13936.263706] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [13936.269233] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [13936.276712] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [13936.284334] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13936.291661] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [13936.296052] Call Trace: [13936.301433] [13936.302383] ? show_regs.cold.9+0x22/0x2f [13936.311790] ? watchdog_timer_fn.cold.6+0x62/0x10a [13936.317530] ? __hrtimer_run_queues+0x1c2/0x4d0 [13936.323923] ? watchdog+0x40/0x40 [13936.325924] ? hrtimer_interrupt+0x128/0x2c0 [13936.331753] ? smp_apic_timer_interrupt+0xa6/0x270 [13936.337556] ? apic_timer_interrupt+0xf/0x20 [13936.342767] [13936.344883] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13936.346948] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13936.354704] do_raw_spin_lock+0x177/0x190 [13936.359007] _raw_spin_lock+0x21/0x30 [13936.362220] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13936.365389] process_one_work+0x2c8/0x700 [13936.369175] worker_thread+0x296/0x6e0 [13936.372067] ? rescuer_thread+0x570/0x570 [13936.378085] kthread+0x1d1/0x200 [13936.379891] ? set_kthread_struct+0x70/0x70 [13936.382411] ret_from_fork+0x1f/0x30 [13952.055053] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [13952.059214] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13952.108758] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13952.117947] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13952.127407] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [13952.133879] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [13952.149038] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13952.156738] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [13952.164149] RDX: 0000000000004f6c RSI: 0000000000000003 RDI: ffff96848811cae0 [13952.168976] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [13952.174018] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [13952.178692] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13952.183164] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [13952.187185] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13952.194740] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [13952.201765] Call Trace: [13952.204702] [13952.207190] ? show_regs.cold.9+0x22/0x2f [13952.212400] ? watchdog_timer_fn.cold.6+0x62/0x10a [13952.215364] ? __hrtimer_run_queues+0x1c2/0x4d0 [13952.218657] ? watchdog+0x40/0x40 [13952.220990] ? hrtimer_interrupt+0x128/0x2c0 [13952.224159] ? smp_apic_timer_interrupt+0xa6/0x270 [13952.230380] ? apic_timer_interrupt+0xf/0x20 [13952.233185] [13952.234006] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [13952.237733] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13952.241557] do_raw_spin_lock+0x177/0x190 [13952.243722] _raw_spin_lock+0x21/0x30 [13952.247180] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13952.256957] obd_mod_free+0x46/0x140 [ptlrpc] [13952.263421] mdc_commit_open+0x91/0xb0 [mdc] [13952.274204] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13952.282908] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13952.296396] after_reply+0xe2c/0x1780 [ptlrpc] [13952.300663] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13952.317549] ptlrpcd+0xa82/0xe40 [ptlrpc] [13952.326847] ? do_wait_intr+0xf0/0xf0 [13952.333198] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13952.335466] kthread+0x1d1/0x200 [13952.345280] ? set_kthread_struct+0x70/0x70 [13952.351078] ret_from_fork+0x1f/0x30 [13955.041162] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11996s! [13955.048615] Showing busy workqueues and worker pools: [13955.051807] workqueue events: flags=0x0 [13955.054039] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13955.063076] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [13955.079137] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [13955.083858] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [13955.098844] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13955.102676] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [13955.109823] workqueue events_long: flags=0x0 [13955.112799] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13955.116340] pending: lu_objects_flush [obdclass] [13955.119322] workqueue events_unbound: flags=0x2 [13955.123153] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [13955.126509] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [13955.144758] workqueue events_power_efficient: flags=0x80 [13955.146862] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13955.152197] pending: neigh_periodic_work [13955.154685] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13955.166755] pending: do_cache_clean, check_lifetime [13955.179469] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13955.185412] pending: neigh_periodic_work [13955.192233] workqueue writeback: flags=0x4a [13955.196839] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [13955.205113] pending: wb_workfn, wb_workfn [13955.213867] workqueue xprtiod: flags=0xa [13955.215693] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [13955.226402] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [13955.233717] workqueue ipv6_addrconf: flags=0x40008 [13955.242189] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13955.254834] pending: addrconf_verify_work [13955.256109] workqueue knbd0-recv: flags=0x1a [13955.262355] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13955.269791] in-flight: 657:recv_work [13955.278013] workqueue ptlrpc_pinger: flags=0x2 [13955.289014] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13955.290756] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [13955.303122] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11996s workers=4 idle: 20831 12642 28859 [13955.316275] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13945s workers=2 idle: 659 [13956.064365] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [13956.070445] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13956.109680] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13956.122726] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13956.133430] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13956.143121] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13956.168175] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13956.177409] RAX: 0000000000006e13 RBX: ffff96848811cae0 RCX: 0000000000000001 [13956.188349] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [13956.202226] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [13956.210068] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [13956.225133] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [13956.234387] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [13956.241240] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13956.247106] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [13956.259702] Call Trace: [13956.262836] [13956.268192] ? show_regs.cold.9+0x22/0x2f [13956.274444] ? watchdog_timer_fn.cold.6+0x62/0x10a [13956.278733] ? __hrtimer_run_queues+0x1c2/0x4d0 [13956.283222] ? watchdog+0x40/0x40 [13956.289028] ? hrtimer_interrupt+0x128/0x2c0 [13956.292374] ? smp_apic_timer_interrupt+0xa6/0x270 [13956.301290] ? apic_timer_interrupt+0xf/0x20 [13956.307455] [13956.311333] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13956.323669] do_raw_spin_lock+0x177/0x190 [13956.325623] _raw_spin_lock+0x21/0x30 [13956.329367] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13956.334758] ? do_raw_spin_unlock+0x75/0x190 [13956.337485] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13956.340759] ? do_wait_intr+0xf0/0xf0 [13956.348596] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13956.355160] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [13956.359074] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13956.372706] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13956.388004] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13956.389990] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [13956.399712] mdc_intent_lock+0x353/0x7e0 [mdc] [13956.405279] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13956.412167] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13956.420971] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13956.424419] lmv_intent_open+0x338/0x1150 [lmv] [13956.428903] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13956.430136] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13956.435287] lmv_intent_lock+0x1dd/0x4c0 [lmv] [13956.445159] ll_intent_lock+0xc5/0xaa0 [lustre] [13956.450778] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13956.458115] ll_lookup_it+0xe0e/0x2270 [lustre] [13956.465835] ? do_raw_spin_unlock+0x75/0x190 [13956.470601] ? _raw_spin_unlock+0x12/0x30 [13956.473722] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [13956.476840] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [13956.479516] ll_atomic_open+0x36a/0xd70 [lustre] [13956.482171] lookup_open+0x356/0xb40 [13956.484025] do_last+0x313/0xfc0 [13956.487320] ? nd_jump_root+0xe5/0x160 [13956.488525] ? path_init+0x437/0x520 [13956.490674] path_openat+0xf7/0x500 [13956.492374] ? do_raw_spin_unlock+0x75/0x190 [13956.494696] do_filp_open+0x99/0x140 [13956.496322] ? getname_flags+0x6e/0x330 [13956.498313] ? __check_object_size+0xff/0x256 [13956.500602] ? do_raw_spin_unlock+0x75/0x190 [13956.502655] ? _raw_spin_unlock+0x12/0x30 [13956.504515] do_sys_openat2+0x2b4/0x410 [13956.506364] do_sys_open+0x73/0xa0 [13956.507492] __x64_sys_openat+0x24/0x30 [13956.508695] do_syscall_64+0xc1/0x440 [13956.510769] entry_SYSCALL_64_after_hwframe+0x49/0xae [13956.523546] RIP: 0033:0x7ff54af45332 [13956.528226] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [13956.540287] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [13956.546202] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [13956.551431] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [13956.556806] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [13956.559774] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [13956.562157] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [13960.060434] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [13960.065765] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13960.112738] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13960.118295] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13960.122734] RIP: 0010:native_safe_halt+0x2e/0x50 [13960.128685] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13960.142155] RSP: 0018:ffffaf9fc1543ae0 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [13960.146238] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [13960.149070] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c20b4154 [13960.152247] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [13960.158317] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [13960.161786] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [13960.164714] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [13960.170945] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13960.177355] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [13960.184273] Call Trace: [13960.185776] [13960.187044] ? show_regs.cold.9+0x22/0x2f [13960.188736] ? watchdog_timer_fn.cold.6+0x62/0x10a [13960.191246] ? __hrtimer_run_queues+0x1c2/0x4d0 [13960.193519] ? watchdog+0x40/0x40 [13960.195447] ? hrtimer_interrupt+0x128/0x2c0 [13960.197534] ? smp_apic_timer_interrupt+0xa6/0x270 [13960.201761] ? apic_timer_interrupt+0xf/0x20 [13960.208166] [13960.209906] ? native_safe_halt+0x2e/0x50 [13960.213398] kvm_wait+0xcd/0xe0 [13960.217469] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13960.223267] do_raw_spin_lock+0x177/0x190 [13960.225362] _raw_spin_lock+0x21/0x30 [13960.227310] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13960.230222] ? do_raw_spin_unlock+0x75/0x190 [13960.232385] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13960.235145] ? do_wait_intr+0xf0/0xf0 [13960.237787] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13960.243319] mdc_close+0x53b/0xd70 [mdc] [13960.244920] lmv_close+0x18d/0x490 [lmv] [13960.247070] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [13960.251423] ll_close_inode_openhandle+0x493/0xea0 [lustre] [13960.254043] ll_md_real_close+0xa5/0x2a0 [lustre] [13960.256341] ll_md_close+0x274/0x800 [lustre] [13960.258965] ll_file_release+0x128/0xc50 [lustre] [13960.260922] __fput+0x10d/0x440 [13960.262193] ____fput+0x12/0x20 [13960.263351] task_work_run+0xc6/0x110 [13960.264906] exit_to_usermode_loop+0x1dd/0x1f0 [13960.269128] do_syscall_64+0x3d6/0x440 [13960.270788] entry_SYSCALL_64_after_hwframe+0x49/0xae [13960.276351] RIP: 0033:0x7ff54af45cd8 [13960.279799] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [13960.297854] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [13960.301674] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [13960.304531] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [13960.310479] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [13960.314079] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [13960.317290] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13964.061868] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [13964.066397] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13964.131546] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13964.153233] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13964.172935] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13964.182296] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13964.188642] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13964.214467] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13964.220458] RAX: 0000000000007084 RBX: ffff96848811cae0 RCX: 0000000000000001 [13964.225095] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [13964.228703] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [13964.232437] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [13964.236433] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [13964.241145] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [13964.245522] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13964.248111] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [13964.252531] Call Trace: [13964.253807] [13964.257225] ? show_regs.cold.9+0x22/0x2f [13964.262028] ? watchdog_timer_fn.cold.6+0x62/0x10a [13964.264660] ? __hrtimer_run_queues+0x1c2/0x4d0 [13964.271258] ? watchdog+0x40/0x40 [13964.274323] ? hrtimer_interrupt+0x128/0x2c0 [13964.277984] ? smp_apic_timer_interrupt+0xa6/0x270 [13964.280246] ? apic_timer_interrupt+0xf/0x20 [13964.282318] [13964.283196] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13964.286770] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13964.290372] do_raw_spin_lock+0x177/0x190 [13964.292498] _raw_spin_lock+0x21/0x30 [13964.298513] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13964.302620] process_one_work+0x2c8/0x700 [13964.309030] worker_thread+0x296/0x6e0 [13964.311963] ? rescuer_thread+0x570/0x570 [13964.314683] kthread+0x1d1/0x200 [13964.319888] ? set_kthread_struct+0x70/0x70 [13964.323101] ret_from_fork+0x1f/0x30 [13980.055080] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [13980.058397] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13980.077123] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13980.081353] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13980.084265] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [13980.085866] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [13980.092628] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13980.095335] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [13980.097988] RDX: 000000000000229f RSI: 0000000000000003 RDI: ffff96848811cae0 [13980.100586] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [13980.103390] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [13980.106197] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13980.108904] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [13980.111522] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13980.113844] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [13980.116452] Call Trace: [13980.117857] [13980.118719] ? show_regs.cold.9+0x22/0x2f [13980.120323] ? watchdog_timer_fn.cold.6+0x62/0x10a [13980.122183] ? __hrtimer_run_queues+0x1c2/0x4d0 [13980.123938] ? watchdog+0x40/0x40 [13980.125389] ? hrtimer_interrupt+0x128/0x2c0 [13980.127168] ? smp_apic_timer_interrupt+0xa6/0x270 [13980.128888] ? apic_timer_interrupt+0xf/0x20 [13980.130682] [13980.131458] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [13980.133697] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13980.140174] do_raw_spin_lock+0x177/0x190 [13980.141965] _raw_spin_lock+0x21/0x30 [13980.143404] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13980.146236] obd_mod_free+0x46/0x140 [ptlrpc] [13980.148460] mdc_commit_open+0x91/0xb0 [mdc] [13980.150335] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13980.153130] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13980.155886] after_reply+0xe2c/0x1780 [ptlrpc] [13980.159086] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13980.162284] ptlrpcd+0xa82/0xe40 [ptlrpc] [13980.164422] ? do_wait_intr+0xf0/0xf0 [13980.166295] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13980.169096] kthread+0x1d1/0x200 [13980.170456] ? set_kthread_struct+0x70/0x70 [13980.172419] ret_from_fork+0x1f/0x30 [13984.064453] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [13984.071453] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13984.129906] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13984.146133] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13984.155152] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13984.169271] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13984.203447] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13984.213491] RAX: 0000000000001323 RBX: ffff96848811cae0 RCX: 0000000000000001 [13984.220201] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [13984.236192] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [13984.244302] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [13984.255523] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [13984.263788] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [13984.271755] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13984.277855] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [13984.286142] Call Trace: [13984.288284] [13984.294358] ? show_regs.cold.9+0x22/0x2f [13984.297230] ? watchdog_timer_fn.cold.6+0x62/0x10a [13984.308119] ? __hrtimer_run_queues+0x1c2/0x4d0 [13984.319201] ? watchdog+0x40/0x40 [13984.323282] ? hrtimer_interrupt+0x128/0x2c0 [13984.329502] ? smp_apic_timer_interrupt+0xa6/0x270 [13984.332327] ? apic_timer_interrupt+0xf/0x20 [13984.338120] [13984.339360] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13984.341313] do_raw_spin_lock+0x177/0x190 [13984.344316] _raw_spin_lock+0x21/0x30 [13984.346598] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13984.348620] ? do_raw_spin_unlock+0x75/0x190 [13984.350197] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13984.353018] ? do_wait_intr+0xf0/0xf0 [13984.354101] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13984.357461] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [13984.359505] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13984.362087] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13984.364557] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13984.366968] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [13984.369958] mdc_intent_lock+0x353/0x7e0 [mdc] [13984.373878] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13984.383820] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13984.385624] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13984.397236] lmv_intent_open+0x338/0x1150 [lmv] [13984.404343] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13984.408507] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13984.415328] lmv_intent_lock+0x1dd/0x4c0 [lmv] [13984.418164] ll_intent_lock+0xc5/0xaa0 [lustre] [13984.424257] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13984.426222] ll_lookup_it+0xe0e/0x2270 [lustre] [13984.442226] ? do_raw_spin_unlock+0x75/0x190 [13984.451510] ? _raw_spin_unlock+0x12/0x30 [13984.455396] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [13984.459204] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [13984.470365] ll_atomic_open+0x36a/0xd70 [lustre] [13984.472093] lookup_open+0x356/0xb40 [13984.473032] do_last+0x313/0xfc0 [13984.481680] ? nd_jump_root+0xe5/0x160 [13984.486200] ? path_init+0x437/0x520 [13984.492544] path_openat+0xf7/0x500 [13984.497478] ? do_raw_spin_unlock+0x75/0x190 [13984.504811] do_filp_open+0x99/0x140 [13984.512277] ? getname_flags+0x6e/0x330 [13984.517865] ? __check_object_size+0xff/0x256 [13984.519497] ? do_raw_spin_unlock+0x75/0x190 [13984.531441] ? _raw_spin_unlock+0x12/0x30 [13984.536395] do_sys_openat2+0x2b4/0x410 [13984.538080] do_sys_open+0x73/0xa0 [13984.542492] __x64_sys_openat+0x24/0x30 [13984.545753] do_syscall_64+0xc1/0x440 [13984.548323] entry_SYSCALL_64_after_hwframe+0x49/0xae [13984.558112] RIP: 0033:0x7ff54af45332 [13984.562468] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [13984.581802] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [13984.586632] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [13984.593366] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [13984.606493] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [13984.611105] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [13984.613681] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [13985.759618] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12026s! [13985.762739] Showing busy workqueues and worker pools: [13985.764396] workqueue events: flags=0x0 [13985.766226] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13985.769874] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [13985.773828] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [13985.777503] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [13985.783991] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13985.786959] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [13985.789994] workqueue events_long: flags=0x0 [13985.792064] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13985.796775] pending: lu_objects_flush [obdclass] [13985.810672] workqueue events_unbound: flags=0x2 [13985.815926] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [13985.819144] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [13985.826049] workqueue events_power_efficient: flags=0x80 [13985.829085] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13985.833557] pending: neigh_periodic_work [13985.838151] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13985.843704] pending: do_cache_clean, check_lifetime [13985.846121] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13985.849981] pending: neigh_periodic_work [13985.851987] workqueue writeback: flags=0x4a [13985.856280] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [13985.860079] pending: wb_workfn, wb_workfn [13985.862581] workqueue xprtiod: flags=0xa [13985.864263] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [13985.867299] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [13985.871578] workqueue ipv6_addrconf: flags=0x40008 [13985.874124] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13985.877371] pending: addrconf_verify_work [13985.880166] workqueue knbd0-recv: flags=0x1a [13985.882263] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13985.885098] in-flight: 657:recv_work [13985.887248] workqueue ptlrpc_pinger: flags=0x2 [13985.889107] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13985.891953] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [13985.895283] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12027s workers=4 idle: 20831 12642 28859 [13985.899729] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13976s workers=2 idle: 659 [13988.058051] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [13988.061983] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13988.083434] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13988.089208] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13988.094680] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13988.097649] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13988.111077] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13988.114632] RAX: 0000000000006107 RBX: ffff96848811cae0 RCX: 0000000000000001 [13988.118113] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [13988.121640] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [13988.126937] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [13988.130812] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [13988.134875] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [13988.139929] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13988.143008] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [13988.147161] Call Trace: [13988.148458] [13988.149663] ? show_regs.cold.9+0x22/0x2f [13988.151929] ? watchdog_timer_fn.cold.6+0x62/0x10a [13988.154843] ? __hrtimer_run_queues+0x1c2/0x4d0 [13988.157046] ? watchdog+0x40/0x40 [13988.158918] ? hrtimer_interrupt+0x128/0x2c0 [13988.164675] ? smp_apic_timer_interrupt+0xa6/0x270 [13988.173114] ? apic_timer_interrupt+0xf/0x20 [13988.179880] [13988.180756] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13988.192020] do_raw_spin_lock+0x177/0x190 [13988.194563] _raw_spin_lock+0x21/0x30 [13988.200155] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13988.207874] ? do_raw_spin_unlock+0x75/0x190 [13988.212690] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13988.214672] ? do_wait_intr+0xf0/0xf0 [13988.222698] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13988.226690] mdc_close+0x53b/0xd70 [mdc] [13988.229985] lmv_close+0x18d/0x490 [lmv] [13988.237240] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [13988.239365] ll_close_inode_openhandle+0x493/0xea0 [lustre] [13988.251465] ll_md_real_close+0xa5/0x2a0 [lustre] [13988.261550] ll_md_close+0x274/0x800 [lustre] [13988.264489] ll_file_release+0x128/0xc50 [lustre] [13988.267709] __fput+0x10d/0x440 [13988.272160] ____fput+0x12/0x20 [13988.275561] task_work_run+0xc6/0x110 [13988.282286] exit_to_usermode_loop+0x1dd/0x1f0 [13988.286381] do_syscall_64+0x3d6/0x440 [13988.293942] entry_SYSCALL_64_after_hwframe+0x49/0xae [13988.299757] RIP: 0033:0x7ff54af45cd8 [13988.301426] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [13988.324246] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [13988.329073] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [13988.338265] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [13988.349104] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [13988.363096] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [13988.370815] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13992.059084] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [13992.062337] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13992.082508] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13992.087651] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13992.091393] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13992.094495] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13992.097283] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13992.105942] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13992.109454] RAX: 0000000000005991 RBX: ffff96848811cae0 RCX: 0000000000000001 [13992.112712] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [13992.115920] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [13992.119316] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [13992.122564] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [13992.125753] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [13992.129391] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13992.132038] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [13992.134940] Call Trace: [13992.136025] [13992.136889] ? show_regs.cold.9+0x22/0x2f [13992.139049] ? watchdog_timer_fn.cold.6+0x62/0x10a [13992.141464] ? __hrtimer_run_queues+0x1c2/0x4d0 [13992.143498] ? watchdog+0x40/0x40 [13992.144979] ? hrtimer_interrupt+0x128/0x2c0 [13992.146974] ? smp_apic_timer_interrupt+0xa6/0x270 [13992.149091] ? apic_timer_interrupt+0xf/0x20 [13992.150917] [13992.151886] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13992.154436] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13992.156754] do_raw_spin_lock+0x177/0x190 [13992.159285] _raw_spin_lock+0x21/0x30 [13992.160995] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13992.164533] process_one_work+0x2c8/0x700 [13992.167080] worker_thread+0x296/0x6e0 [13992.169285] ? rescuer_thread+0x570/0x570 [13992.171792] kthread+0x1d1/0x200 [13992.173569] ? set_kthread_struct+0x70/0x70 [13992.175343] ret_from_fork+0x1f/0x30 [14012.055295] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [14012.059222] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14012.061263] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [14012.084441] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14012.084460] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14012.084463] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [14012.084483] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [14012.084488] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14012.084494] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [14012.084497] RDX: 000000000000170d RSI: 0000000000000003 RDI: ffff96848811cae0 [14012.084499] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [14012.084502] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [14012.084504] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14012.084518] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [14012.084521] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14012.084524] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [14012.084532] Call Trace: [14012.084539] [14012.084549] ? show_regs.cold.9+0x22/0x2f [14012.084561] ? watchdog_timer_fn.cold.6+0x62/0x10a [14012.084568] ? __hrtimer_run_queues+0x1c2/0x4d0 [14012.084573] ? watchdog+0x40/0x40 [14012.084577] ? hrtimer_interrupt+0x128/0x2c0 [14012.084580] ? smp_apic_timer_interrupt+0xa6/0x270 [14012.084590] ? apic_timer_interrupt+0xf/0x20 [14012.084595] [14012.084597] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [14012.084602] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14012.084607] do_raw_spin_lock+0x177/0x190 [14012.084611] _raw_spin_lock+0x21/0x30 [14012.084620] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14012.084969] obd_mod_free+0x46/0x140 [ptlrpc] [14012.085165] mdc_commit_open+0x91/0xb0 [mdc] [14012.085207] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14012.091210] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14012.091290] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14012.091297] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14012.091300] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14012.091316] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14012.091320] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14012.091324] RAX: 0000000000007f86 RBX: ffff96848811cae0 RCX: 0000000000000001 [14012.091327] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [14012.091329] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [14012.091331] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [14012.091333] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [14012.091336] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [14012.091339] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14012.091341] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [14012.091349] Call Trace: [14012.091355] [14012.091360] ? show_regs.cold.9+0x22/0x2f [14012.091369] ? watchdog_timer_fn.cold.6+0x62/0x10a [14012.091375] ? __hrtimer_run_queues+0x1c2/0x4d0 [14012.091379] ? watchdog+0x40/0x40 [14012.091383] ? hrtimer_interrupt+0x128/0x2c0 [14012.091387] ? smp_apic_timer_interrupt+0xa6/0x270 [14012.091397] ? apic_timer_interrupt+0xf/0x20 [14012.091402] [14012.091405] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14012.091413] do_raw_spin_lock+0x177/0x190 [14012.096657] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14012.102271] _raw_spin_lock+0x21/0x30 [14012.102294] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14012.105432] after_reply+0xe2c/0x1780 [ptlrpc] [14012.121092] ? do_raw_spin_unlock+0x75/0x190 [14012.125837] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14012.126484] ptlrpcd+0xa82/0xe40 [ptlrpc] [14012.126872] ? do_wait_intr+0xf0/0xf0 [14012.126887] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14012.127123] kthread+0x1d1/0x200 [14012.127131] ? set_kthread_struct+0x70/0x70 [14012.127135] ret_from_fork+0x1f/0x30 [14012.346178] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14012.348319] ? do_wait_intr+0xf0/0xf0 [14012.353954] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14012.360604] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [14012.364075] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14012.371764] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14012.375804] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14012.382304] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [14012.388158] mdc_intent_lock+0x353/0x7e0 [mdc] [14012.391159] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14012.394188] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14012.398629] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14012.402213] lmv_intent_open+0x338/0x1150 [lmv] [14012.410473] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14012.419175] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14012.421750] lmv_intent_lock+0x1dd/0x4c0 [lmv] [14012.427034] ll_intent_lock+0xc5/0xaa0 [lustre] [14012.431964] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14012.434544] ll_lookup_it+0xe0e/0x2270 [lustre] [14012.436361] ? do_raw_spin_unlock+0x75/0x190 [14012.438407] ? _raw_spin_unlock+0x12/0x30 [14012.440202] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [14012.443097] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [14012.445737] ll_atomic_open+0x36a/0xd70 [lustre] [14012.451022] lookup_open+0x356/0xb40 [14012.455364] do_last+0x313/0xfc0 [14012.458581] ? nd_jump_root+0xe5/0x160 [14012.461251] ? path_init+0x437/0x520 [14012.465162] path_openat+0xf7/0x500 [14012.467920] ? do_raw_spin_unlock+0x75/0x190 [14012.470291] do_filp_open+0x99/0x140 [14012.473849] ? getname_flags+0x6e/0x330 [14012.475997] ? __check_object_size+0xff/0x256 [14012.478663] ? do_raw_spin_unlock+0x75/0x190 [14012.480968] ? _raw_spin_unlock+0x12/0x30 [14012.483257] do_sys_openat2+0x2b4/0x410 [14012.486706] do_sys_open+0x73/0xa0 [14012.489014] __x64_sys_openat+0x24/0x30 [14012.491630] do_syscall_64+0xc1/0x440 [14012.497956] entry_SYSCALL_64_after_hwframe+0x49/0xae [14012.505503] RIP: 0033:0x7ff54af45332 [14012.508444] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [14012.526339] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [14012.531554] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [14012.535118] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [14012.538143] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [14012.541393] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [14012.547071] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [14015.967092] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12057s! [14015.971284] Showing busy workqueues and worker pools: [14015.974101] workqueue events: flags=0x0 [14015.976188] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14015.979804] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [14015.984394] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [14015.987482] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [14015.993086] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14015.997211] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [14015.999678] workqueue events_long: flags=0x0 [14016.001227] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14016.004154] pending: lu_objects_flush [obdclass] [14016.006858] workqueue events_unbound: flags=0x2 [14016.008975] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [14016.011502] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [14016.016218] workqueue events_power_efficient: flags=0x80 [14016.018950] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14016.022160] pending: neigh_periodic_work [14016.024399] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14016.027818] pending: do_cache_clean, check_lifetime [14016.029585] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14016.031965] pending: neigh_periodic_work [14016.033691] workqueue writeback: flags=0x4a [14016.035150] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [14016.037670] pending: wb_workfn, wb_workfn [14016.039185] workqueue xprtiod: flags=0xa [14016.040771] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [14016.043140] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [14016.045804] workqueue ipv6_addrconf: flags=0x40008 [14016.047262] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14016.049120] pending: addrconf_verify_work [14016.050568] workqueue knbd0-recv: flags=0x1a [14016.051798] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14016.054172] in-flight: 657:recv_work [14016.055996] workqueue ptlrpc_pinger: flags=0x2 [14016.057696] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14016.058656] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [14016.059909] in-flight: [14016.063226] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14016.063395] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14016.063406] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14016.063409] RIP: 0010:native_safe_halt+0x2e/0x50 [14016.063428] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14016.063435] RSP: 0018:ffffaf9fc1543ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14016.063440] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [14016.063443] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c20b4154 [14016.063445] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [14016.063447] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [14016.063451] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [14016.063454] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [14016.063457] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14016.063459] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [14016.063468] Call Trace: [14016.063527] [14016.064440] 38:ptlrpc_pinger_main [ptlrpc] [14016.064970] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12057s workers=4 idle: 20831 12642 28859 [14016.064988] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14006s workers=2 idle: 659 [14016.130079] ? show_regs.cold.9+0x22/0x2f [14016.131642] ? watchdog_timer_fn.cold.6+0x62/0x10a [14016.133178] ? __hrtimer_run_queues+0x1c2/0x4d0 [14016.135156] ? watchdog+0x40/0x40 [14016.136296] ? hrtimer_interrupt+0x128/0x2c0 [14016.137771] ? smp_apic_timer_interrupt+0xa6/0x270 [14016.139249] ? apic_timer_interrupt+0xf/0x20 [14016.141147] [14016.141890] ? native_safe_halt+0x2e/0x50 [14016.143488] kvm_wait+0xcd/0xe0 [14016.144459] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14016.146254] do_raw_spin_lock+0x177/0x190 [14016.147690] _raw_spin_lock+0x21/0x30 [14016.148975] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14016.151221] ? do_raw_spin_unlock+0x75/0x190 [14016.152708] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14016.154872] ? do_wait_intr+0xf0/0xf0 [14016.156248] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14016.158277] mdc_close+0x53b/0xd70 [mdc] [14016.159812] lmv_close+0x18d/0x490 [lmv] [14016.161188] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [14016.165541] ll_close_inode_openhandle+0x493/0xea0 [lustre] [14016.169780] ll_md_real_close+0xa5/0x2a0 [lustre] [14016.174818] ll_md_close+0x274/0x800 [lustre] [14016.179126] ll_file_release+0x128/0xc50 [lustre] [14016.182866] __fput+0x10d/0x440 [14016.184137] ____fput+0x12/0x20 [14016.185466] task_work_run+0xc6/0x110 [14016.187236] exit_to_usermode_loop+0x1dd/0x1f0 [14016.188851] do_syscall_64+0x3d6/0x440 [14016.191278] entry_SYSCALL_64_after_hwframe+0x49/0xae [14016.195411] RIP: 0033:0x7ff54af45cd8 [14016.196454] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [14016.210679] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [14016.214572] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [14016.222231] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [14016.225222] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [14016.228476] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [14016.231479] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14020.059817] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [14020.066220] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14020.117731] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14020.124138] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14020.128552] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14020.132438] RIP: 0010:native_safe_halt+0x2e/0x50 [14020.138152] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14020.157229] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14020.165939] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [14020.173540] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c2134154 [14020.177974] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [14020.182825] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [14020.187220] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [14020.191449] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [14020.196560] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14020.200684] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [14020.212006] Call Trace: [14020.213468] [14020.214221] ? show_regs.cold.9+0x22/0x2f [14020.216338] ? watchdog_timer_fn.cold.6+0x62/0x10a [14020.222866] ? __hrtimer_run_queues+0x1c2/0x4d0 [14020.224421] ? watchdog+0x40/0x40 [14020.227574] ? hrtimer_interrupt+0x128/0x2c0 [14020.231586] ? smp_apic_timer_interrupt+0xa6/0x270 [14020.236531] ? apic_timer_interrupt+0xf/0x20 [14020.239046] [14020.241084] ? native_safe_halt+0x2e/0x50 [14020.242882] kvm_wait+0xcd/0xe0 [14020.244710] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14020.249994] do_raw_spin_lock+0x177/0x190 [14020.256344] _raw_spin_lock+0x21/0x30 [14020.260506] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14020.264629] process_one_work+0x2c8/0x700 [14020.266908] worker_thread+0x296/0x6e0 [14020.272681] ? rescuer_thread+0x570/0x570 [14020.275212] kthread+0x1d1/0x200 [14020.279914] ? set_kthread_struct+0x70/0x70 [14020.283193] ret_from_fork+0x1f/0x30 [14040.060064] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [14040.068405] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14040.105383] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14040.111727] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14040.115918] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14040.118576] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14040.140092] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14040.147810] RAX: 0000000000002975 RBX: ffff96848811cae0 RCX: 0000000000000001 [14040.160634] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [14040.164382] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [14040.172871] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [14040.181977] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [14040.189734] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [14040.199734] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14040.203604] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [14040.214532] Call Trace: [14040.217814] [14040.218509] ? show_regs.cold.9+0x22/0x2f [14040.219939] ? watchdog_timer_fn.cold.6+0x62/0x10a [14040.226296] ? __hrtimer_run_queues+0x1c2/0x4d0 [14040.227684] ? watchdog+0x40/0x40 [14040.228720] ? hrtimer_interrupt+0x128/0x2c0 [14040.231283] ? smp_apic_timer_interrupt+0xa6/0x270 [14040.232604] ? apic_timer_interrupt+0xf/0x20 [14040.233920] [14040.235671] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14040.237175] do_raw_spin_lock+0x177/0x190 [14040.238216] _raw_spin_lock+0x21/0x30 [14040.243900] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14040.249959] ? do_raw_spin_unlock+0x75/0x190 [14040.259996] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14040.265772] ? do_wait_intr+0xf0/0xf0 [14040.268773] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14040.271233] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [14040.275520] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14040.277836] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14040.280254] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14040.283200] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [14040.286878] mdc_intent_lock+0x353/0x7e0 [mdc] [14040.288903] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14040.294484] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14040.298071] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14040.300779] lmv_intent_open+0x338/0x1150 [lmv] [14040.303095] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14040.306290] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14040.308552] lmv_intent_lock+0x1dd/0x4c0 [lmv] [14040.310833] ll_intent_lock+0xc5/0xaa0 [lustre] [14040.313915] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14040.316115] ll_lookup_it+0xe0e/0x2270 [lustre] [14040.318789] ? do_raw_spin_unlock+0x75/0x190 [14040.321163] ? _raw_spin_unlock+0x12/0x30 [14040.323243] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [14040.326033] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [14040.328812] ll_atomic_open+0x36a/0xd70 [lustre] [14040.331506] lookup_open+0x356/0xb40 [14040.333805] do_last+0x313/0xfc0 [14040.334983] ? nd_jump_root+0xe5/0x160 [14040.336218] ? path_init+0x437/0x520 [14040.338029] path_openat+0xf7/0x500 [14040.339684] ? do_raw_spin_unlock+0x75/0x190 [14040.342357] do_filp_open+0x99/0x140 [14040.344328] ? getname_flags+0x6e/0x330 [14040.345887] ? __check_object_size+0xff/0x256 [14040.347180] ? do_raw_spin_unlock+0x75/0x190 [14040.348682] ? _raw_spin_unlock+0x12/0x30 [14040.350505] do_sys_openat2+0x2b4/0x410 [14040.352334] do_sys_open+0x73/0xa0 [14040.353976] __x64_sys_openat+0x24/0x30 [14040.356022] do_syscall_64+0xc1/0x440 [14040.358109] entry_SYSCALL_64_after_hwframe+0x49/0xae [14040.360679] RIP: 0033:0x7ff54af45332 [14040.362815] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [14040.370034] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [14040.373260] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [14040.377301] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [14040.381229] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [14040.384672] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [14040.389153] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [14044.055258] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [14044.058211] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [14044.063708] Modules linked in: [14044.073740] Modules linked in: [14044.073748] lustre(O) [14044.080194] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14044.080354] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14044.080361] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14044.080364] RIP: 0010:native_safe_halt+0x2e/0x50 [14044.080377] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14044.080382] RSP: 0018:ffffaf9fc1543ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14044.080387] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [14044.080389] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c20b4154 [14044.080392] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [14044.080394] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [14044.080397] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [14044.080400] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [14044.080403] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14044.080405] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [14044.080412] Call Trace: [14044.080418] [14044.080424] ? show_regs.cold.9+0x22/0x2f [14044.080434] ? watchdog_timer_fn.cold.6+0x62/0x10a [14044.080440] ? __hrtimer_run_queues+0x1c2/0x4d0 [14044.080444] ? watchdog+0x40/0x40 [14044.080449] ? hrtimer_interrupt+0x128/0x2c0 [14044.080453] ? smp_apic_timer_interrupt+0xa6/0x270 [14044.080460] ? apic_timer_interrupt+0xf/0x20 [14044.080464] [14044.080467] ? native_safe_halt+0x2e/0x50 [14044.080471] kvm_wait+0xcd/0xe0 [14044.080478] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14044.080485] do_raw_spin_lock+0x177/0x190 [14044.080490] _raw_spin_lock+0x21/0x30 [14044.080494] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14044.080782] ? do_raw_spin_unlock+0x75/0x190 [14044.080788] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14044.082460] ? do_wait_intr+0xf0/0xf0 [14044.082591] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14044.088801] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14044.088956] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14044.088965] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14044.088969] RIP: 0010:native_safe_halt+0x2e/0x50 [14044.088987] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14044.088992] RSP: 0000:ffffaf9fc1b97bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14044.088997] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [14044.089000] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff96848811cae0 [14044.089002] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [14044.089004] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [14044.089007] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14044.089011] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [14044.089014] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14044.089016] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [14044.089203] Call Trace: [14044.089213] [14044.089310] ? show_regs.cold.9+0x22/0x2f [14044.089327] ? watchdog_timer_fn.cold.6+0x62/0x10a [14044.089334] ? __hrtimer_run_queues+0x1c2/0x4d0 [14044.089339] ? watchdog+0x40/0x40 [14044.089343] ? hrtimer_interrupt+0x128/0x2c0 [14044.089347] ? smp_apic_timer_interrupt+0xa6/0x270 [14044.089355] ? apic_timer_interrupt+0xf/0x20 [14044.089360] [14044.089363] ? native_safe_halt+0x2e/0x50 [14044.089367] kvm_wait+0xcd/0xe0 [14044.089378] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14044.089387] do_raw_spin_lock+0x177/0x190 [14044.089391] _raw_spin_lock+0x21/0x30 [14044.089398] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14044.091714] mdc_close+0x53b/0xd70 [mdc] [14044.091780] lmv_close+0x18d/0x490 [lmv] [14044.092080] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [14044.092688] ll_close_inode_openhandle+0x493/0xea0 [lustre] [14044.137161] obd_mod_free+0x46/0x140 [ptlrpc] [14044.146325] ll_md_real_close+0xa5/0x2a0 [lustre] [14044.146498] ll_md_close+0x274/0x800 [lustre] [14044.146580] ll_file_release+0x128/0xc50 [lustre] [14044.146659] __fput+0x10d/0x440 [14044.146671] ____fput+0x12/0x20 [14044.146676] task_work_run+0xc6/0x110 [14044.146685] exit_to_usermode_loop+0x1dd/0x1f0 [14044.146694] do_syscall_64+0x3d6/0x440 [14044.146699] entry_SYSCALL_64_after_hwframe+0x49/0xae [14044.146709] RIP: 0033:0x7ff54af45cd8 [14044.146718] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [14044.146722] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [14044.146727] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [14044.146730] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [14044.146732] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [14044.146735] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [14044.146738] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14044.493189] mdc_commit_open+0x91/0xb0 [mdc] [14044.495307] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14044.497815] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14044.500727] after_reply+0xe2c/0x1780 [ptlrpc] [14044.503835] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14044.506596] ptlrpcd+0xa82/0xe40 [ptlrpc] [14044.509118] ? do_wait_intr+0xf0/0xf0 [14044.510633] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14044.513541] kthread+0x1d1/0x200 [14044.515310] ? set_kthread_struct+0x70/0x70 [14044.517164] ret_from_fork+0x1f/0x30 [14046.178071] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12087s! [14046.187865] Showing busy workqueues and worker pools: [14046.195637] workqueue events: flags=0x0 [14046.202644] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14046.213626] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [14046.219992] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [14046.235570] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [14046.259068] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14046.274534] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [14046.279805] workqueue events_long: flags=0x0 [14046.291458] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14046.304966] pending: lu_objects_flush [obdclass] [14046.312203] workqueue events_unbound: flags=0x2 [14046.318490] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [14046.330356] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [14046.347400] workqueue events_power_efficient: flags=0x80 [14046.356046] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14046.358404] pending: neigh_periodic_work [14046.367250] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14046.380171] pending: do_cache_clean, check_lifetime [14046.389009] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14046.395627] pending: neigh_periodic_work [14046.397802] workqueue writeback: flags=0x4a [14046.403804] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [14046.412505] pending: wb_workfn, wb_workfn [14046.414945] workqueue xprtiod: flags=0xa [14046.420859] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [14046.429540] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [14046.440444] workqueue ipv6_addrconf: flags=0x40008 [14046.444704] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14046.457839] pending: addrconf_verify_work [14046.465575] workqueue knbd0-recv: flags=0x1a [14046.471667] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14046.478765] in-flight: 657:recv_work [14046.480297] workqueue ptlrpc_pinger: flags=0x2 [14046.487196] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14046.494453] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [14046.500752] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12087s workers=4 idle: 20831 12642 28859 [14046.510809] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14036s workers=2 idle: 659 [14048.059091] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [14048.064360] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14048.104799] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14048.123048] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14048.136170] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14048.139919] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14048.143808] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14048.153386] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14048.158942] RAX: 000000000000140e RBX: ffff96848811cae0 RCX: 0000000000000001 [14048.166413] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [14048.170267] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [14048.173865] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [14048.177958] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [14048.182967] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [14048.188558] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14048.190518] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [14048.198133] Call Trace: [14048.198701] [14048.200684] ? show_regs.cold.9+0x22/0x2f [14048.202095] ? watchdog_timer_fn.cold.6+0x62/0x10a [14048.203773] ? __hrtimer_run_queues+0x1c2/0x4d0 [14048.207668] ? watchdog+0x40/0x40 [14048.211226] ? hrtimer_interrupt+0x128/0x2c0 [14048.213104] ? smp_apic_timer_interrupt+0xa6/0x270 [14048.215281] ? apic_timer_interrupt+0xf/0x20 [14048.220498] [14048.221448] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14048.223645] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14048.225867] do_raw_spin_lock+0x177/0x190 [14048.230346] _raw_spin_lock+0x21/0x30 [14048.231628] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14048.233607] process_one_work+0x2c8/0x700 [14048.234591] worker_thread+0x296/0x6e0 [14048.235661] ? rescuer_thread+0x570/0x570 [14048.237630] kthread+0x1d1/0x200 [14048.238699] ? set_kthread_struct+0x70/0x70 [14048.241176] ret_from_fork+0x1f/0x30 [14068.060804] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [14068.070408] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14068.118536] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14068.123059] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14068.127522] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14068.131142] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14068.139610] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14068.142981] RAX: 0000000000000257 RBX: ffff96848811cae0 RCX: 0000000000000001 [14068.146612] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [14068.149985] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [14068.153852] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [14068.157612] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [14068.161163] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [14068.166529] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14068.169340] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [14068.172253] Call Trace: [14068.173323] [14068.174210] ? show_regs.cold.9+0x22/0x2f [14068.175596] ? watchdog_timer_fn.cold.6+0x62/0x10a [14068.177983] ? __hrtimer_run_queues+0x1c2/0x4d0 [14068.179562] ? watchdog+0x40/0x40 [14068.180510] ? hrtimer_interrupt+0x128/0x2c0 [14068.182106] ? smp_apic_timer_interrupt+0xa6/0x270 [14068.184162] ? apic_timer_interrupt+0xf/0x20 [14068.185971] [14068.186702] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14068.189984] do_raw_spin_lock+0x177/0x190 [14068.191564] _raw_spin_lock+0x21/0x30 [14068.193561] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14068.195600] ? do_raw_spin_unlock+0x75/0x190 [14068.199694] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14068.205168] ? do_wait_intr+0xf0/0xf0 [14068.207923] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14068.209512] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [14068.213002] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14068.221871] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14068.227747] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14068.233562] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [14068.239869] mdc_intent_lock+0x353/0x7e0 [mdc] [14068.243693] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14068.249498] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14068.255192] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14068.261149] lmv_intent_open+0x338/0x1150 [lmv] [14068.267785] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14068.274286] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14068.279952] lmv_intent_lock+0x1dd/0x4c0 [lmv] [14068.281347] ll_intent_lock+0xc5/0xaa0 [lustre] [14068.287258] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14068.291293] ll_lookup_it+0xe0e/0x2270 [lustre] [14068.296716] ? do_raw_spin_unlock+0x75/0x190 [14068.299080] ? _raw_spin_unlock+0x12/0x30 [14068.303449] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [14068.310680] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [14068.318319] ll_atomic_open+0x36a/0xd70 [lustre] [14068.326339] lookup_open+0x356/0xb40 [14068.331401] do_last+0x313/0xfc0 [14068.332582] ? nd_jump_root+0xe5/0x160 [14068.333805] ? path_init+0x437/0x520 [14068.343451] path_openat+0xf7/0x500 [14068.347699] ? do_raw_spin_unlock+0x75/0x190 [14068.355245] do_filp_open+0x99/0x140 [14068.356564] ? getname_flags+0x6e/0x330 [14068.368602] ? __check_object_size+0xff/0x256 [14068.373655] ? do_raw_spin_unlock+0x75/0x190 [14068.376593] ? _raw_spin_unlock+0x12/0x30 [14068.378644] do_sys_openat2+0x2b4/0x410 [14068.381791] do_sys_open+0x73/0xa0 [14068.385689] __x64_sys_openat+0x24/0x30 [14068.388444] do_syscall_64+0xc1/0x440 [14068.391559] entry_SYSCALL_64_after_hwframe+0x49/0xae [14068.395678] RIP: 0033:0x7ff54af45332 [14068.399586] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [14068.412969] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [14068.418690] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [14068.426698] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [14068.431685] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [14068.437513] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [14068.442585] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [14072.055098] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [14072.059455] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14072.059959] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [14072.102975] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14072.102994] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14072.102998] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [14072.103014] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [14072.103019] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14072.103025] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [14072.103027] RDX: 0000000000000be7 RSI: 0000000000000003 RDI: ffff96848811cae0 [14072.103030] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [14072.103031] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [14072.103034] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14072.103037] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [14072.103040] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14072.103042] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [14072.103052] Call Trace: [14072.103058] [14072.103065] ? show_regs.cold.9+0x22/0x2f [14072.103075] ? watchdog_timer_fn.cold.6+0x62/0x10a [14072.103081] ? __hrtimer_run_queues+0x1c2/0x4d0 [14072.103086] ? watchdog+0x40/0x40 [14072.103090] ? hrtimer_interrupt+0x128/0x2c0 [14072.103093] ? smp_apic_timer_interrupt+0xa6/0x270 [14072.103101] ? apic_timer_interrupt+0xf/0x20 [14072.103106] [14072.103109] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [14072.103113] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14072.103117] do_raw_spin_lock+0x177/0x190 [14072.103122] _raw_spin_lock+0x21/0x30 [14072.103139] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14072.107318] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14072.107531] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14072.107540] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14072.107543] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14072.107620] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14072.107625] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14072.107630] RAX: 000000000000336e RBX: ffff96848811cae0 RCX: 0000000000000001 [14072.107632] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [14072.107634] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [14072.107636] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [14072.107638] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [14072.107641] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [14072.107644] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14072.107646] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [14072.107654] Call Trace: [14072.107660] [14072.107666] ? show_regs.cold.9+0x22/0x2f [14072.107675] ? watchdog_timer_fn.cold.6+0x62/0x10a [14072.107682] ? __hrtimer_run_queues+0x1c2/0x4d0 [14072.107686] ? watchdog+0x40/0x40 [14072.107690] ? hrtimer_interrupt+0x128/0x2c0 [14072.107694] ? smp_apic_timer_interrupt+0xa6/0x270 [14072.107702] ? apic_timer_interrupt+0xf/0x20 [14072.107707] [14072.107710] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14072.107716] do_raw_spin_lock+0x177/0x190 [14072.107721] _raw_spin_lock+0x21/0x30 [14072.107726] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14072.108423] ? do_raw_spin_unlock+0x75/0x190 [14072.108434] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14072.108698] ? do_wait_intr+0xf0/0xf0 [14072.108709] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14072.108912] mdc_close+0x53b/0xd70 [mdc] [14072.109431] lmv_close+0x18d/0x490 [lmv] [14072.109468] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [14072.109650] ll_close_inode_openhandle+0x493/0xea0 [lustre] [14072.109882] ll_md_real_close+0xa5/0x2a0 [lustre] [14072.109976] ll_md_close+0x274/0x800 [lustre] [14072.110053] ll_file_release+0x128/0xc50 [lustre] [14072.110198] __fput+0x10d/0x440 [14072.110209] ____fput+0x12/0x20 [14072.110213] task_work_run+0xc6/0x110 [14072.110223] exit_to_usermode_loop+0x1dd/0x1f0 [14072.110231] do_syscall_64+0x3d6/0x440 [14072.110235] entry_SYSCALL_64_after_hwframe+0x49/0xae [14072.110244] RIP: 0033:0x7ff54af45cd8 [14072.110254] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [14072.110258] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [14072.110263] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [14072.110266] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [14072.110268] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [14072.110270] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [14072.110273] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14072.405955] obd_mod_free+0x46/0x140 [ptlrpc] [14072.407854] mdc_commit_open+0x91/0xb0 [mdc] [14072.408997] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14072.411298] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14072.413519] after_reply+0xe2c/0x1780 [ptlrpc] [14072.415634] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14072.418044] ptlrpcd+0xa82/0xe40 [ptlrpc] [14072.421492] ? do_wait_intr+0xf0/0xf0 [14072.423518] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14072.427113] kthread+0x1d1/0x200 [14072.428601] ? set_kthread_struct+0x70/0x70 [14072.430009] ret_from_fork+0x1f/0x30 [14074.169061] rcu: INFO: rcu_sched self-detected stall on CPU [14074.172449] rcu: 0-...!: (6455045 ticks this GP) idle=98a/1/0x4000000000000002 softirq=125154/125154 fqs=959 [14074.180002] (t=12120329 jiffies g=253585 q=6830) [14074.184276] rcu: rcu_sched kthread starved for 12115289 jiffies! g253585 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [14074.187575] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [14074.192519] rcu: RCU grace-period kthread stack dump: [14074.194660] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [14074.199383] Call Trace: [14074.200534] __schedule+0x351/0xcb0 [14074.202509] schedule+0xc0/0x180 [14074.204229] schedule_timeout+0xb4/0x190 [14074.205808] ? do_raw_spin_unlock+0x75/0x190 [14074.207829] ? __next_timer_interrupt+0x160/0x160 [14074.209914] ? prepare_to_swait_event+0xf3/0x140 [14074.211558] rcu_gp_kthread+0x8dd/0x15b0 [14074.213207] ? force_qs_rnp+0x2c0/0x2c0 [14074.214651] kthread+0x1d1/0x200 [14074.216212] ? set_kthread_struct+0x70/0x70 [14074.218211] ret_from_fork+0x1f/0x30 [14074.219339] rcu: Stack dump where RCU GP kthread last ran: [14074.220721] NMI backtrace for cpu 0 [14074.221634] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14074.224580] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14074.226985] Call Trace: [14074.227623] [14074.228256] ? dump_stack+0xbb/0x10e [14074.229215] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [14074.230670] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [14074.232232] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [14074.233762] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [14074.235289] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [14074.236609] ? rcu_check_gp_kthread_starvation+0x16f/0x188 [14074.238202] ? rcu_sched_clock_irq.cold.51+0xce/0x307 [14074.239465] ? update_process_times+0x6d/0xc0 [14074.240628] ? tick_sched_handle.isra.18+0x27/0x90 [14074.241952] ? tick_sched_timer+0x53/0xb0 [14074.243065] ? __hrtimer_run_queues+0x1c2/0x4d0 [14074.244208] ? tick_sched_handle.isra.18+0x90/0x90 [14074.248310] ? hrtimer_interrupt+0x128/0x2c0 [14074.249987] ? smp_apic_timer_interrupt+0xa6/0x270 [14074.251399] ? apic_timer_interrupt+0xf/0x20 [14074.253314] [14074.253995] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [14074.256072] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14074.257925] ? do_raw_spin_lock+0x177/0x190 [14074.259271] ? _raw_spin_lock+0x21/0x30 [14074.261164] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14074.263490] ? obd_mod_free+0x46/0x140 [ptlrpc] [14074.265404] ? mdc_commit_open+0x91/0xb0 [mdc] [14074.267191] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14074.269471] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14074.272170] ? after_reply+0xe2c/0x1780 [ptlrpc] [14074.274393] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14074.278258] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [14074.281308] ? do_wait_intr+0xf0/0xf0 [14074.283701] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14074.290164] ? kthread+0x1d1/0x200 [14074.293698] ? set_kthread_struct+0x70/0x70 [14074.296295] ? ret_from_fork+0x1f/0x30 [14074.299227] NMI backtrace for cpu 0 [14074.301356] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14074.310783] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14074.317261] Call Trace: [14074.318814] [14074.319671] ? dump_stack+0xbb/0x10e [14074.327634] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [14074.334494] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [14074.338248] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [14074.341923] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [14074.347473] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [14074.351411] ? rcu_dump_cpu_stacks+0x11b/0x171 [14074.354728] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [14074.359173] ? update_process_times+0x6d/0xc0 [14074.361991] ? tick_sched_handle.isra.18+0x27/0x90 [14074.364072] ? tick_sched_timer+0x53/0xb0 [14074.367336] ? __hrtimer_run_queues+0x1c2/0x4d0 [14074.370383] ? tick_sched_handle.isra.18+0x90/0x90 [14074.373650] ? hrtimer_interrupt+0x128/0x2c0 [14074.378241] ? smp_apic_timer_interrupt+0xa6/0x270 [14074.383872] ? apic_timer_interrupt+0xf/0x20 [14074.388994] [14074.389746] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [14074.404220] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14074.410363] ? do_raw_spin_lock+0x177/0x190 [14074.413980] ? _raw_spin_lock+0x21/0x30 [14074.415743] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14074.419611] ? obd_mod_free+0x46/0x140 [ptlrpc] [14074.424803] ? mdc_commit_open+0x91/0xb0 [mdc] [14074.427709] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14074.432272] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14074.435453] ? after_reply+0xe2c/0x1780 [ptlrpc] [14074.438294] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14074.441413] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [14074.444454] ? do_wait_intr+0xf0/0xf0 [14074.447391] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14074.451055] ? kthread+0x1d1/0x200 [14074.452991] ? set_kthread_struct+0x70/0x70 [14074.454963] ? ret_from_fork+0x1f/0x30 [14076.059056] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [14076.067269] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14076.123566] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14076.131541] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14076.140484] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14076.156295] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14076.163388] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14076.182640] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14076.185699] RAX: 0000000000006b82 RBX: ffff96848811cae0 RCX: 0000000000000001 [14076.189290] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [14076.200281] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [14076.221236] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [14076.226630] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [14076.229641] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [14076.232876] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14076.236638] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [14076.239528] Call Trace: [14076.240486] [14076.241272] ? show_regs.cold.9+0x22/0x2f [14076.242827] ? watchdog_timer_fn.cold.6+0x62/0x10a [14076.244601] ? __hrtimer_run_queues+0x1c2/0x4d0 [14076.246327] ? watchdog+0x40/0x40 [14076.247523] ? hrtimer_interrupt+0x128/0x2c0 [14076.248982] ? smp_apic_timer_interrupt+0xa6/0x270 [14076.250346] ? apic_timer_interrupt+0xf/0x20 [14076.251852] [14076.252638] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14076.254671] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14076.256690] do_raw_spin_lock+0x177/0x190 [14076.258175] _raw_spin_lock+0x21/0x30 [14076.259556] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14076.261580] process_one_work+0x2c8/0x700 [14076.263146] worker_thread+0x296/0x6e0 [14076.264364] ? rescuer_thread+0x570/0x570 [14076.265854] kthread+0x1d1/0x200 [14076.266957] ? set_kthread_struct+0x70/0x70 [14076.268406] ret_from_fork+0x1f/0x30 [14076.895146] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12118s! [14076.900666] Showing busy workqueues and worker pools: [14076.904989] workqueue events: flags=0x0 [14076.908567] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14076.913832] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [14076.919848] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [14076.923840] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [14076.932399] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14076.936868] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [14076.939867] workqueue events_long: flags=0x0 [14076.942789] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14076.948600] pending: lu_objects_flush [obdclass] [14076.952956] workqueue events_unbound: flags=0x2 [14076.962201] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [14076.967305] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [14076.975013] workqueue events_power_efficient: flags=0x80 [14076.979259] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14076.986368] pending: neigh_periodic_work [14076.989079] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14076.995255] pending: do_cache_clean, check_lifetime [14076.998944] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14077.004663] pending: neigh_periodic_work [14077.014205] workqueue writeback: flags=0x4a [14077.016440] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [14077.027399] pending: wb_workfn, wb_workfn [14077.035295] workqueue xprtiod: flags=0xa [14077.040635] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [14077.047374] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [14077.055940] workqueue ipv6_addrconf: flags=0x40008 [14077.060236] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14077.066671] pending: addrconf_verify_work [14077.069570] workqueue knbd0-recv: flags=0x1a [14077.074551] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14077.082491] in-flight: 657:recv_work [14077.086357] workqueue ptlrpc_pinger: flags=0x2 [14077.089039] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14077.093206] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [14077.105802] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12118s workers=4 idle: 20831 12642 28859 [14077.119448] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14067s workers=2 idle: 659 [14096.063767] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [14096.065908] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14096.102479] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14096.107128] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14096.111367] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14096.114751] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14096.122050] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14096.124628] RAX: 0000000000007f09 RBX: ffff96848811cae0 RCX: 0000000000000001 [14096.127165] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [14096.129954] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [14096.132347] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [14096.134824] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [14096.137185] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [14096.139864] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14096.142017] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [14096.144509] Call Trace: [14096.145402] [14096.146245] ? show_regs.cold.9+0x22/0x2f [14096.147843] ? watchdog_timer_fn.cold.6+0x62/0x10a [14096.149740] ? __hrtimer_run_queues+0x1c2/0x4d0 [14096.151358] ? watchdog+0x40/0x40 [14096.152583] ? hrtimer_interrupt+0x128/0x2c0 [14096.155408] ? smp_apic_timer_interrupt+0xa6/0x270 [14096.157883] ? apic_timer_interrupt+0xf/0x20 [14096.160437] [14096.161535] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14096.165062] do_raw_spin_lock+0x177/0x190 [14096.167641] _raw_spin_lock+0x21/0x30 [14096.171901] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14096.181571] ? do_raw_spin_unlock+0x75/0x190 [14096.185316] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14096.189189] ? do_wait_intr+0xf0/0xf0 [14096.192837] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14096.196236] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [14096.199481] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14096.203533] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14096.208434] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14096.212511] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [14096.215187] mdc_intent_lock+0x353/0x7e0 [mdc] [14096.217499] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14096.221335] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14096.225210] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14096.227792] lmv_intent_open+0x338/0x1150 [lmv] [14096.231514] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14096.234565] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14096.238130] lmv_intent_lock+0x1dd/0x4c0 [lmv] [14096.240267] ll_intent_lock+0xc5/0xaa0 [lustre] [14096.242791] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14096.245337] ll_lookup_it+0xe0e/0x2270 [lustre] [14096.246873] ? do_raw_spin_unlock+0x75/0x190 [14096.249024] ? _raw_spin_unlock+0x12/0x30 [14096.250883] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [14096.253445] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [14096.256474] ll_atomic_open+0x36a/0xd70 [lustre] [14096.259458] lookup_open+0x356/0xb40 [14096.261470] do_last+0x313/0xfc0 [14096.264512] ? nd_jump_root+0xe5/0x160 [14096.266344] ? path_init+0x437/0x520 [14096.268379] path_openat+0xf7/0x500 [14096.270054] ? do_raw_spin_unlock+0x75/0x190 [14096.271371] do_filp_open+0x99/0x140 [14096.272971] ? getname_flags+0x6e/0x330 [14096.274450] ? __check_object_size+0xff/0x256 [14096.275907] ? do_raw_spin_unlock+0x75/0x190 [14096.277370] ? _raw_spin_unlock+0x12/0x30 [14096.279442] do_sys_openat2+0x2b4/0x410 [14096.281331] do_sys_open+0x73/0xa0 [14096.283010] __x64_sys_openat+0x24/0x30 [14096.286107] do_syscall_64+0xc1/0x440 [14096.288973] entry_SYSCALL_64_after_hwframe+0x49/0xae [14096.292441] RIP: 0033:0x7ff54af45332 [14096.294371] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [14096.311994] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [14096.318227] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [14096.326468] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [14096.331438] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [14096.338239] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [14096.342052] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [14100.058946] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [14100.063990] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14100.106956] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14100.121165] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14100.130061] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14100.135850] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14100.148840] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14100.152397] RAX: 0000000000004508 RBX: ffff96848811cae0 RCX: 0000000000000001 [14100.156644] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [14100.168815] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [14100.175181] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [14100.180083] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [14100.183397] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [14100.186682] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14100.188952] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [14100.191596] Call Trace: [14100.192918] [14100.193873] ? show_regs.cold.9+0x22/0x2f [14100.196605] ? watchdog_timer_fn.cold.6+0x62/0x10a [14100.199651] ? __hrtimer_run_queues+0x1c2/0x4d0 [14100.202928] ? watchdog+0x40/0x40 [14100.204362] ? hrtimer_interrupt+0x128/0x2c0 [14100.205974] ? smp_apic_timer_interrupt+0xa6/0x270 [14100.214466] ? apic_timer_interrupt+0xf/0x20 [14100.219058] [14100.221696] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14100.224131] do_raw_spin_lock+0x177/0x190 [14100.228212] _raw_spin_lock+0x21/0x30 [14100.230225] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14100.232311] ? do_raw_spin_unlock+0x75/0x190 [14100.239425] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14100.242965] ? do_wait_intr+0xf0/0xf0 [14100.244166] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14100.247842] mdc_close+0x53b/0xd70 [mdc] [14100.253716] lmv_close+0x18d/0x490 [lmv] [14100.257012] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [14100.265248] ll_close_inode_openhandle+0x493/0xea0 [lustre] [14100.270965] ll_md_real_close+0xa5/0x2a0 [lustre] [14100.277891] ll_md_close+0x274/0x800 [lustre] [14100.279708] ll_file_release+0x128/0xc50 [lustre] [14100.281993] __fput+0x10d/0x440 [14100.283846] ____fput+0x12/0x20 [14100.285901] task_work_run+0xc6/0x110 [14100.287457] exit_to_usermode_loop+0x1dd/0x1f0 [14100.289391] do_syscall_64+0x3d6/0x440 [14100.291180] entry_SYSCALL_64_after_hwframe+0x49/0xae [14100.296236] RIP: 0033:0x7ff54af45cd8 [14100.300181] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [14100.309337] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [14100.312558] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [14100.315227] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [14100.317443] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [14100.319933] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [14100.322277] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14104.057572] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [14104.066141] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14104.066291] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [14104.158999] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14104.159014] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14104.159019] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [14104.159036] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [14104.159040] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14104.159046] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [14104.159048] RDX: 0000000000003a60 RSI: 0000000000000003 RDI: ffff96848811cae0 [14104.159051] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [14104.159053] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [14104.159055] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14104.159072] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [14104.159076] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14104.159078] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [14104.159085] Call Trace: [14104.159091] [14104.162362] ? show_regs.cold.9+0x22/0x2f [14104.162387] ? watchdog_timer_fn.cold.6+0x62/0x10a [14104.162394] ? __hrtimer_run_queues+0x1c2/0x4d0 [14104.162398] ? watchdog+0x40/0x40 [14104.162403] ? hrtimer_interrupt+0x128/0x2c0 [14104.162407] ? smp_apic_timer_interrupt+0xa6/0x270 [14104.162416] ? apic_timer_interrupt+0xf/0x20 [14104.162421] [14104.162426] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [14104.162433] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14104.162806] do_raw_spin_lock+0x177/0x190 [14104.162819] _raw_spin_lock+0x21/0x30 [14104.162826] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14104.166751] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14104.166833] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14104.166841] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14104.166848] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14104.167138] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14104.167150] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14104.167155] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [14104.167160] RAX: 0000000000007c93 RBX: ffff96848811cae0 RCX: 0000000000000001 [14104.167163] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [14104.167165] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [14104.167168] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [14104.167170] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [14104.167173] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [14104.167175] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14104.167177] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [14104.167186] Call Trace: [14104.167191] [14104.167195] ? show_regs.cold.9+0x22/0x2f [14104.167204] ? watchdog_timer_fn.cold.6+0x62/0x10a [14104.167213] ? __hrtimer_run_queues+0x1c2/0x4d0 [14104.167217] ? watchdog+0x40/0x40 [14104.167222] ? hrtimer_interrupt+0x128/0x2c0 [14104.167227] ? smp_apic_timer_interrupt+0xa6/0x270 [14104.167234] ? apic_timer_interrupt+0xf/0x20 [14104.167238] [14104.167241] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14104.167246] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14104.167252] do_raw_spin_lock+0x177/0x190 [14104.167256] _raw_spin_lock+0x21/0x30 [14104.167261] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14104.167569] process_one_work+0x2c8/0x700 [14104.167577] worker_thread+0x296/0x6e0 [14104.167582] ? rescuer_thread+0x570/0x570 [14104.167585] kthread+0x1d1/0x200 [14104.167589] ? set_kthread_struct+0x70/0x70 [14104.167593] ret_from_fork+0x1f/0x30 [14104.772438] obd_mod_free+0x46/0x140 [ptlrpc] [14104.774222] mdc_commit_open+0x91/0xb0 [mdc] [14104.796989] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14104.805569] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14104.815701] after_reply+0xe2c/0x1780 [ptlrpc] [14104.817559] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14104.822698] ptlrpcd+0xa82/0xe40 [ptlrpc] [14104.839604] ? do_wait_intr+0xf0/0xf0 [14104.855839] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14104.863829] kthread+0x1d1/0x200 [14104.866595] ? set_kthread_struct+0x70/0x70 [14104.871593] ret_from_fork+0x1f/0x30 [14107.619860] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12148s! [14107.628142] Showing busy workqueues and worker pools: [14107.635917] workqueue events: flags=0x0 [14107.653468] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14107.665481] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [14107.689461] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [14107.702745] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [14107.727020] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14107.729334] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [14107.747807] workqueue events_long: flags=0x0 [14107.756865] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14107.768914] pending: lu_objects_flush [obdclass] [14107.782449] workqueue events_unbound: flags=0x2 [14107.790453] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [14107.800186] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [14107.805089] workqueue events_power_efficient: flags=0x80 [14107.818907] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14107.830569] pending: neigh_periodic_work [14107.833280] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14107.843659] pending: do_cache_clean, check_lifetime [14107.851970] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14107.865571] pending: neigh_periodic_work [14107.873322] workqueue writeback: flags=0x4a [14107.881344] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [14107.890226] pending: wb_workfn, wb_workfn [14107.896605] workqueue xprtiod: flags=0xa [14107.901981] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [14107.910213] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [14107.916297] workqueue ipv6_addrconf: flags=0x40008 [14107.921834] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14107.937958] pending: addrconf_verify_work [14107.941555] workqueue knbd0-recv: flags=0x1a [14107.949365] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14107.961699] in-flight: 657:recv_work [14107.966543] workqueue ptlrpc_pinger: flags=0x2 [14107.968483] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14107.971837] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [14107.978415] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12149s workers=4 idle: 20831 12642 28859 [14107.988569] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14098s workers=2 idle: 659 [14124.062440] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [14124.079793] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14124.133717] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14124.141371] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14124.148493] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14124.152680] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14124.168897] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14124.174060] RAX: 0000000000002009 RBX: ffff96848811cae0 RCX: 0000000000000001 [14124.178930] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [14124.183510] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [14124.192168] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [14124.201476] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [14124.206642] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [14124.217027] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14124.224417] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [14124.233653] Call Trace: [14124.238158] [14124.241217] ? show_regs.cold.9+0x22/0x2f [14124.245532] ? watchdog_timer_fn.cold.6+0x62/0x10a [14124.249721] ? __hrtimer_run_queues+0x1c2/0x4d0 [14124.254257] ? watchdog+0x40/0x40 [14124.260024] ? hrtimer_interrupt+0x128/0x2c0 [14124.266226] ? smp_apic_timer_interrupt+0xa6/0x270 [14124.271015] ? apic_timer_interrupt+0xf/0x20 [14124.274313] [14124.276911] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14124.283784] do_raw_spin_lock+0x177/0x190 [14124.289029] _raw_spin_lock+0x21/0x30 [14124.290371] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14124.306662] ? do_raw_spin_unlock+0x75/0x190 [14124.314150] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14124.318367] ? do_wait_intr+0xf0/0xf0 [14124.320788] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14124.328479] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [14124.332056] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14124.335043] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14124.337193] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14124.339514] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [14124.342016] mdc_intent_lock+0x353/0x7e0 [mdc] [14124.343740] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14124.346033] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14124.348627] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14124.350803] lmv_intent_open+0x338/0x1150 [lmv] [14124.352705] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14124.354996] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14124.357230] lmv_intent_lock+0x1dd/0x4c0 [lmv] [14124.359083] ll_intent_lock+0xc5/0xaa0 [lustre] [14124.361028] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14124.363228] ll_lookup_it+0xe0e/0x2270 [lustre] [14124.365192] ? do_raw_spin_unlock+0x75/0x190 [14124.366975] ? _raw_spin_unlock+0x12/0x30 [14124.368603] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [14124.370745] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [14124.372883] ll_atomic_open+0x36a/0xd70 [lustre] [14124.374660] lookup_open+0x356/0xb40 [14124.377700] do_last+0x313/0xfc0 [14124.380461] ? nd_jump_root+0xe5/0x160 [14124.382579] ? path_init+0x437/0x520 [14124.384561] path_openat+0xf7/0x500 [14124.386429] ? do_raw_spin_unlock+0x75/0x190 [14124.388194] do_filp_open+0x99/0x140 [14124.389953] ? getname_flags+0x6e/0x330 [14124.392007] ? __check_object_size+0xff/0x256 [14124.394536] ? do_raw_spin_unlock+0x75/0x190 [14124.396699] ? _raw_spin_unlock+0x12/0x30 [14124.398790] do_sys_openat2+0x2b4/0x410 [14124.400779] do_sys_open+0x73/0xa0 [14124.402575] __x64_sys_openat+0x24/0x30 [14124.406215] do_syscall_64+0xc1/0x440 [14124.409562] entry_SYSCALL_64_after_hwframe+0x49/0xae [14124.412480] RIP: 0033:0x7ff54af45332 [14124.414595] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [14124.425138] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [14124.429151] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [14124.433519] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [14124.440070] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [14124.444169] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [14124.447895] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [14128.058025] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [14128.062727] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14128.105028] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14128.112070] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14128.122648] RIP: 0010:native_safe_halt+0x2e/0x50 [14128.126638] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14128.153713] RSP: 0018:ffffaf9fc1543ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14128.157134] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [14128.166789] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c20b4154 [14128.173505] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [14128.185298] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [14128.191087] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [14128.195069] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [14128.202925] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14128.205869] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [14128.209172] Call Trace: [14128.210071] [14128.214112] ? show_regs.cold.9+0x22/0x2f [14128.216303] ? watchdog_timer_fn.cold.6+0x62/0x10a [14128.220008] ? __hrtimer_run_queues+0x1c2/0x4d0 [14128.221564] ? watchdog+0x40/0x40 [14128.225754] ? hrtimer_interrupt+0x128/0x2c0 [14128.235810] ? smp_apic_timer_interrupt+0xa6/0x270 [14128.242992] ? apic_timer_interrupt+0xf/0x20 [14128.245899] [14128.246965] ? native_safe_halt+0x2e/0x50 [14128.249986] kvm_wait+0xcd/0xe0 [14128.252141] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14128.255563] do_raw_spin_lock+0x177/0x190 [14128.257980] _raw_spin_lock+0x21/0x30 [14128.262068] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14128.268631] ? do_raw_spin_unlock+0x75/0x190 [14128.274622] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14128.291441] ? do_wait_intr+0xf0/0xf0 [14128.295357] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14128.297973] mdc_close+0x53b/0xd70 [mdc] [14128.306006] lmv_close+0x18d/0x490 [lmv] [14128.309840] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [14128.314951] ll_close_inode_openhandle+0x493/0xea0 [lustre] [14128.321592] ll_md_real_close+0xa5/0x2a0 [lustre] [14128.324589] ll_md_close+0x274/0x800 [lustre] [14128.328822] ll_file_release+0x128/0xc50 [lustre] [14128.331842] __fput+0x10d/0x440 [14128.334442] ____fput+0x12/0x20 [14128.340136] task_work_run+0xc6/0x110 [14128.341906] exit_to_usermode_loop+0x1dd/0x1f0 [14128.346683] do_syscall_64+0x3d6/0x440 [14128.348927] entry_SYSCALL_64_after_hwframe+0x49/0xae [14128.351716] RIP: 0033:0x7ff54af45cd8 [14128.353141] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [14128.364194] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [14128.369958] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [14128.374773] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [14128.380054] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [14128.385003] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [14128.390181] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14132.055138] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [14132.059650] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14132.060589] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [14132.081498] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14132.083974] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw [14132.088567] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14132.088577] RIP: 0010:native_safe_halt+0x2e/0x50 [14132.088593] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14132.088598] RSP: 0000:ffffaf9fc1b97bb8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [14132.088603] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [14132.088605] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff96848811cae0 [14132.088608] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [14132.088611] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [14132.088614] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14132.088618] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [14132.088621] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14132.088623] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [14132.088633] Call Trace: [14132.088640] [14132.088647] ? show_regs.cold.9+0x22/0x2f [14132.088661] ? watchdog_timer_fn.cold.6+0x62/0x10a [14132.088668] ? __hrtimer_run_queues+0x1c2/0x4d0 [14132.088672] ? watchdog+0x40/0x40 [14132.088676] ? hrtimer_interrupt+0x128/0x2c0 [14132.088680] ? smp_apic_timer_interrupt+0xa6/0x270 [14132.088688] ? apic_timer_interrupt+0xf/0x20 [14132.088693] [14132.088696] ? native_safe_halt+0x2e/0x50 [14132.088701] kvm_wait+0xcd/0xe0 [14132.088709] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14132.088718] do_raw_spin_lock+0x177/0x190 [14132.088723] _raw_spin_lock+0x21/0x30 [14132.088730] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14132.101715] libata [14132.104750] obd_mod_free+0x46/0x140 [ptlrpc] [14132.106369] dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14132.106386] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14132.106393] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14132.106400] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14132.106683] RIP: 0010:native_safe_halt+0x2e/0x50 [14132.106693] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14132.106698] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14132.106703] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [14132.106705] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c2134154 [14132.106708] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [14132.106710] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [14132.106713] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [14132.106716] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [14132.106719] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14132.106721] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [14132.106729] Call Trace: [14132.116004] mdc_commit_open+0x91/0xb0 [mdc] [14132.116244] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14132.116602] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14132.116871] after_reply+0xe2c/0x1780 [ptlrpc] [14132.117271] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14132.117482] ptlrpcd+0xa82/0xe40 [ptlrpc] [14132.117762] ? do_wait_intr+0xf0/0xf0 [14132.117774] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14132.118033] kthread+0x1d1/0x200 [14132.118165] ? set_kthread_struct+0x70/0x70 [14132.118225] ret_from_fork+0x1f/0x30 [14132.314792] [14132.318366] ? show_regs.cold.9+0x22/0x2f [14132.321628] ? watchdog_timer_fn.cold.6+0x62/0x10a [14132.328876] ? __hrtimer_run_queues+0x1c2/0x4d0 [14132.333830] ? watchdog+0x40/0x40 [14132.335017] ? hrtimer_interrupt+0x128/0x2c0 [14132.342804] ? smp_apic_timer_interrupt+0xa6/0x270 [14132.353488] ? apic_timer_interrupt+0xf/0x20 [14132.355311] [14132.356817] ? native_safe_halt+0x2e/0x50 [14132.364281] kvm_wait+0xcd/0xe0 [14132.368717] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14132.371966] do_raw_spin_lock+0x177/0x190 [14132.374607] _raw_spin_lock+0x21/0x30 [14132.377022] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14132.380728] process_one_work+0x2c8/0x700 [14132.382934] worker_thread+0x296/0x6e0 [14132.385103] ? rescuer_thread+0x570/0x570 [14132.388087] kthread+0x1d1/0x200 [14132.390625] ? set_kthread_struct+0x70/0x70 [14132.392467] ret_from_fork+0x1f/0x30 [14138.344029] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12179s! [14138.349381] Showing busy workqueues and worker pools: [14138.356651] workqueue events: flags=0x0 [14138.360168] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14138.365367] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [14138.370347] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [14138.377228] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [14138.387033] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14138.392399] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [14138.396304] workqueue events_long: flags=0x0 [14138.399084] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14138.410639] pending: lu_objects_flush [obdclass] [14138.420074] workqueue events_unbound: flags=0x2 [14138.425033] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [14138.428941] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [14138.439817] workqueue events_power_efficient: flags=0x80 [14138.444125] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14138.450185] pending: neigh_periodic_work [14138.455023] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14138.460232] pending: do_cache_clean, check_lifetime [14138.465368] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14138.472087] pending: neigh_periodic_work [14138.473987] workqueue writeback: flags=0x4a [14138.477867] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [14138.484084] pending: wb_workfn, wb_workfn [14138.485741] workqueue xprtiod: flags=0xa [14138.494877] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [14138.503383] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [14138.511274] workqueue ipv6_addrconf: flags=0x40008 [14138.515402] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14138.526914] pending: addrconf_verify_work [14138.531131] workqueue knbd0-recv: flags=0x1a [14138.534860] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14138.540936] in-flight: 657:recv_work [14138.542977] workqueue ptlrpc_pinger: flags=0x2 [14138.546013] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14138.552354] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [14138.555841] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12179s workers=4 idle: 20831 12642 28859 [14138.563950] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14128s workers=2 idle: 659 [14152.061542] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [14152.078870] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14152.133943] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14152.152465] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14152.159878] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14152.164143] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14152.180293] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [14152.192443] RAX: 000000000000294c RBX: ffff96848811cae0 RCX: 0000000000000001 [14152.201256] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [14152.213712] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [14152.216121] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [14152.227246] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [14152.239387] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [14152.246497] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14152.252945] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [14152.258009] Call Trace: [14152.259605] [14152.262576] ? show_regs.cold.9+0x22/0x2f [14152.268540] ? watchdog_timer_fn.cold.6+0x62/0x10a [14152.271548] ? __hrtimer_run_queues+0x1c2/0x4d0 [14152.276246] ? watchdog+0x40/0x40 [14152.278383] ? hrtimer_interrupt+0x128/0x2c0 [14152.281081] ? smp_apic_timer_interrupt+0xa6/0x270 [14152.285525] ? apic_timer_interrupt+0xf/0x20 [14152.288699] [14152.290511] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14152.293399] do_raw_spin_lock+0x177/0x190 [14152.296576] _raw_spin_lock+0x21/0x30 [14152.299365] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14152.302701] ? do_raw_spin_unlock+0x75/0x190 [14152.304962] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14152.307942] ? do_wait_intr+0xf0/0xf0 [14152.309989] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14152.312980] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [14152.316653] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14152.326896] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14152.330288] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14152.333332] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [14152.336271] mdc_intent_lock+0x353/0x7e0 [mdc] [14152.342457] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14152.345489] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14152.350381] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14152.356763] lmv_intent_open+0x338/0x1150 [lmv] [14152.359235] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14152.363239] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14152.368722] lmv_intent_lock+0x1dd/0x4c0 [lmv] [14152.371048] ll_intent_lock+0xc5/0xaa0 [lustre] [14152.373403] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14152.375688] ll_lookup_it+0xe0e/0x2270 [lustre] [14152.377850] ? do_raw_spin_unlock+0x75/0x190 [14152.379608] ? _raw_spin_unlock+0x12/0x30 [14152.381311] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [14152.383357] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [14152.385578] ll_atomic_open+0x36a/0xd70 [lustre] [14152.387272] lookup_open+0x356/0xb40 [14152.389070] do_last+0x313/0xfc0 [14152.390434] ? nd_jump_root+0xe5/0x160 [14152.393087] ? path_init+0x437/0x520 [14152.397465] path_openat+0xf7/0x500 [14152.403681] ? do_raw_spin_unlock+0x75/0x190 [14152.408483] do_filp_open+0x99/0x140 [14152.410491] ? getname_flags+0x6e/0x330 [14152.413601] ? __check_object_size+0xff/0x256 [14152.416701] ? do_raw_spin_unlock+0x75/0x190 [14152.420566] ? _raw_spin_unlock+0x12/0x30 [14152.424188] do_sys_openat2+0x2b4/0x410 [14152.425537] do_sys_open+0x73/0xa0 [14152.426706] __x64_sys_openat+0x24/0x30 [14152.430560] do_syscall_64+0xc1/0x440 [14152.434797] entry_SYSCALL_64_after_hwframe+0x49/0xae [14152.441622] RIP: 0033:0x7ff54af45332 [14152.444523] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [14152.457776] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [14152.463769] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [14152.470580] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [14152.473482] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [14152.481910] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [14152.485517] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [14156.059437] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [14156.069517] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14156.122771] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14156.127946] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14156.133128] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14156.135060] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14156.144360] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14156.147886] RAX: 000000000000779e RBX: ffff96848811cae0 RCX: 0000000000000001 [14156.150951] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [14156.154283] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [14156.157874] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [14156.160926] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [14156.164375] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [14156.169236] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14156.171357] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [14156.172994] Call Trace: [14156.174399] [14156.175306] ? show_regs.cold.9+0x22/0x2f [14156.176923] ? watchdog_timer_fn.cold.6+0x62/0x10a [14156.178605] ? __hrtimer_run_queues+0x1c2/0x4d0 [14156.180879] ? watchdog+0x40/0x40 [14156.182469] ? hrtimer_interrupt+0x128/0x2c0 [14156.185365] ? smp_apic_timer_interrupt+0xa6/0x270 [14156.190437] ? apic_timer_interrupt+0xf/0x20 [14156.191883] [14156.194312] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14156.210089] do_raw_spin_lock+0x177/0x190 [14156.213191] _raw_spin_lock+0x21/0x30 [14156.220364] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14156.226773] ? do_raw_spin_unlock+0x75/0x190 [14156.229865] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14156.235446] ? do_wait_intr+0xf0/0xf0 [14156.239595] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14156.242870] mdc_close+0x53b/0xd70 [mdc] [14156.245489] lmv_close+0x18d/0x490 [lmv] [14156.250557] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [14156.258239] ll_close_inode_openhandle+0x493/0xea0 [lustre] [14156.263852] ll_md_real_close+0xa5/0x2a0 [lustre] [14156.268449] ll_md_close+0x274/0x800 [lustre] [14156.278576] ll_file_release+0x128/0xc50 [lustre] [14156.282393] __fput+0x10d/0x440 [14156.286194] ____fput+0x12/0x20 [14156.289808] task_work_run+0xc6/0x110 [14156.292558] exit_to_usermode_loop+0x1dd/0x1f0 [14156.298811] do_syscall_64+0x3d6/0x440 [14156.300494] entry_SYSCALL_64_after_hwframe+0x49/0xae [14156.307225] RIP: 0033:0x7ff54af45cd8 [14156.309025] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [14156.336603] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [14156.345324] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [14156.349319] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [14156.362247] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [14156.374620] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [14156.384146] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14160.059056] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [14160.071093] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14160.109546] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14160.120987] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14160.129366] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14160.135658] RIP: 0010:native_safe_halt+0x2e/0x50 [14160.143739] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14160.154821] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14160.159159] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [14160.163469] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c2134154 [14160.169842] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [14160.178476] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [14160.184846] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [14160.192194] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [14160.208666] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14160.210392] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [14160.220989] Call Trace: [14160.221860] [14160.222559] ? show_regs.cold.9+0x22/0x2f [14160.236683] ? watchdog_timer_fn.cold.6+0x62/0x10a [14160.244316] ? __hrtimer_run_queues+0x1c2/0x4d0 [14160.250142] ? watchdog+0x40/0x40 [14160.254598] ? hrtimer_interrupt+0x128/0x2c0 [14160.256816] ? smp_apic_timer_interrupt+0xa6/0x270 [14160.259684] ? apic_timer_interrupt+0xf/0x20 [14160.261598] [14160.263203] ? native_safe_halt+0x2e/0x50 [14160.265850] kvm_wait+0xcd/0xe0 [14160.268173] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14160.271393] do_raw_spin_lock+0x177/0x190 [14160.273878] _raw_spin_lock+0x21/0x30 [14160.276048] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14160.279555] process_one_work+0x2c8/0x700 [14160.280999] worker_thread+0x296/0x6e0 [14160.283366] ? rescuer_thread+0x570/0x570 [14160.286229] kthread+0x1d1/0x200 [14160.288199] ? set_kthread_struct+0x70/0x70 [14160.290567] ret_from_fork+0x1f/0x30 [14164.056028] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [14164.065569] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14164.115394] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14164.123598] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14164.129205] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [14164.136595] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [14164.155280] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14164.158143] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [14164.161422] RDX: 0000000000007248 RSI: 0000000000000003 RDI: ffff96848811cae0 [14164.164589] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [14164.168019] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [14164.171435] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14164.174420] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [14164.179232] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14164.182200] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [14164.189012] Call Trace: [14164.190199] [14164.192439] ? show_regs.cold.9+0x22/0x2f [14164.196190] ? watchdog_timer_fn.cold.6+0x62/0x10a [14164.199187] ? __hrtimer_run_queues+0x1c2/0x4d0 [14164.204280] ? watchdog+0x40/0x40 [14164.208495] ? hrtimer_interrupt+0x128/0x2c0 [14164.211592] ? smp_apic_timer_interrupt+0xa6/0x270 [14164.215576] ? apic_timer_interrupt+0xf/0x20 [14164.220591] [14164.221562] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [14164.225389] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14164.228209] do_raw_spin_lock+0x177/0x190 [14164.231372] _raw_spin_lock+0x21/0x30 [14164.238665] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14164.242362] obd_mod_free+0x46/0x140 [ptlrpc] [14164.248093] mdc_commit_open+0x91/0xb0 [mdc] [14164.251282] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14164.255097] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14164.260081] after_reply+0xe2c/0x1780 [ptlrpc] [14164.261612] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14164.264071] ptlrpcd+0xa82/0xe40 [ptlrpc] [14164.267860] ? do_wait_intr+0xf0/0xf0 [14164.270452] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14164.273594] kthread+0x1d1/0x200 [14164.275370] ? set_kthread_struct+0x70/0x70 [14164.278416] ret_from_fork+0x1f/0x30 [14169.055359] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12210s! [14169.071214] Showing busy workqueues and worker pools: [14169.075240] workqueue events: flags=0x0 [14169.078387] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14169.100404] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [14169.110531] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [14169.117310] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [14169.123161] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14169.128561] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [14169.132844] workqueue events_long: flags=0x0 [14169.134827] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14169.140271] pending: lu_objects_flush [obdclass] [14169.144740] workqueue events_unbound: flags=0x2 [14169.147473] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [14169.150517] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [14169.156214] workqueue events_power_efficient: flags=0x80 [14169.160644] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14169.165121] pending: neigh_periodic_work [14169.168260] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14169.173202] pending: do_cache_clean, check_lifetime [14169.177206] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14169.181559] pending: neigh_periodic_work [14169.185474] workqueue writeback: flags=0x4a [14169.188173] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [14169.194758] pending: wb_workfn, wb_workfn [14169.201387] workqueue xprtiod: flags=0xa [14169.205650] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [14169.210662] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [14169.215300] workqueue ipv6_addrconf: flags=0x40008 [14169.221692] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14169.231532] pending: addrconf_verify_work [14169.236573] workqueue knbd0-recv: flags=0x1a [14169.238687] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14169.242497] in-flight: 657:recv_work [14169.244637] workqueue ptlrpc_pinger: flags=0x2 [14169.246718] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14169.249056] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [14169.251559] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12210s workers=4 idle: 20831 12642 28859 [14169.256320] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14159s workers=2 idle: 659 [14180.060138] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [14180.068413] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14180.120125] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14180.131140] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14180.139651] RIP: 0010:native_safe_halt+0x2e/0x50 [14180.143582] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14180.162230] RSP: 0000:ffffaf9fc15234f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14180.170090] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [14180.172632] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [14180.179419] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [14180.187666] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [14180.192114] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [14180.196107] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [14180.200899] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14180.205675] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [14180.211608] Call Trace: [14180.212545] [14180.213283] ? show_regs.cold.9+0x22/0x2f [14180.219631] ? watchdog_timer_fn.cold.6+0x62/0x10a [14180.224868] ? __hrtimer_run_queues+0x1c2/0x4d0 [14180.229378] ? watchdog+0x40/0x40 [14180.233560] ? hrtimer_interrupt+0x128/0x2c0 [14180.236784] ? smp_apic_timer_interrupt+0xa6/0x270 [14180.239605] ? apic_timer_interrupt+0xf/0x20 [14180.245262] [14180.247505] ? native_safe_halt+0x2e/0x50 [14180.248956] kvm_wait+0xcd/0xe0 [14180.250909] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14180.261158] do_raw_spin_lock+0x177/0x190 [14180.266268] _raw_spin_lock+0x21/0x30 [14180.271330] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14180.276465] ? do_raw_spin_unlock+0x75/0x190 [14180.284439] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14180.288661] ? do_wait_intr+0xf0/0xf0 [14180.293948] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14180.301407] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [14180.308893] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14180.318754] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14180.327324] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14180.331128] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [14180.335183] mdc_intent_lock+0x353/0x7e0 [mdc] [14180.338795] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14180.341643] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14180.347433] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14180.354320] lmv_intent_open+0x338/0x1150 [lmv] [14180.361399] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14180.363740] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14180.367349] lmv_intent_lock+0x1dd/0x4c0 [lmv] [14180.370615] ll_intent_lock+0xc5/0xaa0 [lustre] [14180.373541] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14180.380559] ll_lookup_it+0xe0e/0x2270 [lustre] [14180.388129] ? do_raw_spin_unlock+0x75/0x190 [14180.392124] ? _raw_spin_unlock+0x12/0x30 [14180.394376] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [14180.396235] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [14180.401376] ll_atomic_open+0x36a/0xd70 [lustre] [14180.403415] lookup_open+0x356/0xb40 [14180.406550] do_last+0x313/0xfc0 [14180.408935] ? nd_jump_root+0xe5/0x160 [14180.414381] ? path_init+0x437/0x520 [14180.418436] path_openat+0xf7/0x500 [14180.421350] ? do_raw_spin_unlock+0x75/0x190 [14180.426192] do_filp_open+0x99/0x140 [14180.430522] ? getname_flags+0x6e/0x330 [14180.433322] ? __check_object_size+0xff/0x256 [14180.437601] ? do_raw_spin_unlock+0x75/0x190 [14180.441084] ? _raw_spin_unlock+0x12/0x30 [14180.442441] do_sys_openat2+0x2b4/0x410 [14180.445623] do_sys_open+0x73/0xa0 [14180.447291] __x64_sys_openat+0x24/0x30 [14180.454333] do_syscall_64+0xc1/0x440 [14180.456624] entry_SYSCALL_64_after_hwframe+0x49/0xae [14180.463582] RIP: 0033:0x7ff54af45332 [14180.466160] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [14180.482769] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [14180.492273] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [14180.496342] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [14180.504305] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [14180.513706] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [14180.521317] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [14184.058066] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [14184.063836] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14184.092124] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14184.099080] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14184.104572] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14184.108354] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14184.120546] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14184.125682] RAX: 0000000000004789 RBX: ffff96848811cae0 RCX: 0000000000000001 [14184.129681] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [14184.134416] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [14184.138861] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [14184.143343] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [14184.147653] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [14184.152128] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14184.154772] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [14184.158329] Call Trace: [14184.159573] [14184.160579] ? show_regs.cold.9+0x22/0x2f [14184.164603] ? watchdog_timer_fn.cold.6+0x62/0x10a [14184.169645] ? __hrtimer_run_queues+0x1c2/0x4d0 [14184.172354] ? watchdog+0x40/0x40 [14184.174361] ? hrtimer_interrupt+0x128/0x2c0 [14184.176894] ? smp_apic_timer_interrupt+0xa6/0x270 [14184.181760] ? apic_timer_interrupt+0xf/0x20 [14184.185247] [14184.188897] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14184.197435] do_raw_spin_lock+0x177/0x190 [14184.201135] _raw_spin_lock+0x21/0x30 [14184.205611] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14184.218571] ? do_raw_spin_unlock+0x75/0x190 [14184.223446] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14184.230532] ? do_wait_intr+0xf0/0xf0 [14184.233847] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14184.235969] mdc_close+0x53b/0xd70 [mdc] [14184.239445] lmv_close+0x18d/0x490 [lmv] [14184.242238] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [14184.246357] ll_close_inode_openhandle+0x493/0xea0 [lustre] [14184.250830] ll_md_real_close+0xa5/0x2a0 [lustre] [14184.254184] ll_md_close+0x274/0x800 [lustre] [14184.262302] ll_file_release+0x128/0xc50 [lustre] [14184.272414] __fput+0x10d/0x440 [14184.273969] ____fput+0x12/0x20 [14184.277084] task_work_run+0xc6/0x110 [14184.279027] exit_to_usermode_loop+0x1dd/0x1f0 [14184.281365] do_syscall_64+0x3d6/0x440 [14184.284033] entry_SYSCALL_64_after_hwframe+0x49/0xae [14184.287974] RIP: 0033:0x7ff54af45cd8 [14184.290003] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [14184.298192] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [14184.304562] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [14184.308331] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [14184.312360] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [14184.315422] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [14184.318383] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14188.059043] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [14188.061751] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14188.104472] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14188.133843] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14188.152710] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14188.163643] RIP: 0010:native_safe_halt+0x2e/0x50 [14188.165014] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14188.180728] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14188.183687] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [14188.185712] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c2134154 [14188.188792] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [14188.191505] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [14188.193920] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [14188.196546] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [14188.199464] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14188.202261] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [14188.209824] Call Trace: [14188.210458] [14188.211558] ? show_regs.cold.9+0x22/0x2f [14188.214145] ? watchdog_timer_fn.cold.6+0x62/0x10a [14188.217311] ? __hrtimer_run_queues+0x1c2/0x4d0 [14188.219352] ? watchdog+0x40/0x40 [14188.220939] ? hrtimer_interrupt+0x128/0x2c0 [14188.222217] ? smp_apic_timer_interrupt+0xa6/0x270 [14188.223494] ? apic_timer_interrupt+0xf/0x20 [14188.230294] [14188.231777] ? native_safe_halt+0x2e/0x50 [14188.233455] kvm_wait+0xcd/0xe0 [14188.234894] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14188.236701] do_raw_spin_lock+0x177/0x190 [14188.238862] _raw_spin_lock+0x21/0x30 [14188.240211] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14188.244152] process_one_work+0x2c8/0x700 [14188.247189] worker_thread+0x296/0x6e0 [14188.249367] ? rescuer_thread+0x570/0x570 [14188.252078] kthread+0x1d1/0x200 [14188.253195] ? set_kthread_struct+0x70/0x70 [14188.255388] ret_from_fork+0x1f/0x30 [14196.061016] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [14196.066762] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14196.109224] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14196.115365] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14196.118422] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [14196.122311] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [14196.131242] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14196.143043] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [14196.155289] RDX: 0000000000002bb2 RSI: 0000000000000003 RDI: ffff96848811cae0 [14196.157453] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [14196.165486] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [14196.172065] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14196.178185] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [14196.190814] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14196.200076] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [14196.207854] Call Trace: [14196.209070] [14196.209895] ? show_regs.cold.9+0x22/0x2f [14196.212429] ? watchdog_timer_fn.cold.6+0x62/0x10a [14196.214339] ? __hrtimer_run_queues+0x1c2/0x4d0 [14196.220734] ? watchdog+0x40/0x40 [14196.228357] ? hrtimer_interrupt+0x128/0x2c0 [14196.229757] ? smp_apic_timer_interrupt+0xa6/0x270 [14196.232950] ? apic_timer_interrupt+0xf/0x20 [14196.234918] [14196.235626] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [14196.239499] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14196.242040] do_raw_spin_lock+0x177/0x190 [14196.243698] _raw_spin_lock+0x21/0x30 [14196.244861] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14196.248265] obd_mod_free+0x46/0x140 [ptlrpc] [14196.250731] mdc_commit_open+0x91/0xb0 [mdc] [14196.252964] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14196.255135] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14196.257047] after_reply+0xe2c/0x1780 [ptlrpc] [14196.260683] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14196.265327] ptlrpcd+0xa82/0xe40 [ptlrpc] [14196.269535] ? do_wait_intr+0xf0/0xf0 [14196.277384] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14196.286504] kthread+0x1d1/0x200 [14196.293915] ? set_kthread_struct+0x70/0x70 [14196.295432] ret_from_fork+0x1f/0x30 [14199.266933] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12240s! [14199.271408] Showing busy workqueues and worker pools: [14199.278512] workqueue events: flags=0x0 [14199.287657] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14199.297653] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [14199.310057] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [14199.317918] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [14199.334163] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14199.339399] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [14199.350729] workqueue events_long: flags=0x0 [14199.356390] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14199.360536] pending: lu_objects_flush [obdclass] [14199.363679] workqueue events_unbound: flags=0x2 [14199.367140] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [14199.376615] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [14199.386514] workqueue events_power_efficient: flags=0x80 [14199.392126] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14199.401877] pending: neigh_periodic_work [14199.407875] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14199.420795] pending: do_cache_clean, check_lifetime [14199.430674] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14199.433942] pending: neigh_periodic_work [14199.438666] workqueue writeback: flags=0x4a [14199.444819] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [14199.455403] pending: wb_workfn, wb_workfn [14199.458984] workqueue xprtiod: flags=0xa [14199.460752] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [14199.465210] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [14199.473943] workqueue ipv6_addrconf: flags=0x40008 [14199.479893] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14199.488502] pending: addrconf_verify_work [14199.492114] workqueue knbd0-recv: flags=0x1a [14199.496942] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14199.502408] in-flight: 657:recv_work [14199.504643] workqueue ptlrpc_pinger: flags=0x2 [14199.508309] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14199.514251] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [14199.521650] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12240s workers=4 idle: 20831 12642 28859 [14199.528889] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14189s workers=2 idle: 659 [14208.060591] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [14208.066827] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14208.110328] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14208.130121] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14208.136257] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14208.141373] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14208.156699] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14208.161969] RAX: 0000000000007f89 RBX: ffff96848811cae0 RCX: 0000000000000001 [14208.165764] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [14208.169960] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [14208.176272] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [14208.180814] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [14208.185381] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [14208.193177] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14208.196388] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [14208.200667] Call Trace: [14208.202507] [14208.203674] ? show_regs.cold.9+0x22/0x2f [14208.206690] ? watchdog_timer_fn.cold.6+0x62/0x10a [14208.209310] ? __hrtimer_run_queues+0x1c2/0x4d0 [14208.211886] ? watchdog+0x40/0x40 [14208.213674] ? hrtimer_interrupt+0x128/0x2c0 [14208.215845] ? smp_apic_timer_interrupt+0xa6/0x270 [14208.219304] ? apic_timer_interrupt+0xf/0x20 [14208.222312] [14208.223287] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14208.227271] do_raw_spin_lock+0x177/0x190 [14208.229851] _raw_spin_lock+0x21/0x30 [14208.232732] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14208.235814] ? do_raw_spin_unlock+0x75/0x190 [14208.238336] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14208.241741] ? do_wait_intr+0xf0/0xf0 [14208.243901] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14208.247259] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [14208.250353] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14208.254483] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14208.257832] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14208.261595] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [14208.264612] mdc_intent_lock+0x353/0x7e0 [mdc] [14208.267290] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14208.272199] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14208.279749] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14208.284984] lmv_intent_open+0x338/0x1150 [lmv] [14208.288644] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14208.292781] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14208.296210] lmv_intent_lock+0x1dd/0x4c0 [lmv] [14208.298922] ll_intent_lock+0xc5/0xaa0 [lustre] [14208.302815] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14208.309735] ll_lookup_it+0xe0e/0x2270 [lustre] [14208.314295] ? do_raw_spin_unlock+0x75/0x190 [14208.316013] ? _raw_spin_unlock+0x12/0x30 [14208.318686] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [14208.321371] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [14208.323900] ll_atomic_open+0x36a/0xd70 [lustre] [14208.326772] lookup_open+0x356/0xb40 [14208.328968] do_last+0x313/0xfc0 [14208.331674] ? nd_jump_root+0xe5/0x160 [14208.335763] ? path_init+0x437/0x520 [14208.343893] path_openat+0xf7/0x500 [14208.345919] ? do_raw_spin_unlock+0x75/0x190 [14208.350824] do_filp_open+0x99/0x140 [14208.354708] ? getname_flags+0x6e/0x330 [14208.358925] ? __check_object_size+0xff/0x256 [14208.364335] ? do_raw_spin_unlock+0x75/0x190 [14208.367168] ? _raw_spin_unlock+0x12/0x30 [14208.372205] do_sys_openat2+0x2b4/0x410 [14208.377169] do_sys_open+0x73/0xa0 [14208.380882] __x64_sys_openat+0x24/0x30 [14208.383160] do_syscall_64+0xc1/0x440 [14208.385209] entry_SYSCALL_64_after_hwframe+0x49/0xae [14208.388972] RIP: 0033:0x7ff54af45332 [14208.393440] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [14208.408823] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [14208.414255] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [14208.419429] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [14208.425327] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [14208.430385] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [14208.435494] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [14212.058041] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [14212.061880] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14212.086256] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14212.092162] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14212.097172] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14212.101028] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14212.112759] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14212.117071] RAX: 00000000000036b0 RBX: ffff96848811cae0 RCX: 0000000000000001 [14212.121257] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [14212.125216] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [14212.130175] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [14212.134216] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [14212.137782] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [14212.142701] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14212.146344] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [14212.150042] Call Trace: [14212.151031] [14212.152002] ? show_regs.cold.9+0x22/0x2f [14212.153426] ? watchdog_timer_fn.cold.6+0x62/0x10a [14212.155786] ? __hrtimer_run_queues+0x1c2/0x4d0 [14212.158045] ? watchdog+0x40/0x40 [14212.159234] ? hrtimer_interrupt+0x128/0x2c0 [14212.161318] ? smp_apic_timer_interrupt+0xa6/0x270 [14212.163909] ? apic_timer_interrupt+0xf/0x20 [14212.166237] [14212.166995] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14212.169650] do_raw_spin_lock+0x177/0x190 [14212.171711] _raw_spin_lock+0x21/0x30 [14212.173428] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14212.175387] ? do_raw_spin_unlock+0x75/0x190 [14212.177415] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14212.179915] ? do_wait_intr+0xf0/0xf0 [14212.181632] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14212.183710] mdc_close+0x53b/0xd70 [mdc] [14212.184951] lmv_close+0x18d/0x490 [lmv] [14212.186646] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [14212.188921] ll_close_inode_openhandle+0x493/0xea0 [lustre] [14212.191360] ll_md_real_close+0xa5/0x2a0 [lustre] [14212.193489] ll_md_close+0x274/0x800 [lustre] [14212.195315] ll_file_release+0x128/0xc50 [lustre] [14212.197530] __fput+0x10d/0x440 [14212.198788] ____fput+0x12/0x20 [14212.200041] task_work_run+0xc6/0x110 [14212.201826] exit_to_usermode_loop+0x1dd/0x1f0 [14212.203907] do_syscall_64+0x3d6/0x440 [14212.205153] entry_SYSCALL_64_after_hwframe+0x49/0xae [14212.207193] RIP: 0033:0x7ff54af45cd8 [14212.208752] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [14212.216580] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [14212.219151] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [14212.222031] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [14212.225190] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [14212.228322] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [14212.231657] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14216.066861] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [14216.075323] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14216.095607] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14216.100438] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14216.115292] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14216.123983] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14216.126028] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14216.148262] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14216.157679] RAX: 0000000000002a8f RBX: ffff96848811cae0 RCX: 0000000000000001 [14216.164797] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [14216.167591] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [14216.178599] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [14216.185700] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [14216.196652] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [14216.204305] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14216.208186] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [14216.212389] Call Trace: [14216.213232] [14216.214136] ? show_regs.cold.9+0x22/0x2f [14216.215709] ? watchdog_timer_fn.cold.6+0x62/0x10a [14216.217722] ? __hrtimer_run_queues+0x1c2/0x4d0 [14216.221894] ? watchdog+0x40/0x40 [14216.223597] ? hrtimer_interrupt+0x128/0x2c0 [14216.226071] ? smp_apic_timer_interrupt+0xa6/0x270 [14216.231480] ? apic_timer_interrupt+0xf/0x20 [14216.237013] [14216.244996] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14216.254209] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14216.263724] do_raw_spin_lock+0x177/0x190 [14216.272828] _raw_spin_lock+0x21/0x30 [14216.277994] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14216.292674] process_one_work+0x2c8/0x700 [14216.296840] worker_thread+0x296/0x6e0 [14216.302335] ? rescuer_thread+0x570/0x570 [14216.303776] kthread+0x1d1/0x200 [14216.307245] ? set_kthread_struct+0x70/0x70 [14216.316511] ret_from_fork+0x1f/0x30 [14224.060771] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [14224.072567] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14224.117982] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14224.129955] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14224.139089] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [14224.151641] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [14224.176919] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14224.181791] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [14224.189012] RDX: 0000000000007b96 RSI: 0000000000000003 RDI: ffff96848811cae0 [14224.196015] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [14224.202255] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [14224.205995] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14224.208867] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [14224.221567] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14224.228566] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [14224.236723] Call Trace: [14224.237793] [14224.240733] ? show_regs.cold.9+0x22/0x2f [14224.243717] ? watchdog_timer_fn.cold.6+0x62/0x10a [14224.245999] ? __hrtimer_run_queues+0x1c2/0x4d0 [14224.248643] ? watchdog+0x40/0x40 [14224.251970] ? hrtimer_interrupt+0x128/0x2c0 [14224.253751] ? smp_apic_timer_interrupt+0xa6/0x270 [14224.262852] ? apic_timer_interrupt+0xf/0x20 [14224.265861] [14224.271272] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [14224.282952] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14224.291150] do_raw_spin_lock+0x177/0x190 [14224.304927] _raw_spin_lock+0x21/0x30 [14224.306326] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14224.320063] obd_mod_free+0x46/0x140 [ptlrpc] [14224.325158] mdc_commit_open+0x91/0xb0 [mdc] [14224.327628] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14224.331148] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14224.336868] after_reply+0xe2c/0x1780 [ptlrpc] [14224.343059] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14224.345133] ptlrpcd+0xa82/0xe40 [ptlrpc] [14224.350733] ? do_wait_intr+0xf0/0xf0 [14224.353839] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14224.357322] kthread+0x1d1/0x200 [14224.360895] ? set_kthread_struct+0x70/0x70 [14224.369057] ret_from_fork+0x1f/0x30 [14229.983050] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12271s! [14229.986972] Showing busy workqueues and worker pools: [14229.989593] workqueue events: flags=0x0 [14230.003110] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14230.008730] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [14230.016059] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [14230.018896] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [14230.035798] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14230.043129] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [14230.049829] workqueue events_long: flags=0x0 [14230.054853] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14230.060000] pending: lu_objects_flush [obdclass] [14230.066746] workqueue events_unbound: flags=0x2 [14230.069657] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [14230.074905] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [14230.081809] workqueue events_power_efficient: flags=0x80 [14230.086460] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14230.091135] pending: neigh_periodic_work [14230.094139] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14230.098012] pending: do_cache_clean, check_lifetime [14230.103040] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14230.107974] pending: neigh_periodic_work [14230.110282] workqueue writeback: flags=0x4a [14230.111954] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [14230.117181] pending: wb_workfn, wb_workfn [14230.120930] workqueue xprtiod: flags=0xa [14230.123859] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [14230.129998] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [14230.136552] workqueue ipv6_addrconf: flags=0x40008 [14230.138827] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14230.145138] pending: addrconf_verify_work [14230.147157] workqueue knbd0-recv: flags=0x1a [14230.153184] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14230.157485] in-flight: 657:recv_work [14230.160610] workqueue ptlrpc_pinger: flags=0x2 [14230.161850] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14230.164182] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [14230.166527] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12271s workers=4 idle: 20831 12642 28859 [14230.168956] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14220s workers=2 idle: 659 [14236.060030] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [14236.062907] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14236.095570] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14236.100725] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14236.103926] RIP: 0010:native_safe_halt+0x2e/0x50 [14236.107596] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14236.144795] RSP: 0000:ffffaf9fc15234f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14236.151850] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [14236.156145] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c21b4154 [14236.159959] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [14236.165840] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [14236.168819] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [14236.176755] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [14236.180626] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14236.182998] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [14236.186925] Call Trace: [14236.188925] [14236.191786] ? show_regs.cold.9+0x22/0x2f [14236.195291] ? watchdog_timer_fn.cold.6+0x62/0x10a [14236.199699] ? __hrtimer_run_queues+0x1c2/0x4d0 [14236.202840] ? watchdog+0x40/0x40 [14236.205636] ? hrtimer_interrupt+0x128/0x2c0 [14236.207627] ? smp_apic_timer_interrupt+0xa6/0x270 [14236.210629] ? apic_timer_interrupt+0xf/0x20 [14236.212711] [14236.213847] ? native_safe_halt+0x2e/0x50 [14236.217396] kvm_wait+0xcd/0xe0 [14236.219862] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14236.221949] do_raw_spin_lock+0x177/0x190 [14236.224830] _raw_spin_lock+0x21/0x30 [14236.236582] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14236.239743] ? do_raw_spin_unlock+0x75/0x190 [14236.241931] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14236.245653] ? do_wait_intr+0xf0/0xf0 [14236.249192] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14236.251992] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [14236.254824] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14236.267805] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14236.276091] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14236.284919] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [14236.291923] mdc_intent_lock+0x353/0x7e0 [mdc] [14236.298875] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14236.300812] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14236.315367] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14236.321575] lmv_intent_open+0x338/0x1150 [lmv] [14236.325342] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14236.333706] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14236.339398] lmv_intent_lock+0x1dd/0x4c0 [lmv] [14236.341652] ll_intent_lock+0xc5/0xaa0 [lustre] [14236.347844] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14236.353358] ll_lookup_it+0xe0e/0x2270 [lustre] [14236.359850] ? do_raw_spin_unlock+0x75/0x190 [14236.362809] ? _raw_spin_unlock+0x12/0x30 [14236.365443] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [14236.369720] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [14236.377077] ll_atomic_open+0x36a/0xd70 [lustre] [14236.380580] lookup_open+0x356/0xb40 [14236.382785] do_last+0x313/0xfc0 [14236.386164] ? nd_jump_root+0xe5/0x160 [14236.390481] ? path_init+0x437/0x520 [14236.393783] path_openat+0xf7/0x500 [14236.395602] ? do_raw_spin_unlock+0x75/0x190 [14236.399770] do_filp_open+0x99/0x140 [14236.400873] ? getname_flags+0x6e/0x330 [14236.403833] ? __check_object_size+0xff/0x256 [14236.409904] ? do_raw_spin_unlock+0x75/0x190 [14236.415021] ? _raw_spin_unlock+0x12/0x30 [14236.417638] do_sys_openat2+0x2b4/0x410 [14236.419962] do_sys_open+0x73/0xa0 [14236.423886] __x64_sys_openat+0x24/0x30 [14236.431885] do_syscall_64+0xc1/0x440 [14236.435244] entry_SYSCALL_64_after_hwframe+0x49/0xae [14236.436789] RIP: 0033:0x7ff54af45332 [14236.440935] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [14236.467783] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [14236.470845] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [14236.474297] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [14236.477029] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [14236.488074] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [14236.494926] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [14240.058070] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [14240.060832] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14240.109515] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14240.120230] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14240.128500] RIP: 0010:native_safe_halt+0x2e/0x50 [14240.133074] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14240.147806] RSP: 0018:ffffaf9fc1543ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14240.166570] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [14240.175944] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c20b4154 [14240.177923] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [14240.187914] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [14240.193741] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [14240.197028] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [14240.205085] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14240.211722] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [14240.222329] Call Trace: [14240.225390] [14240.226894] ? show_regs.cold.9+0x22/0x2f [14240.234098] ? watchdog_timer_fn.cold.6+0x62/0x10a [14240.235543] ? __hrtimer_run_queues+0x1c2/0x4d0 [14240.236920] ? watchdog+0x40/0x40 [14240.246569] ? hrtimer_interrupt+0x128/0x2c0 [14240.251515] ? smp_apic_timer_interrupt+0xa6/0x270 [14240.254573] ? apic_timer_interrupt+0xf/0x20 [14240.260775] [14240.261709] ? native_safe_halt+0x2e/0x50 [14240.266219] kvm_wait+0xcd/0xe0 [14240.267762] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14240.270047] do_raw_spin_lock+0x177/0x190 [14240.277663] _raw_spin_lock+0x21/0x30 [14240.281266] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14240.287999] ? do_raw_spin_unlock+0x75/0x190 [14240.291370] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14240.296371] ? do_wait_intr+0xf0/0xf0 [14240.297611] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14240.299815] mdc_close+0x53b/0xd70 [mdc] [14240.303394] lmv_close+0x18d/0x490 [lmv] [14240.305417] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [14240.307463] ll_close_inode_openhandle+0x493/0xea0 [lustre] [14240.309055] ll_md_real_close+0xa5/0x2a0 [lustre] [14240.311053] ll_md_close+0x274/0x800 [lustre] [14240.313504] ll_file_release+0x128/0xc50 [lustre] [14240.316025] __fput+0x10d/0x440 [14240.317350] ____fput+0x12/0x20 [14240.318945] task_work_run+0xc6/0x110 [14240.320647] exit_to_usermode_loop+0x1dd/0x1f0 [14240.322665] do_syscall_64+0x3d6/0x440 [14240.323959] entry_SYSCALL_64_after_hwframe+0x49/0xae [14240.326564] RIP: 0033:0x7ff54af45cd8 [14240.328179] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [14240.334417] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [14240.337389] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [14240.340782] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [14240.343756] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [14240.345681] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [14240.348033] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14244.059508] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [14244.071807] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14244.118532] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14244.131607] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14244.133926] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14244.144350] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14244.151729] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14244.166400] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14244.169197] RAX: 0000000000004483 RBX: ffff96848811cae0 RCX: 0000000000000001 [14244.170947] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [14244.177340] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [14244.183418] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [14244.189690] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [14244.199199] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [14244.203648] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14244.206857] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [14244.212379] Call Trace: [14244.213450] [14244.218700] ? show_regs.cold.9+0x22/0x2f [14244.221422] ? watchdog_timer_fn.cold.6+0x62/0x10a [14244.222801] ? __hrtimer_run_queues+0x1c2/0x4d0 [14244.224367] ? watchdog+0x40/0x40 [14244.226333] ? hrtimer_interrupt+0x128/0x2c0 [14244.228627] ? smp_apic_timer_interrupt+0xa6/0x270 [14244.231079] ? apic_timer_interrupt+0xf/0x20 [14244.232852] [14244.233954] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14244.239201] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14244.240462] do_raw_spin_lock+0x177/0x190 [14244.252672] _raw_spin_lock+0x21/0x30 [14244.254920] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14244.258635] process_one_work+0x2c8/0x700 [14244.260721] worker_thread+0x296/0x6e0 [14244.261608] ? rescuer_thread+0x570/0x570 [14244.262633] kthread+0x1d1/0x200 [14244.270631] ? set_kthread_struct+0x70/0x70 [14244.271895] ret_from_fork+0x1f/0x30 [14254.172199] rcu: INFO: rcu_sched self-detected stall on CPU [14254.174422] rcu: 0-...!: (6557954 ticks this GP) idle=98a/1/0x4000000000000002 softirq=125154/125154 fqs=959 [14254.177949] (t=12300332 jiffies g=253585 q=6864) [14254.179582] rcu: rcu_sched kthread starved for 12295292 jiffies! g253585 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [14254.183851] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [14254.187288] rcu: RCU grace-period kthread stack dump: [14254.188710] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [14254.192046] Call Trace: [14254.192762] __schedule+0x351/0xcb0 [14254.193963] schedule+0xc0/0x180 [14254.195272] schedule_timeout+0xb4/0x190 [14254.196591] ? do_raw_spin_unlock+0x75/0x190 [14254.198216] ? __next_timer_interrupt+0x160/0x160 [14254.199681] ? prepare_to_swait_event+0xf3/0x140 [14254.201045] rcu_gp_kthread+0x8dd/0x15b0 [14254.202661] ? force_qs_rnp+0x2c0/0x2c0 [14254.204268] kthread+0x1d1/0x200 [14254.205522] ? set_kthread_struct+0x70/0x70 [14254.207339] ret_from_fork+0x1f/0x30 [14254.208728] rcu: Stack dump where RCU GP kthread last ran: [14254.210794] Sending NMI from CPU 0 to CPUs 2: [14254.212815] NMI backtrace for cpu 2 skipped: idling at native_safe_halt+0x2e/0x50 [14254.215112] NMI backtrace for cpu 0 [14254.217289] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14254.221221] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14254.224728] Call Trace: [14254.225614] [14254.226441] ? dump_stack+0xbb/0x10e [14254.227670] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [14254.229330] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [14254.230936] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [14254.232830] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [14254.234449] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [14254.236952] ? rcu_dump_cpu_stacks+0x11b/0x171 [14254.238816] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [14254.241781] ? update_process_times+0x6d/0xc0 [14254.244858] ? tick_sched_handle.isra.18+0x27/0x90 [14254.246657] ? tick_sched_timer+0x53/0xb0 [14254.248526] ? __hrtimer_run_queues+0x1c2/0x4d0 [14254.250502] ? tick_sched_handle.isra.18+0x90/0x90 [14254.253809] ? hrtimer_interrupt+0x128/0x2c0 [14254.257914] ? smp_apic_timer_interrupt+0xa6/0x270 [14254.260053] ? apic_timer_interrupt+0xf/0x20 [14254.262291] [14254.263805] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [14254.266543] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14254.269372] ? do_raw_spin_lock+0x177/0x190 [14254.271552] ? _raw_spin_lock+0x21/0x30 [14254.273314] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14254.275623] ? obd_mod_free+0x46/0x140 [ptlrpc] [14254.277907] ? mdc_commit_open+0x91/0xb0 [mdc] [14254.279922] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14254.283060] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14254.286341] ? after_reply+0xe2c/0x1780 [ptlrpc] [14254.288735] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14254.291503] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [14254.293712] ? do_wait_intr+0xf0/0xf0 [14254.295398] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14254.297824] ? kthread+0x1d1/0x200 [14254.299423] ? set_kthread_struct+0x70/0x70 [14254.301134] ? ret_from_fork+0x1f/0x30 [14260.191042] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12301s! [14260.196610] Showing busy workqueues and worker pools: [14260.199605] workqueue events: flags=0x0 [14260.201615] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14260.205292] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [14260.209899] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [14260.213251] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [14260.218691] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14260.221739] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [14260.224848] workqueue events_long: flags=0x0 [14260.227018] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14260.231239] pending: lu_objects_flush [obdclass] [14260.233628] workqueue events_unbound: flags=0x2 [14260.235016] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [14260.237788] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [14260.243565] workqueue events_power_efficient: flags=0x80 [14260.245776] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14260.249256] pending: neigh_periodic_work [14260.251509] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14260.255319] pending: do_cache_clean, check_lifetime [14260.258111] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14260.261720] pending: neigh_periodic_work [14260.263378] workqueue writeback: flags=0x4a [14260.265281] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [14260.268097] pending: wb_workfn, wb_workfn [14260.270411] workqueue xprtiod: flags=0xa [14260.273858] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [14260.278399] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [14260.288299] workqueue ipv6_addrconf: flags=0x40008 [14260.291324] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14260.294485] pending: addrconf_verify_work [14260.296627] workqueue knbd0-recv: flags=0x1a [14260.300071] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14260.309266] in-flight: 657:recv_work [14260.314251] workqueue ptlrpc_pinger: flags=0x2 [14260.316527] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14260.320256] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [14260.324494] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12301s workers=4 idle: 20831 12642 28859 [14260.329014] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14250s workers=2 idle: 659 [14264.061289] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [14264.064640] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14264.096981] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14264.103082] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14264.113759] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14264.118373] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14264.133025] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [14264.138458] RAX: 0000000000006b86 RBX: ffff96848811cae0 RCX: 0000000000000001 [14264.149328] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [14264.152487] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [14264.154911] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [14264.157315] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [14264.160285] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [14264.163746] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14264.166248] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [14264.169339] Call Trace: [14264.170339] [14264.171312] ? show_regs.cold.9+0x22/0x2f [14264.174149] ? watchdog_timer_fn.cold.6+0x62/0x10a [14264.176170] ? __hrtimer_run_queues+0x1c2/0x4d0 [14264.178653] ? watchdog+0x40/0x40 [14264.180341] ? hrtimer_interrupt+0x128/0x2c0 [14264.182002] ? smp_apic_timer_interrupt+0xa6/0x270 [14264.183835] ? apic_timer_interrupt+0xf/0x20 [14264.185753] [14264.186630] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14264.188628] do_raw_spin_lock+0x177/0x190 [14264.190133] _raw_spin_lock+0x21/0x30 [14264.191490] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14264.193626] ? do_raw_spin_unlock+0x75/0x190 [14264.195265] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14264.197313] ? do_wait_intr+0xf0/0xf0 [14264.198402] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14264.200438] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [14264.202498] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14264.204954] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14264.207349] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14264.209410] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [14264.211079] mdc_intent_lock+0x353/0x7e0 [mdc] [14264.212674] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14264.214828] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14264.218148] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14264.220239] lmv_intent_open+0x338/0x1150 [lmv] [14264.227044] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14264.229700] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14264.232151] lmv_intent_lock+0x1dd/0x4c0 [lmv] [14264.234163] ll_intent_lock+0xc5/0xaa0 [lustre] [14264.236218] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14264.238668] ll_lookup_it+0xe0e/0x2270 [lustre] [14264.240517] ? do_raw_spin_unlock+0x75/0x190 [14264.243675] ? _raw_spin_unlock+0x12/0x30 [14264.246029] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [14264.248672] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [14264.250949] ll_atomic_open+0x36a/0xd70 [lustre] [14264.253649] lookup_open+0x356/0xb40 [14264.255262] do_last+0x313/0xfc0 [14264.256715] ? nd_jump_root+0xe5/0x160 [14264.258572] ? path_init+0x437/0x520 [14264.260149] path_openat+0xf7/0x500 [14264.261658] ? do_raw_spin_unlock+0x75/0x190 [14264.263496] do_filp_open+0x99/0x140 [14264.265774] ? getname_flags+0x6e/0x330 [14264.268363] ? __check_object_size+0xff/0x256 [14264.269826] ? do_raw_spin_unlock+0x75/0x190 [14264.271969] ? _raw_spin_unlock+0x12/0x30 [14264.274050] do_sys_openat2+0x2b4/0x410 [14264.280321] do_sys_open+0x73/0xa0 [14264.284200] __x64_sys_openat+0x24/0x30 [14264.287241] do_syscall_64+0xc1/0x440 [14264.289527] entry_SYSCALL_64_after_hwframe+0x49/0xae [14264.292583] RIP: 0033:0x7ff54af45332 [14264.294290] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [14264.302739] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [14264.306327] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [14264.309019] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [14264.311568] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [14264.314637] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [14264.317826] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [14268.058283] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [14268.061156] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14268.107384] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14268.114522] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14268.121207] RIP: 0010:native_safe_halt+0x2e/0x50 [14268.126354] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14268.145752] RSP: 0018:ffffaf9fc1543ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14268.150310] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [14268.152783] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [14268.155974] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [14268.158696] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [14268.161737] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [14268.164428] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [14268.167498] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14268.170219] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [14268.172261] Call Trace: [14268.173500] [14268.174337] ? show_regs.cold.9+0x22/0x2f [14268.176299] ? watchdog_timer_fn.cold.6+0x62/0x10a [14268.178563] ? __hrtimer_run_queues+0x1c2/0x4d0 [14268.180487] ? watchdog+0x40/0x40 [14268.182079] ? hrtimer_interrupt+0x128/0x2c0 [14268.184016] ? smp_apic_timer_interrupt+0xa6/0x270 [14268.187212] ? apic_timer_interrupt+0xf/0x20 [14268.189200] [14268.189917] ? native_safe_halt+0x2e/0x50 [14268.192121] kvm_wait+0xcd/0xe0 [14268.193334] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14268.195531] do_raw_spin_lock+0x177/0x190 [14268.197293] _raw_spin_lock+0x21/0x30 [14268.199012] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14268.201779] ? do_raw_spin_unlock+0x75/0x190 [14268.203784] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14268.206415] ? do_wait_intr+0xf0/0xf0 [14268.209374] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14268.212462] mdc_close+0x53b/0xd70 [mdc] [14268.214310] lmv_close+0x18d/0x490 [lmv] [14268.215639] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [14268.218181] ll_close_inode_openhandle+0x493/0xea0 [lustre] [14268.223912] ll_md_real_close+0xa5/0x2a0 [lustre] [14268.228654] ll_md_close+0x274/0x800 [lustre] [14268.233410] ll_file_release+0x128/0xc50 [lustre] [14268.236774] __fput+0x10d/0x440 [14268.238544] ____fput+0x12/0x20 [14268.240122] task_work_run+0xc6/0x110 [14268.242601] exit_to_usermode_loop+0x1dd/0x1f0 [14268.248797] do_syscall_64+0x3d6/0x440 [14268.250431] entry_SYSCALL_64_after_hwframe+0x49/0xae [14268.252450] RIP: 0033:0x7ff54af45cd8 [14268.253806] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [14268.262242] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [14268.263949] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [14268.266292] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [14268.269380] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [14268.272800] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [14268.276722] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14280.059019] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [14280.064289] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14280.106570] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14280.112311] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14280.121192] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14280.125562] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14280.137834] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14280.155879] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14280.159483] RAX: 000000000000784b RBX: ffff96848811cae0 RCX: 0000000000000001 [14280.170246] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [14280.181003] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [14280.192786] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [14280.203223] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [14280.215130] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [14280.220082] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14280.228232] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [14280.234944] Call Trace: [14280.240919] [14280.242177] ? show_regs.cold.9+0x22/0x2f [14280.247099] ? watchdog_timer_fn.cold.6+0x62/0x10a [14280.248805] ? __hrtimer_run_queues+0x1c2/0x4d0 [14280.258748] ? watchdog+0x40/0x40 [14280.259999] ? hrtimer_interrupt+0x128/0x2c0 [14280.261479] ? smp_apic_timer_interrupt+0xa6/0x270 [14280.268820] ? apic_timer_interrupt+0xf/0x20 [14280.275206] [14280.275804] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14280.282811] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14280.291753] do_raw_spin_lock+0x177/0x190 [14280.300420] _raw_spin_lock+0x21/0x30 [14280.301915] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14280.306764] process_one_work+0x2c8/0x700 [14280.309554] worker_thread+0x296/0x6e0 [14280.311559] ? rescuer_thread+0x570/0x570 [14280.314730] kthread+0x1d1/0x200 [14280.322081] ? set_kthread_struct+0x70/0x70 [14280.330543] ret_from_fork+0x1f/0x30 [14288.061663] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [14288.063976] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14288.093233] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14288.098596] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14288.106751] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [14288.112519] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [14288.122479] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14288.126076] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [14288.136379] RDX: 000000000000670f RSI: 0000000000000003 RDI: ffff96848811cae0 [14288.144115] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [14288.148872] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [14288.159994] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14288.166253] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [14288.174522] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14288.179265] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [14288.184315] Call Trace: [14288.185431] [14288.186269] ? show_regs.cold.9+0x22/0x2f [14288.191064] ? watchdog_timer_fn.cold.6+0x62/0x10a [14288.193827] ? __hrtimer_run_queues+0x1c2/0x4d0 [14288.196432] ? watchdog+0x40/0x40 [14288.200143] ? hrtimer_interrupt+0x128/0x2c0 [14288.209961] ? smp_apic_timer_interrupt+0xa6/0x270 [14288.215726] ? apic_timer_interrupt+0xf/0x20 [14288.217910] [14288.218930] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [14288.225205] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14288.229305] do_raw_spin_lock+0x177/0x190 [14288.232114] _raw_spin_lock+0x21/0x30 [14288.234931] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14288.237635] obd_mod_free+0x46/0x140 [ptlrpc] [14288.239978] mdc_commit_open+0x91/0xb0 [mdc] [14288.245462] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14288.249265] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14288.252593] after_reply+0xe2c/0x1780 [ptlrpc] [14288.257253] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14288.262646] ptlrpcd+0xa82/0xe40 [ptlrpc] [14288.264536] ? do_wait_intr+0xf0/0xf0 [14288.268196] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14288.273645] kthread+0x1d1/0x200 [14288.275031] ? set_kthread_struct+0x70/0x70 [14288.280189] ret_from_fork+0x1f/0x30 [14290.400528] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12331s! [14290.408503] Showing busy workqueues and worker pools: [14290.417652] workqueue events: flags=0x0 [14290.424191] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14290.434567] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [14290.441171] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [14290.447990] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [14290.465274] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14290.475288] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [14290.484242] workqueue events_long: flags=0x0 [14290.487279] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14290.491234] pending: lu_objects_flush [obdclass] [14290.494253] workqueue events_unbound: flags=0x2 [14290.496936] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [14290.500192] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [14290.505283] workqueue events_power_efficient: flags=0x80 [14290.517568] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14290.529194] pending: neigh_periodic_work [14290.535804] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14290.538174] pending: do_cache_clean, check_lifetime [14290.548406] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14290.554441] pending: neigh_periodic_work [14290.558790] workqueue writeback: flags=0x4a [14290.561096] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [14290.568506] pending: wb_workfn, wb_workfn [14290.577254] workqueue xprtiod: flags=0xa [14290.579511] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [14290.584542] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [14290.591550] workqueue ipv6_addrconf: flags=0x40008 [14290.597304] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14290.603787] pending: addrconf_verify_work [14290.605237] workqueue knbd0-recv: flags=0x1a [14290.608288] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14290.613874] in-flight: 657:recv_work [14290.618939] workqueue ptlrpc_pinger: flags=0x2 [14290.621752] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14290.632188] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [14290.638242] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12331s workers=4 idle: 20831 12642 28859 [14290.645408] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14280s workers=2 idle: 659 [14292.060014] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [14292.066001] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14292.144101] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14292.164452] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14292.174444] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14292.183870] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14292.205254] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14292.214194] RAX: 0000000000007591 RBX: ffff96848811cae0 RCX: 0000000000000001 [14292.222899] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [14292.231888] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [14292.240179] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [14292.245198] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [14292.250021] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [14292.255982] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14292.262366] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [14292.270385] Call Trace: [14292.277010] [14292.278316] ? show_regs.cold.9+0x22/0x2f [14292.283078] ? watchdog_timer_fn.cold.6+0x62/0x10a [14292.287825] ? __hrtimer_run_queues+0x1c2/0x4d0 [14292.295248] ? watchdog+0x40/0x40 [14292.297422] ? hrtimer_interrupt+0x128/0x2c0 [14292.306396] ? smp_apic_timer_interrupt+0xa6/0x270 [14292.311544] ? apic_timer_interrupt+0xf/0x20 [14292.320315] [14292.322174] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14292.328058] do_raw_spin_lock+0x177/0x190 [14292.331426] _raw_spin_lock+0x21/0x30 [14292.335046] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14292.341364] ? do_raw_spin_unlock+0x75/0x190 [14292.344062] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14292.347429] ? do_wait_intr+0xf0/0xf0 [14292.349550] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14292.353130] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [14292.357367] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14292.360922] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14292.364802] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14292.369425] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [14292.372630] mdc_intent_lock+0x353/0x7e0 [mdc] [14292.376246] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14292.380507] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14292.385274] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14292.389460] lmv_intent_open+0x338/0x1150 [lmv] [14292.392281] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14292.396277] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14292.400356] lmv_intent_lock+0x1dd/0x4c0 [lmv] [14292.406123] ll_intent_lock+0xc5/0xaa0 [lustre] [14292.411042] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14292.416899] ll_lookup_it+0xe0e/0x2270 [lustre] [14292.420381] ? do_raw_spin_unlock+0x75/0x190 [14292.424265] ? _raw_spin_unlock+0x12/0x30 [14292.430082] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [14292.433298] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [14292.436402] ll_atomic_open+0x36a/0xd70 [lustre] [14292.440166] lookup_open+0x356/0xb40 [14292.443382] do_last+0x313/0xfc0 [14292.446443] ? nd_jump_root+0xe5/0x160 [14292.451376] ? path_init+0x437/0x520 [14292.453535] path_openat+0xf7/0x500 [14292.457526] ? do_raw_spin_unlock+0x75/0x190 [14292.463144] do_filp_open+0x99/0x140 [14292.466168] ? getname_flags+0x6e/0x330 [14292.469414] ? __check_object_size+0xff/0x256 [14292.476576] ? do_raw_spin_unlock+0x75/0x190 [14292.481218] ? _raw_spin_unlock+0x12/0x30 [14292.484259] do_sys_openat2+0x2b4/0x410 [14292.487969] do_sys_open+0x73/0xa0 [14292.490375] __x64_sys_openat+0x24/0x30 [14292.493178] do_syscall_64+0xc1/0x440 [14292.498389] entry_SYSCALL_64_after_hwframe+0x49/0xae [14292.502485] RIP: 0033:0x7ff54af45332 [14292.504162] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [14292.517607] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [14292.521747] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [14292.526822] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [14292.531627] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [14292.537138] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [14292.542488] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [14296.060245] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [14296.065765] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14296.106388] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14296.115117] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14296.122521] RIP: 0010:native_safe_halt+0x2e/0x50 [14296.125324] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14296.142315] RSP: 0018:ffffaf9fc1543ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14296.147888] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [14296.154496] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [14296.163423] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [14296.169729] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [14296.180918] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [14296.185484] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [14296.192505] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14296.196532] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [14296.211144] Call Trace: [14296.212386] [14296.213131] ? show_regs.cold.9+0x22/0x2f [14296.214417] ? watchdog_timer_fn.cold.6+0x62/0x10a [14296.216422] ? __hrtimer_run_queues+0x1c2/0x4d0 [14296.218183] ? watchdog+0x40/0x40 [14296.219184] ? hrtimer_interrupt+0x128/0x2c0 [14296.222413] ? smp_apic_timer_interrupt+0xa6/0x270 [14296.224568] ? apic_timer_interrupt+0xf/0x20 [14296.225681] [14296.226256] ? native_safe_halt+0x2e/0x50 [14296.227980] kvm_wait+0xcd/0xe0 [14296.229182] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14296.231057] do_raw_spin_lock+0x177/0x190 [14296.233006] _raw_spin_lock+0x21/0x30 [14296.236461] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14296.239400] ? do_raw_spin_unlock+0x75/0x190 [14296.240831] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14296.242789] ? do_wait_intr+0xf0/0xf0 [14296.243687] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14296.244957] mdc_close+0x53b/0xd70 [mdc] [14296.248126] lmv_close+0x18d/0x490 [lmv] [14296.249394] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [14296.253449] ll_close_inode_openhandle+0x493/0xea0 [lustre] [14296.255986] ll_md_real_close+0xa5/0x2a0 [lustre] [14296.257543] ll_md_close+0x274/0x800 [lustre] [14296.260996] ll_file_release+0x128/0xc50 [lustre] [14296.264332] __fput+0x10d/0x440 [14296.266424] ____fput+0x12/0x20 [14296.267666] task_work_run+0xc6/0x110 [14296.268942] exit_to_usermode_loop+0x1dd/0x1f0 [14296.270546] do_syscall_64+0x3d6/0x440 [14296.271531] entry_SYSCALL_64_after_hwframe+0x49/0xae [14296.273498] RIP: 0033:0x7ff54af45cd8 [14296.274306] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [14296.280461] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [14296.283950] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [14296.286127] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [14296.292341] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [14296.298510] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [14296.304653] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14308.060809] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [14308.068191] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14308.093589] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14308.100033] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14308.107062] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14308.112776] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14308.120203] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14308.133690] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14308.137404] RAX: 0000000000007a11 RBX: ffff96848811cae0 RCX: 0000000000000001 [14308.141412] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [14308.145303] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [14308.148719] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [14308.152256] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [14308.155948] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [14308.159826] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14308.162707] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [14308.166575] Call Trace: [14308.167653] [14308.168703] ? show_regs.cold.9+0x22/0x2f [14308.170716] ? watchdog_timer_fn.cold.6+0x62/0x10a [14308.173574] ? __hrtimer_run_queues+0x1c2/0x4d0 [14308.176020] ? watchdog+0x40/0x40 [14308.177812] ? hrtimer_interrupt+0x128/0x2c0 [14308.180716] ? smp_apic_timer_interrupt+0xa6/0x270 [14308.186205] ? apic_timer_interrupt+0xf/0x20 [14308.192059] [14308.193819] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14308.200726] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14308.207519] do_raw_spin_lock+0x177/0x190 [14308.209349] _raw_spin_lock+0x21/0x30 [14308.216416] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14308.222197] process_one_work+0x2c8/0x700 [14308.226777] worker_thread+0x296/0x6e0 [14308.230776] ? rescuer_thread+0x570/0x570 [14308.234545] kthread+0x1d1/0x200 [14308.238797] ? set_kthread_struct+0x70/0x70 [14308.242313] ret_from_fork+0x1f/0x30 [14316.055788] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [14316.067540] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14316.103797] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14316.114308] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14316.119584] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [14316.128769] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [14316.148989] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14316.152922] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [14316.159754] RDX: 00000000000079ac RSI: 0000000000000003 RDI: ffff96848811cae0 [14316.165075] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [14316.171340] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [14316.177959] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14316.181996] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [14316.190523] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14316.196043] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [14316.201764] Call Trace: [14316.202740] [14316.203510] ? show_regs.cold.9+0x22/0x2f [14316.206815] ? watchdog_timer_fn.cold.6+0x62/0x10a [14316.213617] ? __hrtimer_run_queues+0x1c2/0x4d0 [14316.216567] ? watchdog+0x40/0x40 [14316.221540] ? hrtimer_interrupt+0x128/0x2c0 [14316.225604] ? smp_apic_timer_interrupt+0xa6/0x270 [14316.228696] ? apic_timer_interrupt+0xf/0x20 [14316.231111] [14316.234052] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [14316.239570] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14316.248695] do_raw_spin_lock+0x177/0x190 [14316.251002] _raw_spin_lock+0x21/0x30 [14316.254034] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14316.259060] obd_mod_free+0x46/0x140 [ptlrpc] [14316.262228] mdc_commit_open+0x91/0xb0 [mdc] [14316.265969] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14316.270990] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14316.275743] after_reply+0xe2c/0x1780 [ptlrpc] [14316.284272] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14316.290866] ptlrpcd+0xa82/0xe40 [ptlrpc] [14316.296852] ? do_wait_intr+0xf0/0xf0 [14316.299800] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14316.305072] kthread+0x1d1/0x200 [14316.308461] ? set_kthread_struct+0x70/0x70 [14316.310879] ret_from_fork+0x1f/0x30 [14320.060309] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [14320.064600] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14320.094148] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14320.098295] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14320.109828] RIP: 0010:native_safe_halt+0x2e/0x50 [14320.118165] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14320.140296] RSP: 0000:ffffaf9fc15234f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14320.143800] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [14320.154344] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [14320.156484] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [14320.164903] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [14320.172190] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [14320.175004] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [14320.179132] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14320.189282] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [14320.196813] Call Trace: [14320.203900] [14320.212312] ? show_regs.cold.9+0x22/0x2f [14320.216137] ? watchdog_timer_fn.cold.6+0x62/0x10a [14320.223214] ? __hrtimer_run_queues+0x1c2/0x4d0 [14320.228630] ? watchdog+0x40/0x40 [14320.231250] ? hrtimer_interrupt+0x128/0x2c0 [14320.232799] ? smp_apic_timer_interrupt+0xa6/0x270 [14320.236219] ? apic_timer_interrupt+0xf/0x20 [14320.238729] [14320.241661] ? native_safe_halt+0x2e/0x50 [14320.244259] kvm_wait+0xcd/0xe0 [14320.247642] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14320.250602] do_raw_spin_lock+0x177/0x190 [14320.252659] _raw_spin_lock+0x21/0x30 [14320.255794] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14320.259291] ? do_raw_spin_unlock+0x75/0x190 [14320.264126] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14320.267338] ? do_wait_intr+0xf0/0xf0 [14320.268741] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14320.274331] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [14320.278617] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14320.284571] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14320.287723] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14320.294567] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [14320.309022] mdc_intent_lock+0x353/0x7e0 [mdc] [14320.311258] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14320.316826] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14320.320741] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14320.324794] lmv_intent_open+0x338/0x1150 [lmv] [14320.334731] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14320.343324] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14320.354533] lmv_intent_lock+0x1dd/0x4c0 [lmv] [14320.363553] ll_intent_lock+0xc5/0xaa0 [lustre] [14320.367794] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14320.371207] ll_lookup_it+0xe0e/0x2270 [lustre] [14320.380254] ? do_raw_spin_unlock+0x75/0x190 [14320.387583] ? _raw_spin_unlock+0x12/0x30 [14320.397068] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [14320.399762] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [14320.408491] ll_atomic_open+0x36a/0xd70 [lustre] [14320.411882] lookup_open+0x356/0xb40 [14320.420643] do_last+0x313/0xfc0 [14320.421821] ? nd_jump_root+0xe5/0x160 [14320.427246] ? path_init+0x437/0x520 [14320.428920] path_openat+0xf7/0x500 [14320.431504] ? do_raw_spin_unlock+0x75/0x190 [14320.435447] do_filp_open+0x99/0x140 [14320.439895] ? getname_flags+0x6e/0x330 [14320.446651] ? __check_object_size+0xff/0x256 [14320.450863] ? do_raw_spin_unlock+0x75/0x190 [14320.456661] ? _raw_spin_unlock+0x12/0x30 [14320.460921] do_sys_openat2+0x2b4/0x410 [14320.466758] do_sys_open+0x73/0xa0 [14320.470171] __x64_sys_openat+0x24/0x30 [14320.473731] do_syscall_64+0xc1/0x440 [14320.476825] entry_SYSCALL_64_after_hwframe+0x49/0xae [14320.484294] RIP: 0033:0x7ff54af45332 [14320.489553] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [14320.506318] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [14320.512218] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [14320.520254] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [14320.526742] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [14320.531013] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [14320.534232] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [14321.119644] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12362s! [14321.126877] Showing busy workqueues and worker pools: [14321.131735] workqueue events: flags=0x0 [14321.133984] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14321.138374] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [14321.145787] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [14321.150738] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [14321.157666] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14321.162479] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [14321.166910] workqueue events_long: flags=0x0 [14321.169828] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14321.174923] pending: lu_objects_flush [obdclass] [14321.178905] workqueue events_unbound: flags=0x2 [14321.181982] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [14321.186038] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [14321.202732] workqueue events_power_efficient: flags=0x80 [14321.217057] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14321.222275] pending: neigh_periodic_work [14321.229274] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14321.240551] pending: do_cache_clean, check_lifetime [14321.246625] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14321.254896] pending: neigh_periodic_work [14321.259701] workqueue writeback: flags=0x4a [14321.263037] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [14321.270155] pending: wb_workfn, wb_workfn [14321.273292] workqueue xprtiod: flags=0xa [14321.277793] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [14321.288147] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [14321.303177] workqueue ipv6_addrconf: flags=0x40008 [14321.305181] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14321.317113] pending: addrconf_verify_work [14321.325207] workqueue knbd0-recv: flags=0x1a [14321.329711] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14321.337631] in-flight: 657:recv_work [14321.340200] workqueue ptlrpc_pinger: flags=0x2 [14321.348563] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14321.358633] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [14321.361126] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12362s workers=4 idle: 20831 12642 28859 [14321.371911] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14311s workers=2 idle: 659 [14324.058839] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [14324.078437] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14324.143654] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14324.163814] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14324.170783] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14324.172179] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14324.178939] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14324.181783] RAX: 000000000000594b RBX: ffff96848811cae0 RCX: 0000000000000001 [14324.184674] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [14324.189679] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [14324.192191] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [14324.206112] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [14324.212009] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [14324.218823] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14324.222837] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [14324.226415] Call Trace: [14324.227958] [14324.229681] ? show_regs.cold.9+0x22/0x2f [14324.232691] ? watchdog_timer_fn.cold.6+0x62/0x10a [14324.242101] ? __hrtimer_run_queues+0x1c2/0x4d0 [14324.244332] ? watchdog+0x40/0x40 [14324.250572] ? hrtimer_interrupt+0x128/0x2c0 [14324.254224] ? smp_apic_timer_interrupt+0xa6/0x270 [14324.257184] ? apic_timer_interrupt+0xf/0x20 [14324.259699] [14324.261130] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14324.265580] do_raw_spin_lock+0x177/0x190 [14324.268541] _raw_spin_lock+0x21/0x30 [14324.273166] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14324.276588] ? do_raw_spin_unlock+0x75/0x190 [14324.279935] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14324.283214] ? do_wait_intr+0xf0/0xf0 [14324.285676] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14324.292212] mdc_close+0x53b/0xd70 [mdc] [14324.294181] lmv_close+0x18d/0x490 [lmv] [14324.299188] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [14324.305665] ll_close_inode_openhandle+0x493/0xea0 [lustre] [14324.310161] ll_md_real_close+0xa5/0x2a0 [lustre] [14324.313368] ll_md_close+0x274/0x800 [lustre] [14324.321844] ll_file_release+0x128/0xc50 [lustre] [14324.325216] __fput+0x10d/0x440 [14324.332587] ____fput+0x12/0x20 [14324.334152] task_work_run+0xc6/0x110 [14324.337079] exit_to_usermode_loop+0x1dd/0x1f0 [14324.340160] do_syscall_64+0x3d6/0x440 [14324.342832] entry_SYSCALL_64_after_hwframe+0x49/0xae [14324.345324] RIP: 0033:0x7ff54af45cd8 [14324.348635] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [14324.366294] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [14324.372264] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [14324.376012] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [14324.386235] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [14324.397827] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [14324.417096] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14336.068867] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [14336.077999] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14336.161739] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14336.181719] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14336.204106] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14336.214934] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14336.228930] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14336.273803] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14336.276292] RAX: 0000000000004b83 RBX: ffff96848811cae0 RCX: 0000000000000001 [14336.301526] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [14336.306419] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [14336.326289] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [14336.345386] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [14336.352399] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [14336.369625] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14336.386523] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [14336.407027] Call Trace: [14336.408803] [14336.411786] ? show_regs.cold.9+0x22/0x2f [14336.418919] ? watchdog_timer_fn.cold.6+0x62/0x10a [14336.432824] ? __hrtimer_run_queues+0x1c2/0x4d0 [14336.444561] ? watchdog+0x40/0x40 [14336.456416] ? hrtimer_interrupt+0x128/0x2c0 [14336.457837] ? smp_apic_timer_interrupt+0xa6/0x270 [14336.479381] ? apic_timer_interrupt+0xf/0x20 [14336.490587] [14336.496688] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14336.510848] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14336.521766] do_raw_spin_lock+0x177/0x190 [14336.534562] _raw_spin_lock+0x21/0x30 [14336.543497] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14336.558510] process_one_work+0x2c8/0x700 [14336.565592] worker_thread+0x296/0x6e0 [14336.573408] ? rescuer_thread+0x570/0x570 [14336.577077] kthread+0x1d1/0x200 [14336.578285] ? set_kthread_struct+0x70/0x70 [14336.596880] ret_from_fork+0x1f/0x30 [14348.055020] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [14348.060620] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14348.066351] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [14348.109929] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14348.109947] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14348.109951] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [14348.109967] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [14348.109972] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14348.109978] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [14348.109981] RDX: 0000000000006249 RSI: 0000000000000003 RDI: ffff96848811cae0 [14348.109984] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [14348.109986] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [14348.109989] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14348.109992] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [14348.109996] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14348.109998] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [14348.110007] Call Trace: [14348.110013] [14348.110019] ? show_regs.cold.9+0x22/0x2f [14348.110030] ? watchdog_timer_fn.cold.6+0x62/0x10a [14348.110036] ? __hrtimer_run_queues+0x1c2/0x4d0 [14348.110041] ? watchdog+0x40/0x40 [14348.110045] ? hrtimer_interrupt+0x128/0x2c0 [14348.110050] ? smp_apic_timer_interrupt+0xa6/0x270 [14348.110058] ? apic_timer_interrupt+0xf/0x20 [14348.110062] [14348.110065] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [14348.110070] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14348.110075] do_raw_spin_lock+0x177/0x190 [14348.110080] _raw_spin_lock+0x21/0x30 [14348.110087] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14348.110573] obd_mod_free+0x46/0x140 [ptlrpc] [14348.110782] mdc_commit_open+0x91/0xb0 [mdc] [14348.110823] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14348.112221] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14348.119952] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14348.120033] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14348.120043] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14348.120046] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14348.120061] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14348.120065] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14348.120070] RAX: 0000000000007d8b RBX: ffff96848811cae0 RCX: 0000000000000001 [14348.120072] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [14348.120074] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [14348.120076] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [14348.120078] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [14348.120082] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [14348.120085] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14348.120088] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [14348.120096] Call Trace: [14348.120102] [14348.120107] ? show_regs.cold.9+0x22/0x2f [14348.120119] ? watchdog_timer_fn.cold.6+0x62/0x10a [14348.120126] ? __hrtimer_run_queues+0x1c2/0x4d0 [14348.120131] ? watchdog+0x40/0x40 [14348.120134] ? hrtimer_interrupt+0x128/0x2c0 [14348.120138] ? smp_apic_timer_interrupt+0xa6/0x270 [14348.120146] ? apic_timer_interrupt+0xf/0x20 [14348.120151] [14348.120153] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14348.120160] do_raw_spin_lock+0x177/0x190 [14348.120165] _raw_spin_lock+0x21/0x30 [14348.120170] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14348.121114] ? do_raw_spin_unlock+0x75/0x190 [14348.121122] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14348.121628] ? do_wait_intr+0xf0/0xf0 [14348.121638] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14348.121837] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [14348.122066] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14348.131228] after_reply+0xe2c/0x1780 [ptlrpc] [14348.131588] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14348.131801] ptlrpcd+0xa82/0xe40 [ptlrpc] [14348.132344] ? do_wait_intr+0xf0/0xf0 [14348.132357] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14348.132577] kthread+0x1d1/0x200 [14348.132582] ? set_kthread_struct+0x70/0x70 [14348.132586] ret_from_fork+0x1f/0x30 [14348.646824] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14348.651140] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14348.653132] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [14348.662556] mdc_intent_lock+0x353/0x7e0 [mdc] [14348.666995] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14348.672031] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14348.676965] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14348.679747] lmv_intent_open+0x338/0x1150 [lmv] [14348.681809] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14348.684123] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14348.686259] lmv_intent_lock+0x1dd/0x4c0 [lmv] [14348.687786] ll_intent_lock+0xc5/0xaa0 [lustre] [14348.695678] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14348.700756] ll_lookup_it+0xe0e/0x2270 [lustre] [14348.705921] ? do_raw_spin_unlock+0x75/0x190 [14348.707946] ? _raw_spin_unlock+0x12/0x30 [14348.709819] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [14348.712250] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [14348.714776] ll_atomic_open+0x36a/0xd70 [lustre] [14348.717279] lookup_open+0x356/0xb40 [14348.718839] do_last+0x313/0xfc0 [14348.720385] ? nd_jump_root+0xe5/0x160 [14348.722148] ? path_init+0x437/0x520 [14348.724375] path_openat+0xf7/0x500 [14348.726340] ? do_raw_spin_unlock+0x75/0x190 [14348.728209] do_filp_open+0x99/0x140 [14348.729763] ? getname_flags+0x6e/0x330 [14348.731400] ? __check_object_size+0xff/0x256 [14348.733634] ? do_raw_spin_unlock+0x75/0x190 [14348.735596] ? _raw_spin_unlock+0x12/0x30 [14348.737731] do_sys_openat2+0x2b4/0x410 [14348.739677] do_sys_open+0x73/0xa0 [14348.741307] __x64_sys_openat+0x24/0x30 [14348.742818] do_syscall_64+0xc1/0x440 [14348.748796] entry_SYSCALL_64_after_hwframe+0x49/0xae [14348.756147] RIP: 0033:0x7ff54af45332 [14348.759753] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [14348.771802] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [14348.775563] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [14348.780666] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [14348.789176] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [14348.799269] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [14348.804510] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [14351.841060] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12392s! [14351.861494] Showing busy workqueues and worker pools: [14351.865905] workqueue events: flags=0x0 [14351.868666] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14351.886997] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [14351.897094] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [14351.909185] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [14351.926851] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14351.938873] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [14351.948508] workqueue events_long: flags=0x0 [14351.950868] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14351.956146] pending: lu_objects_flush [obdclass] [14351.959296] workqueue events_unbound: flags=0x2 [14351.961736] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [14351.966644] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [14351.980050] workqueue events_power_efficient: flags=0x80 [14351.985032] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14351.988429] pending: neigh_periodic_work [14351.990897] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14351.993989] pending: do_cache_clean, check_lifetime [14351.996868] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14352.000375] pending: neigh_periodic_work [14352.003989] workqueue writeback: flags=0x4a [14352.005897] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [14352.008106] pending: wb_workfn, wb_workfn [14352.010018] workqueue xprtiod: flags=0xa [14352.011298] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [14352.018369] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [14352.021729] workqueue ipv6_addrconf: flags=0x40008 [14352.023986] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14352.026938] pending: addrconf_verify_work [14352.031907] workqueue knbd0-recv: flags=0x1a [14352.036270] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14352.040832] in-flight: 657:recv_work [14352.043070] workqueue ptlrpc_pinger: flags=0x2 [14352.044972] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14352.048785] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [14352.050610] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12393s workers=4 idle: 20831 12642 28859 [14352.055901] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14342s workers=2 idle: 659 [14352.058094] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [14352.070870] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14352.116520] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14352.128949] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14352.134547] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14352.138748] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14352.157555] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14352.164811] RAX: 0000000000007e1e RBX: ffff96848811cae0 RCX: 0000000000000001 [14352.174617] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [14352.178667] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [14352.182089] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [14352.185335] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [14352.189500] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [14352.196686] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14352.199479] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [14352.203453] Call Trace: [14352.204559] [14352.205367] ? show_regs.cold.9+0x22/0x2f [14352.207666] ? watchdog_timer_fn.cold.6+0x62/0x10a [14352.210490] ? __hrtimer_run_queues+0x1c2/0x4d0 [14352.217691] ? watchdog+0x40/0x40 [14352.221330] ? hrtimer_interrupt+0x128/0x2c0 [14352.225444] ? smp_apic_timer_interrupt+0xa6/0x270 [14352.227664] ? apic_timer_interrupt+0xf/0x20 [14352.230217] [14352.231041] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14352.235594] do_raw_spin_lock+0x177/0x190 [14352.244033] _raw_spin_lock+0x21/0x30 [14352.248841] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14352.253600] ? do_raw_spin_unlock+0x75/0x190 [14352.256023] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14352.259959] ? do_wait_intr+0xf0/0xf0 [14352.263850] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14352.269046] mdc_close+0x53b/0xd70 [mdc] [14352.271812] lmv_close+0x18d/0x490 [lmv] [14352.274918] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [14352.278649] ll_close_inode_openhandle+0x493/0xea0 [lustre] [14352.282846] ll_md_real_close+0xa5/0x2a0 [lustre] [14352.286838] ll_md_close+0x274/0x800 [lustre] [14352.289471] ll_file_release+0x128/0xc50 [lustre] [14352.292501] __fput+0x10d/0x440 [14352.293978] ____fput+0x12/0x20 [14352.295935] task_work_run+0xc6/0x110 [14352.301202] exit_to_usermode_loop+0x1dd/0x1f0 [14352.306940] do_syscall_64+0x3d6/0x440 [14352.308939] entry_SYSCALL_64_after_hwframe+0x49/0xae [14352.313442] RIP: 0033:0x7ff54af45cd8 [14352.315641] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [14352.330865] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [14352.337348] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [14352.342485] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [14352.350994] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [14352.357614] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [14352.362946] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14364.059266] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [14364.069311] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14364.110806] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14364.129564] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14364.148535] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14364.167476] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14364.173985] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14364.199272] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14364.203707] RAX: 0000000000007d41 RBX: ffff96848811cae0 RCX: 0000000000000001 [14364.209786] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [14364.216833] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [14364.226934] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [14364.234585] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [14364.239840] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [14364.249554] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14364.254009] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [14364.258825] Call Trace: [14364.261886] [14364.262626] ? show_regs.cold.9+0x22/0x2f [14364.264616] ? watchdog_timer_fn.cold.6+0x62/0x10a [14364.265999] ? __hrtimer_run_queues+0x1c2/0x4d0 [14364.267904] ? watchdog+0x40/0x40 [14364.269709] ? hrtimer_interrupt+0x128/0x2c0 [14364.274729] ? smp_apic_timer_interrupt+0xa6/0x270 [14364.279772] ? apic_timer_interrupt+0xf/0x20 [14364.282762] [14364.283813] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14364.289880] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14364.291644] do_raw_spin_lock+0x177/0x190 [14364.292875] _raw_spin_lock+0x21/0x30 [14364.307637] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14364.309610] process_one_work+0x2c8/0x700 [14364.315118] worker_thread+0x296/0x6e0 [14364.320682] ? rescuer_thread+0x570/0x570 [14364.326403] kthread+0x1d1/0x200 [14364.330323] ? set_kthread_struct+0x70/0x70 [14364.334864] ret_from_fork+0x1f/0x30 [14376.060930] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [14376.071486] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14376.136960] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14376.149962] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14376.164440] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14376.178642] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14376.222427] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14376.235408] RAX: 0000000000007c72 RBX: ffff96848811cae0 RCX: 0000000000000001 [14376.239921] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [14376.242920] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [14376.246166] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [14376.249273] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [14376.252410] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [14376.255972] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14376.258446] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [14376.261782] Call Trace: [14376.262792] [14376.263649] ? show_regs.cold.9+0x22/0x2f [14376.265440] ? watchdog_timer_fn.cold.6+0x62/0x10a [14376.267464] ? __hrtimer_run_queues+0x1c2/0x4d0 [14376.269189] ? watchdog+0x40/0x40 [14376.270555] ? hrtimer_interrupt+0x128/0x2c0 [14376.272391] ? smp_apic_timer_interrupt+0xa6/0x270 [14376.274418] ? apic_timer_interrupt+0xf/0x20 [14376.276382] [14376.277322] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14376.279909] do_raw_spin_lock+0x177/0x190 [14376.281632] _raw_spin_lock+0x21/0x30 [14376.283281] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14376.285617] ? do_raw_spin_unlock+0x75/0x190 [14376.287431] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14376.289667] ? do_wait_intr+0xf0/0xf0 [14376.291242] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14376.293562] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [14376.296001] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14376.298573] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14376.300702] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14376.303323] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [14376.305364] mdc_intent_lock+0x353/0x7e0 [mdc] [14376.307365] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14376.310110] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14376.312563] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14376.314948] lmv_intent_open+0x338/0x1150 [lmv] [14376.317025] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14376.319344] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14376.321299] lmv_intent_lock+0x1dd/0x4c0 [lmv] [14376.323271] ll_intent_lock+0xc5/0xaa0 [lustre] [14376.325181] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14376.327415] ll_lookup_it+0xe0e/0x2270 [lustre] [14376.329387] ? do_raw_spin_unlock+0x75/0x190 [14376.331228] ? _raw_spin_unlock+0x12/0x30 [14376.333014] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [14376.335188] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [14376.337277] ll_atomic_open+0x36a/0xd70 [lustre] [14376.339328] lookup_open+0x356/0xb40 [14376.340764] do_last+0x313/0xfc0 [14376.342007] ? nd_jump_root+0xe5/0x160 [14376.343418] ? path_init+0x437/0x520 [14376.344898] path_openat+0xf7/0x500 [14376.346210] ? do_raw_spin_unlock+0x75/0x190 [14376.347693] do_filp_open+0x99/0x140 [14376.349190] ? getname_flags+0x6e/0x330 [14376.350677] ? __check_object_size+0xff/0x256 [14376.352258] ? do_raw_spin_unlock+0x75/0x190 [14376.353941] ? _raw_spin_unlock+0x12/0x30 [14376.355428] do_sys_openat2+0x2b4/0x410 [14376.356993] do_sys_open+0x73/0xa0 [14376.358275] __x64_sys_openat+0x24/0x30 [14376.359439] do_syscall_64+0xc1/0x440 [14376.360917] entry_SYSCALL_64_after_hwframe+0x49/0xae [14376.362298] RIP: 0033:0x7ff54af45332 [14376.363297] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [14376.370057] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [14376.372914] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [14376.375370] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [14376.378020] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [14376.380592] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [14376.383347] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [14380.055032] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [14380.059180] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14380.075073] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [14380.079056] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14380.083467] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14380.083548] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14380.083555] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14380.083558] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14380.083573] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14380.083578] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14380.083582] RAX: 0000000000007d8a RBX: ffff96848811cae0 RCX: 0000000000000001 [14380.083585] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [14380.083586] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [14380.083589] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [14380.083592] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [14380.083595] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [14380.083598] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14380.083601] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [14380.083609] Call Trace: [14380.083615] [14380.083621] ? show_regs.cold.9+0x22/0x2f [14380.083631] ? watchdog_timer_fn.cold.6+0x62/0x10a [14380.083638] ? __hrtimer_run_queues+0x1c2/0x4d0 [14380.083642] ? watchdog+0x40/0x40 [14380.083646] ? hrtimer_interrupt+0x128/0x2c0 [14380.083649] ? smp_apic_timer_interrupt+0xa6/0x270 [14380.083659] ? apic_timer_interrupt+0xf/0x20 [14380.083663] [14380.083666] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14380.083673] do_raw_spin_lock+0x177/0x190 [14380.083678] _raw_spin_lock+0x21/0x30 [14380.083683] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14380.090615] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14380.090626] RIP: 0010:native_safe_halt+0x2e/0x50 [14380.090641] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14380.090645] RSP: 0000:ffffaf9fc1b97bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14380.090651] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [14380.090654] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff96848811cae0 [14380.090656] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [14380.090658] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [14380.090661] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14380.090664] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [14380.090668] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14380.090670] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [14380.090679] Call Trace: [14380.090685] [14380.090691] ? show_regs.cold.9+0x22/0x2f [14380.090702] ? watchdog_timer_fn.cold.6+0x62/0x10a [14380.090709] ? __hrtimer_run_queues+0x1c2/0x4d0 [14380.090714] ? watchdog+0x40/0x40 [14380.090718] ? hrtimer_interrupt+0x128/0x2c0 [14380.090723] ? smp_apic_timer_interrupt+0xa6/0x270 [14380.090730] ? apic_timer_interrupt+0xf/0x20 [14380.090735] [14380.090737] ? native_safe_halt+0x2e/0x50 [14380.090742] kvm_wait+0xcd/0xe0 [14380.090751] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14380.090758] do_raw_spin_lock+0x177/0x190 [14380.090763] _raw_spin_lock+0x21/0x30 [14380.090770] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14380.139037] ? do_raw_spin_unlock+0x75/0x190 [14380.143040] obd_mod_free+0x46/0x140 [ptlrpc] [14380.143364] mdc_commit_open+0x91/0xb0 [mdc] [14380.143408] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14380.143613] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14380.143885] after_reply+0xe2c/0x1780 [ptlrpc] [14380.144100] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14380.144302] ptlrpcd+0xa82/0xe40 [ptlrpc] [14380.144512] ? do_wait_intr+0xf0/0xf0 [14380.144524] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14380.144890] kthread+0x1d1/0x200 [14380.144897] ? set_kthread_struct+0x70/0x70 [14380.144901] ret_from_fork+0x1f/0x30 [14380.626761] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14380.629494] ? do_wait_intr+0xf0/0xf0 [14380.631566] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14380.639584] mdc_close+0x53b/0xd70 [mdc] [14380.643028] lmv_close+0x18d/0x490 [lmv] [14380.646651] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [14380.651601] ll_close_inode_openhandle+0x493/0xea0 [lustre] [14380.660610] ll_md_real_close+0xa5/0x2a0 [lustre] [14380.670422] ll_md_close+0x274/0x800 [lustre] [14380.675502] ll_file_release+0x128/0xc50 [lustre] [14380.686255] __fput+0x10d/0x440 [14380.687781] ____fput+0x12/0x20 [14380.692369] task_work_run+0xc6/0x110 [14380.696505] exit_to_usermode_loop+0x1dd/0x1f0 [14380.699637] do_syscall_64+0x3d6/0x440 [14380.704174] entry_SYSCALL_64_after_hwframe+0x49/0xae [14380.706107] RIP: 0033:0x7ff54af45cd8 [14380.711513] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [14380.745241] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [14380.759660] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [14380.762270] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [14380.777546] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [14380.782312] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [14380.790268] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14382.559048] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12423s! [14382.563712] Showing busy workqueues and worker pools: [14382.566760] workqueue events: flags=0x0 [14382.568759] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14382.572511] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [14382.576536] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [14382.581176] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [14382.587415] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14382.590138] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [14382.592494] workqueue events_long: flags=0x0 [14382.595394] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14382.597671] pending: lu_objects_flush [obdclass] [14382.599530] workqueue events_unbound: flags=0x2 [14382.601083] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [14382.603890] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [14382.614481] workqueue events_power_efficient: flags=0x80 [14382.617733] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14382.621938] pending: neigh_periodic_work [14382.623767] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14382.626803] pending: do_cache_clean, check_lifetime [14382.663116] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14382.669824] pending: neigh_periodic_work [14382.677320] workqueue writeback: flags=0x4a [14382.688215] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [14382.701388] pending: wb_workfn, wb_workfn [14382.703073] workqueue xprtiod: flags=0xa [14382.712310] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [14382.726748] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [14382.729593] workqueue ipv6_addrconf: flags=0x40008 [14382.751043] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14382.764155] pending: addrconf_verify_work [14382.775954] workqueue knbd0-recv: flags=0x1a [14382.777940] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14382.784105] in-flight: 657:recv_work [14382.787675] workqueue ptlrpc_pinger: flags=0x2 [14382.789228] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14382.795133] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [14382.798474] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12423s workers=4 idle: 20831 12642 28859 [14382.806223] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14373s workers=2 idle: 659 [14392.059327] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [14392.068235] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14392.135333] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14392.162349] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14392.180598] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14392.203174] RIP: 0010:native_safe_halt+0x2e/0x50 [14392.209263] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14392.243928] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14392.247290] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [14392.253358] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c2134154 [14392.262258] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [14392.266779] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [14392.273375] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [14392.285987] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [14392.299390] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14392.303092] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [14392.314931] Call Trace: [14392.316021] [14392.319271] ? show_regs.cold.9+0x22/0x2f [14392.327599] ? watchdog_timer_fn.cold.6+0x62/0x10a [14392.331474] ? __hrtimer_run_queues+0x1c2/0x4d0 [14392.348033] ? watchdog+0x40/0x40 [14392.351489] ? hrtimer_interrupt+0x128/0x2c0 [14392.361539] ? smp_apic_timer_interrupt+0xa6/0x270 [14392.367523] ? apic_timer_interrupt+0xf/0x20 [14392.370403] [14392.374358] ? native_safe_halt+0x2e/0x50 [14392.380548] kvm_wait+0xcd/0xe0 [14392.403570] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14392.411612] do_raw_spin_lock+0x177/0x190 [14392.414589] _raw_spin_lock+0x21/0x30 [14392.417285] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14392.419509] process_one_work+0x2c8/0x700 [14392.421095] worker_thread+0x296/0x6e0 [14392.424111] ? rescuer_thread+0x570/0x570 [14392.425749] kthread+0x1d1/0x200 [14392.428084] ? set_kthread_struct+0x70/0x70 [14392.431205] ret_from_fork+0x1f/0x30 [14404.071876] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [14404.074113] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14404.097498] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14404.106284] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14404.110194] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14404.112000] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14404.121665] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14404.128573] RAX: 0000000000007707 RBX: ffff96848811cae0 RCX: 0000000000000001 [14404.132662] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [14404.135985] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [14404.139159] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [14404.141271] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [14404.144852] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [14404.148482] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14404.150221] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [14404.160470] Call Trace: [14404.161595] [14404.163526] ? show_regs.cold.9+0x22/0x2f [14404.168989] ? watchdog_timer_fn.cold.6+0x62/0x10a [14404.171473] ? __hrtimer_run_queues+0x1c2/0x4d0 [14404.174000] ? watchdog+0x40/0x40 [14404.176221] ? hrtimer_interrupt+0x128/0x2c0 [14404.178766] ? smp_apic_timer_interrupt+0xa6/0x270 [14404.181436] ? apic_timer_interrupt+0xf/0x20 [14404.184444] [14404.186508] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14404.189098] do_raw_spin_lock+0x177/0x190 [14404.190107] _raw_spin_lock+0x21/0x30 [14404.194416] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14404.199044] ? do_raw_spin_unlock+0x75/0x190 [14404.201277] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14404.204066] ? do_wait_intr+0xf0/0xf0 [14404.209324] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14404.211193] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [14404.218075] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14404.226654] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14404.230162] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14404.233061] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [14404.237125] mdc_intent_lock+0x353/0x7e0 [mdc] [14404.241231] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14404.243952] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14404.247763] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14404.250640] lmv_intent_open+0x338/0x1150 [lmv] [14404.253486] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14404.255929] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14404.260299] lmv_intent_lock+0x1dd/0x4c0 [lmv] [14404.273904] ll_intent_lock+0xc5/0xaa0 [lustre] [14404.276034] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14404.279233] ll_lookup_it+0xe0e/0x2270 [lustre] [14404.283871] ? do_raw_spin_unlock+0x75/0x190 [14404.287161] ? _raw_spin_unlock+0x12/0x30 [14404.289327] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [14404.295308] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [14404.299442] ll_atomic_open+0x36a/0xd70 [lustre] [14404.304397] lookup_open+0x356/0xb40 [14404.306810] do_last+0x313/0xfc0 [14404.307922] ? nd_jump_root+0xe5/0x160 [14404.311909] ? path_init+0x437/0x520 [14404.313768] path_openat+0xf7/0x500 [14404.317805] ? do_raw_spin_unlock+0x75/0x190 [14404.321450] do_filp_open+0x99/0x140 [14404.323922] ? getname_flags+0x6e/0x330 [14404.326090] ? __check_object_size+0xff/0x256 [14404.329276] ? do_raw_spin_unlock+0x75/0x190 [14404.331483] ? _raw_spin_unlock+0x12/0x30 [14404.333246] do_sys_openat2+0x2b4/0x410 [14404.337098] do_sys_open+0x73/0xa0 [14404.338834] __x64_sys_openat+0x24/0x30 [14404.342475] do_syscall_64+0xc1/0x440 [14404.345764] entry_SYSCALL_64_after_hwframe+0x49/0xae [14404.348398] RIP: 0033:0x7ff54af45332 [14404.351978] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [14404.361769] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [14404.368586] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [14404.375467] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [14404.379943] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [14404.388598] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [14404.394523] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [14408.055087] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [14408.058022] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [14408.059914] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14408.059990] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14408.059998] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14408.060001] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [14408.060016] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [14408.060021] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14408.060026] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [14408.060029] RDX: 000000000000207d RSI: 0000000000000003 RDI: ffff96848811cae0 [14408.060032] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [14408.060034] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [14408.060037] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14408.060040] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [14408.060043] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14408.060046] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [14408.060053] Call Trace: [14408.060059] [14408.060065] ? show_regs.cold.9+0x22/0x2f [14408.060073] ? watchdog_timer_fn.cold.6+0x62/0x10a [14408.060080] ? __hrtimer_run_queues+0x1c2/0x4d0 [14408.060084] ? watchdog+0x40/0x40 [14408.060088] ? hrtimer_interrupt+0x128/0x2c0 [14408.060093] ? smp_apic_timer_interrupt+0xa6/0x270 [14408.060101] ? apic_timer_interrupt+0xf/0x20 [14408.060105] [14408.060109] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [14408.060113] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14408.060117] do_raw_spin_lock+0x177/0x190 [14408.060122] _raw_spin_lock+0x21/0x30 [14408.060128] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14408.060394] obd_mod_free+0x46/0x140 [ptlrpc] [14408.060597] mdc_commit_open+0x91/0xb0 [mdc] [14408.060633] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14408.060849] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14408.071628] Modules linked in: lustre(O) [14408.125681] after_reply+0xe2c/0x1780 [ptlrpc] [14408.125981] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14408.126936] ptlrpcd+0xa82/0xe40 [ptlrpc] [14408.154442] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14408.154744] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14408.154753] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14408.154756] RIP: 0010:native_safe_halt+0x2e/0x50 [14408.154771] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14408.154776] RSP: 0018:ffffaf9fc1543ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14408.154780] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [14408.154783] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [14408.154786] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [14408.154788] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [14408.154791] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [14408.154794] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [14408.154797] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14408.154800] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [14408.154807] Call Trace: [14408.154813] [14408.154820] ? show_regs.cold.9+0x22/0x2f [14408.154831] ? watchdog_timer_fn.cold.6+0x62/0x10a [14408.154837] ? __hrtimer_run_queues+0x1c2/0x4d0 [14408.154841] ? watchdog+0x40/0x40 [14408.154845] ? hrtimer_interrupt+0x128/0x2c0 [14408.154849] ? smp_apic_timer_interrupt+0xa6/0x270 [14408.154856] ? apic_timer_interrupt+0xf/0x20 [14408.154861] [14408.154863] ? native_safe_halt+0x2e/0x50 [14408.154867] kvm_wait+0xcd/0xe0 [14408.154876] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14408.154886] do_raw_spin_lock+0x177/0x190 [14408.154891] _raw_spin_lock+0x21/0x30 [14408.154895] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14408.155188] ? do_raw_spin_unlock+0x75/0x190 [14408.155194] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14408.155387] ? do_wait_intr+0xf0/0xf0 [14408.155395] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14408.164406] ? do_wait_intr+0xf0/0xf0 [14408.164426] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14408.182156] mdc_close+0x53b/0xd70 [mdc] [14408.182206] lmv_close+0x18d/0x490 [lmv] [14408.182236] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [14408.182417] ll_close_inode_openhandle+0x493/0xea0 [lustre] [14408.184355] ll_md_real_close+0xa5/0x2a0 [lustre] [14408.184437] ll_md_close+0x274/0x800 [lustre] [14408.184639] ll_file_release+0x128/0xc50 [lustre] [14408.184732] __fput+0x10d/0x440 [14408.184742] ____fput+0x12/0x20 [14408.184746] task_work_run+0xc6/0x110 [14408.184755] exit_to_usermode_loop+0x1dd/0x1f0 [14408.184763] do_syscall_64+0x3d6/0x440 [14408.184767] entry_SYSCALL_64_after_hwframe+0x49/0xae [14408.184778] RIP: 0033:0x7ff54af45cd8 [14408.184786] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [14408.184790] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [14408.184795] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [14408.184797] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [14408.184799] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [14408.184801] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [14408.184803] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14408.581261] kthread+0x1d1/0x200 [14408.582380] ? set_kthread_struct+0x70/0x70 [14408.583871] ret_from_fork+0x1f/0x30 [14413.279381] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12454s! [14413.297707] Showing busy workqueues and worker pools: [14413.309410] workqueue events: flags=0x0 [14413.315745] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14413.322861] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [14413.343775] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [14413.349950] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [14413.370780] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14413.379642] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [14413.386842] workqueue events_long: flags=0x0 [14413.390415] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14413.402512] pending: lu_objects_flush [obdclass] [14413.426444] workqueue events_unbound: flags=0x2 [14413.439220] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [14413.452192] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [14413.457299] workqueue events_power_efficient: flags=0x80 [14413.481496] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14413.499198] pending: neigh_periodic_work [14413.507732] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14413.522892] pending: do_cache_clean, check_lifetime [14413.528631] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14413.551167] pending: neigh_periodic_work [14413.559359] workqueue writeback: flags=0x4a [14413.561020] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [14413.569374] pending: wb_workfn, wb_workfn [14413.584756] workqueue xprtiod: flags=0xa [14413.589753] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [14413.606813] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [14413.625491] workqueue ipv6_addrconf: flags=0x40008 [14413.631667] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14413.636217] pending: addrconf_verify_work [14413.642116] workqueue knbd0-recv: flags=0x1a [14413.651783] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14413.659877] in-flight: 657:recv_work [14413.663374] workqueue ptlrpc_pinger: flags=0x2 [14413.666302] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14413.676414] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [14413.695591] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12454s workers=4 idle: 20831 12642 28859 [14413.715655] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14403s workers=2 idle: 659 [14420.059036] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [14420.061809] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14420.134737] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14420.161698] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14420.182732] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14420.191791] RIP: 0010:native_safe_halt+0x2e/0x50 [14420.203356] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14420.245033] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14420.258818] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [14420.272991] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c2134154 [14420.280870] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [14420.289201] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [14420.308869] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [14420.316261] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [14420.336655] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14420.352985] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [14420.380189] Call Trace: [14420.393919] [14420.400868] ? show_regs.cold.9+0x22/0x2f [14420.409943] ? watchdog_timer_fn.cold.6+0x62/0x10a [14420.411524] ? __hrtimer_run_queues+0x1c2/0x4d0 [14420.425179] ? watchdog+0x40/0x40 [14420.437645] ? hrtimer_interrupt+0x128/0x2c0 [14420.439062] ? smp_apic_timer_interrupt+0xa6/0x270 [14420.440515] ? apic_timer_interrupt+0xf/0x20 [14420.458321] [14420.463735] ? native_safe_halt+0x2e/0x50 [14420.468118] kvm_wait+0xcd/0xe0 [14420.478807] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14420.484708] do_raw_spin_lock+0x177/0x190 [14420.488884] _raw_spin_lock+0x21/0x30 [14420.491571] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14420.504379] process_one_work+0x2c8/0x700 [14420.505753] worker_thread+0x296/0x6e0 [14420.516800] ? rescuer_thread+0x570/0x570 [14420.521725] kthread+0x1d1/0x200 [14420.539226] ? set_kthread_struct+0x70/0x70 [14420.541303] ret_from_fork+0x1f/0x30 [14432.060067] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [14432.064448] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14432.129625] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14432.138810] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14432.145720] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14432.154817] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14432.173371] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14432.178710] RAX: 0000000000004987 RBX: ffff96848811cae0 RCX: 0000000000000001 [14432.185602] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [14432.191663] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [14432.199732] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [14432.215547] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [14432.226199] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [14432.238030] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14432.240029] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [14432.257777] Call Trace: [14432.258740] [14432.273500] ? show_regs.cold.9+0x22/0x2f [14432.274911] ? watchdog_timer_fn.cold.6+0x62/0x10a [14432.284325] ? __hrtimer_run_queues+0x1c2/0x4d0 [14432.287132] ? watchdog+0x40/0x40 [14432.288819] ? hrtimer_interrupt+0x128/0x2c0 [14432.291507] ? smp_apic_timer_interrupt+0xa6/0x270 [14432.298986] ? apic_timer_interrupt+0xf/0x20 [14432.302782] [14432.308312] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14432.313108] do_raw_spin_lock+0x177/0x190 [14432.317533] _raw_spin_lock+0x21/0x30 [14432.320162] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14432.330152] ? do_raw_spin_unlock+0x75/0x190 [14432.331638] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14432.334831] ? do_wait_intr+0xf0/0xf0 [14432.336123] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14432.341182] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [14432.351335] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14432.354951] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14432.364316] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14432.372805] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [14432.383705] mdc_intent_lock+0x353/0x7e0 [mdc] [14432.394914] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14432.406237] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14432.417170] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14432.418794] lmv_intent_open+0x338/0x1150 [lmv] [14432.429894] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14432.442199] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14432.448128] lmv_intent_lock+0x1dd/0x4c0 [lmv] [14432.456741] ll_intent_lock+0xc5/0xaa0 [lustre] [14432.463818] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14432.469036] ll_lookup_it+0xe0e/0x2270 [lustre] [14432.472508] ? do_raw_spin_unlock+0x75/0x190 [14432.482378] ? _raw_spin_unlock+0x12/0x30 [14432.483504] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [14432.487643] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [14432.496036] ll_atomic_open+0x36a/0xd70 [lustre] [14432.504000] lookup_open+0x356/0xb40 [14432.508001] do_last+0x313/0xfc0 [14432.513831] ? nd_jump_root+0xe5/0x160 [14432.522630] ? path_init+0x437/0x520 [14432.527922] path_openat+0xf7/0x500 [14432.531968] ? do_raw_spin_unlock+0x75/0x190 [14432.541795] do_filp_open+0x99/0x140 [14432.542996] ? getname_flags+0x6e/0x330 [14432.544272] ? __check_object_size+0xff/0x256 [14432.553171] ? do_raw_spin_unlock+0x75/0x190 [14432.562160] ? _raw_spin_unlock+0x12/0x30 [14432.564209] do_sys_openat2+0x2b4/0x410 [14432.566813] do_sys_open+0x73/0xa0 [14432.568227] __x64_sys_openat+0x24/0x30 [14432.573965] do_syscall_64+0xc1/0x440 [14432.579157] entry_SYSCALL_64_after_hwframe+0x49/0xae [14432.583707] RIP: 0033:0x7ff54af45332 [14432.587113] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [14432.606096] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [14432.612736] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [14432.615305] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [14432.619309] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [14432.627009] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [14432.641587] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [14434.178448] rcu: INFO: rcu_sched self-detected stall on CPU [14434.182643] rcu: 0-...!: (6646070 ticks this GP) idle=98a/1/0x4000000000000002 softirq=125154/125154 fqs=959 [14434.194249] (t=12480338 jiffies g=253585 q=6943) [14434.200211] rcu: rcu_sched kthread starved for 12475298 jiffies! g253585 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [14434.206993] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [14434.219636] rcu: RCU grace-period kthread stack dump: [14434.225560] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [14434.234581] Call Trace: [14434.236765] __schedule+0x351/0xcb0 [14434.239206] schedule+0xc0/0x180 [14434.245383] schedule_timeout+0xb4/0x190 [14434.251300] ? do_raw_spin_unlock+0x75/0x190 [14434.258639] ? __next_timer_interrupt+0x160/0x160 [14434.276220] ? prepare_to_swait_event+0xf3/0x140 [14434.280391] rcu_gp_kthread+0x8dd/0x15b0 [14434.291347] ? force_qs_rnp+0x2c0/0x2c0 [14434.303832] kthread+0x1d1/0x200 [14434.307242] ? set_kthread_struct+0x70/0x70 [14434.316006] ret_from_fork+0x1f/0x30 [14434.324385] rcu: Stack dump where RCU GP kthread last ran: [14434.331172] Sending NMI from CPU 0 to CPUs 2: [14434.336473] NMI backtrace for cpu 2 [14434.336481] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14434.336484] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14434.336485] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14434.336488] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14434.336489] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14434.336491] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 [14434.336493] RAX: 0000000000004e1c RBX: ffff96848811cae0 RCX: 0000000000000001 [14434.336494] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [14434.336496] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [14434.336497] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [14434.336498] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [14434.336499] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [14434.336500] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14434.336502] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [14434.336503] Call Trace: [14434.336504] [14434.336505] ? show_regs.cold.9+0x22/0x2f [14434.336506] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [14434.336507] ? nmi_cpu_backtrace_handler+0x15/0x40 [14434.336508] ? nmi_handle+0xbd/0x210 [14434.336509] ? default_do_nmi+0x4b/0x1b0 [14434.336510] ? do_nmi+0x280/0x380 [14434.336511] ? end_repeat_nmi+0x16/0x1f [14434.336512] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14434.336514] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14434.336515] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14434.336516] [14434.336517] do_raw_spin_lock+0x177/0x190 [14434.336518] _raw_spin_lock+0x21/0x30 [14434.336519] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14434.336520] process_one_work+0x2c8/0x700 [14434.336521] worker_thread+0x296/0x6e0 [14434.336522] ? rescuer_thread+0x570/0x570 [14434.336524] kthread+0x1d1/0x200 [14434.336524] ? set_kthread_struct+0x70/0x70 [14434.336526] ret_from_fork+0x1f/0x30 [14434.339928] NMI backtrace for cpu 0 [14434.453961] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14434.459077] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14434.467645] Call Trace: [14434.468238] [14434.469529] ? dump_stack+0xbb/0x10e [14434.471193] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [14434.473570] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [14434.477974] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [14434.481022] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [14434.482840] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [14434.486822] ? rcu_dump_cpu_stacks+0x11b/0x171 [14434.488756] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [14434.490569] ? update_process_times+0x6d/0xc0 [14434.491814] ? tick_sched_handle.isra.18+0x27/0x90 [14434.493315] ? tick_sched_timer+0x53/0xb0 [14434.495480] ? __hrtimer_run_queues+0x1c2/0x4d0 [14434.498111] ? tick_sched_handle.isra.18+0x90/0x90 [14434.501700] ? hrtimer_interrupt+0x128/0x2c0 [14434.503874] ? smp_apic_timer_interrupt+0xa6/0x270 [14434.510886] ? apic_timer_interrupt+0xf/0x20 [14434.515766] [14434.518615] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [14434.525481] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14434.533734] ? do_raw_spin_lock+0x177/0x190 [14434.538747] ? _raw_spin_lock+0x21/0x30 [14434.542115] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14434.547841] ? obd_mod_free+0x46/0x140 [ptlrpc] [14434.552643] ? mdc_commit_open+0x91/0xb0 [mdc] [14434.555230] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14434.559681] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14434.562057] ? after_reply+0xe2c/0x1780 [ptlrpc] [14434.570737] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14434.575702] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [14434.579807] ? do_wait_intr+0xf0/0xf0 [14434.583701] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14434.588944] ? kthread+0x1d1/0x200 [14434.590177] ? set_kthread_struct+0x70/0x70 [14434.598724] ? ret_from_fork+0x1f/0x30 [14436.063912] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [14436.070564] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14436.104861] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14436.113139] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14436.119094] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14436.122566] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14436.140654] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14436.143640] RAX: 0000000000000b86 RBX: ffff96848811cae0 RCX: 0000000000000001 [14436.146954] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [14436.150757] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [14436.154100] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [14436.157200] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [14436.159854] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [14436.164063] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14436.166517] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [14436.170562] Call Trace: [14436.171829] [14436.173008] ? show_regs.cold.9+0x22/0x2f [14436.174948] ? watchdog_timer_fn.cold.6+0x62/0x10a [14436.177029] ? __hrtimer_run_queues+0x1c2/0x4d0 [14436.179809] ? watchdog+0x40/0x40 [14436.181123] ? hrtimer_interrupt+0x128/0x2c0 [14436.183518] ? smp_apic_timer_interrupt+0xa6/0x270 [14436.185674] ? apic_timer_interrupt+0xf/0x20 [14436.188149] [14436.189737] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14436.192826] do_raw_spin_lock+0x177/0x190 [14436.199500] _raw_spin_lock+0x21/0x30 [14436.201027] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14436.207063] ? do_raw_spin_unlock+0x75/0x190 [14436.208473] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14436.213237] ? do_wait_intr+0xf0/0xf0 [14436.214425] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14436.217656] mdc_close+0x53b/0xd70 [mdc] [14436.219810] lmv_close+0x18d/0x490 [lmv] [14436.221794] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [14436.224765] ll_close_inode_openhandle+0x493/0xea0 [lustre] [14436.228090] ll_md_real_close+0xa5/0x2a0 [lustre] [14436.229761] ll_md_close+0x274/0x800 [lustre] [14436.231788] ll_file_release+0x128/0xc50 [lustre] [14436.234231] __fput+0x10d/0x440 [14436.235555] ____fput+0x12/0x20 [14436.236731] task_work_run+0xc6/0x110 [14436.238502] exit_to_usermode_loop+0x1dd/0x1f0 [14436.240529] do_syscall_64+0x3d6/0x440 [14436.242175] entry_SYSCALL_64_after_hwframe+0x49/0xae [14436.244616] RIP: 0033:0x7ff54af45cd8 [14436.245528] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [14436.251880] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [14436.255591] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [14436.259019] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [14436.262487] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [14436.265523] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [14436.268897] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14444.002416] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12485s! [14444.009990] Showing busy workqueues and worker pools: [14444.015101] workqueue events: flags=0x0 [14444.019068] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14444.030621] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [14444.047328] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [14444.062223] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [14444.075540] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14444.086733] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [14444.092977] workqueue events_long: flags=0x0 [14444.098033] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14444.107432] pending: lu_objects_flush [obdclass] [14444.112067] workqueue events_unbound: flags=0x2 [14444.120726] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [14444.131667] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [14444.145264] workqueue events_power_efficient: flags=0x80 [14444.149505] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14444.157419] pending: neigh_periodic_work [14444.163818] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14444.172829] pending: do_cache_clean, check_lifetime [14444.179018] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14444.185514] pending: neigh_periodic_work [14444.186901] workqueue writeback: flags=0x4a [14444.191734] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [14444.200339] pending: wb_workfn, wb_workfn [14444.205979] workqueue xprtiod: flags=0xa [14444.210756] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [14444.217459] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [14444.230111] workqueue ipv6_addrconf: flags=0x40008 [14444.239888] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14444.252483] pending: addrconf_verify_work [14444.254173] workqueue knbd0-recv: flags=0x1a [14444.261447] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14444.273108] in-flight: 657:recv_work [14444.279050] workqueue ptlrpc_pinger: flags=0x2 [14444.294061] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14444.305369] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [14444.314612] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12485s workers=4 idle: 20831 12642 28859 [14444.329463] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14434s workers=2 idle: 659 [14460.062014] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [14460.062901] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [14460.076999] Modules linked in: lustre(O) [14460.090277] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14460.090466] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14460.090473] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14460.090483] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14460.127875] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14460.127956] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14460.127966] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14460.127969] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14460.127988] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14460.127992] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14460.127997] RAX: 0000000000007c23 RBX: ffff96848811cae0 RCX: 0000000000000001 [14460.127999] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [14460.128001] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [14460.128003] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [14460.128007] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [14460.128010] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [14460.128013] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14460.128016] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [14460.128024] Call Trace: [14460.128056] [14460.128063] ? show_regs.cold.9+0x22/0x2f [14460.128075] ? watchdog_timer_fn.cold.6+0x62/0x10a [14460.128084] ? __hrtimer_run_queues+0x1c2/0x4d0 [14460.128088] ? watchdog+0x40/0x40 [14460.128092] ? hrtimer_interrupt+0x128/0x2c0 [14460.128097] ? smp_apic_timer_interrupt+0xa6/0x270 [14460.128105] ? apic_timer_interrupt+0xf/0x20 [14460.128110] [14460.128113] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14460.128120] do_raw_spin_lock+0x177/0x190 [14460.128125] _raw_spin_lock+0x21/0x30 [14460.128130] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14460.137474] ? do_raw_spin_unlock+0x75/0x190 [14460.137592] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14460.152798] [14460.152808] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14460.152827] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14460.152832] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14460.153738] RAX: 0000000000005e89 RBX: ffff96848811cae0 RCX: 0000000000000001 [14460.153741] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [14460.153744] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [14460.153747] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [14460.153750] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [14460.153754] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [14460.153757] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14460.153759] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [14460.153769] Call Trace: [14460.153776] [14460.153784] ? show_regs.cold.9+0x22/0x2f [14460.153799] ? watchdog_timer_fn.cold.6+0x62/0x10a [14460.153805] ? __hrtimer_run_queues+0x1c2/0x4d0 [14460.153810] ? watchdog+0x40/0x40 [14460.153814] ? hrtimer_interrupt+0x128/0x2c0 [14460.153818] ? smp_apic_timer_interrupt+0xa6/0x270 [14460.153828] ? apic_timer_interrupt+0xf/0x20 [14460.153833] [14460.154054] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14460.154063] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14460.154223] do_raw_spin_lock+0x177/0x190 [14460.154232] _raw_spin_lock+0x21/0x30 [14460.154238] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14460.154650] process_one_work+0x2c8/0x700 [14460.154658] worker_thread+0x296/0x6e0 [14460.154663] ? rescuer_thread+0x570/0x570 [14460.154667] kthread+0x1d1/0x200 [14460.154671] ? set_kthread_struct+0x70/0x70 [14460.154675] ret_from_fork+0x1f/0x30 [14460.562647] ? do_wait_intr+0xf0/0xf0 [14460.564307] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14460.573785] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [14460.577882] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14460.587843] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14460.592682] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14460.596214] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [14460.599406] mdc_intent_lock+0x353/0x7e0 [mdc] [14460.608596] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14460.627173] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14460.631649] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14460.648247] lmv_intent_open+0x338/0x1150 [lmv] [14460.658975] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14460.669782] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14460.677670] lmv_intent_lock+0x1dd/0x4c0 [lmv] [14460.688950] ll_intent_lock+0xc5/0xaa0 [lustre] [14460.692296] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14460.696630] ll_lookup_it+0xe0e/0x2270 [lustre] [14460.700297] ? do_raw_spin_unlock+0x75/0x190 [14460.702859] ? _raw_spin_unlock+0x12/0x30 [14460.706818] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [14460.710452] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [14460.726271] ll_atomic_open+0x36a/0xd70 [lustre] [14460.736348] lookup_open+0x356/0xb40 [14460.741558] do_last+0x313/0xfc0 [14460.746368] ? nd_jump_root+0xe5/0x160 [14460.750175] ? path_init+0x437/0x520 [14460.751969] path_openat+0xf7/0x500 [14460.755264] ? do_raw_spin_unlock+0x75/0x190 [14460.759670] do_filp_open+0x99/0x140 [14460.763267] ? getname_flags+0x6e/0x330 [14460.764467] ? __check_object_size+0xff/0x256 [14460.767414] ? do_raw_spin_unlock+0x75/0x190 [14460.770867] ? _raw_spin_unlock+0x12/0x30 [14460.772658] do_sys_openat2+0x2b4/0x410 [14460.776788] do_sys_open+0x73/0xa0 [14460.778588] __x64_sys_openat+0x24/0x30 [14460.781333] do_syscall_64+0xc1/0x440 [14460.782793] entry_SYSCALL_64_after_hwframe+0x49/0xae [14460.786044] RIP: 0033:0x7ff54af45332 [14460.787665] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [14460.801747] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [14460.806392] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [14460.814746] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [14460.821143] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [14460.828249] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [14460.833933] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [14464.064389] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [14464.087800] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14464.184371] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14464.201723] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14464.215353] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14464.225513] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14464.254560] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14464.262314] RAX: 0000000000007e8a RBX: ffff96848811cae0 RCX: 0000000000000001 [14464.278859] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [14464.288418] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [14464.305098] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [14464.320286] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [14464.337811] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [14464.350320] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14464.355312] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [14464.371519] Call Trace: [14464.382954] [14464.384915] ? show_regs.cold.9+0x22/0x2f [14464.397147] ? watchdog_timer_fn.cold.6+0x62/0x10a [14464.405718] ? __hrtimer_run_queues+0x1c2/0x4d0 [14464.415429] ? watchdog+0x40/0x40 [14464.420472] ? hrtimer_interrupt+0x128/0x2c0 [14464.433483] ? smp_apic_timer_interrupt+0xa6/0x270 [14464.445492] ? apic_timer_interrupt+0xf/0x20 [14464.455366] [14464.456093] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14464.466405] do_raw_spin_lock+0x177/0x190 [14464.468730] _raw_spin_lock+0x21/0x30 [14464.474149] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14464.484574] ? do_raw_spin_unlock+0x75/0x190 [14464.502206] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14464.506660] ? do_wait_intr+0xf0/0xf0 [14464.507848] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14464.516830] mdc_close+0x53b/0xd70 [mdc] [14464.520524] lmv_close+0x18d/0x490 [lmv] [14464.522683] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [14464.530324] ll_close_inode_openhandle+0x493/0xea0 [lustre] [14464.540105] ll_md_real_close+0xa5/0x2a0 [lustre] [14464.541721] ll_md_close+0x274/0x800 [lustre] [14464.548224] ll_file_release+0x128/0xc50 [lustre] [14464.552249] __fput+0x10d/0x440 [14464.553367] ____fput+0x12/0x20 [14464.557206] task_work_run+0xc6/0x110 [14464.559329] exit_to_usermode_loop+0x1dd/0x1f0 [14464.562690] do_syscall_64+0x3d6/0x440 [14464.564716] entry_SYSCALL_64_after_hwframe+0x49/0xae [14464.567716] RIP: 0033:0x7ff54af45cd8 [14464.573600] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [14464.613868] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [14464.625644] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [14464.648032] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [14464.658007] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [14464.670418] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [14464.690281] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14472.057129] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [14472.063462] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14472.124600] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14472.138718] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14472.154274] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [14472.173114] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [14472.221041] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14472.233626] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [14472.253031] RDX: 0000000000001ffb RSI: 0000000000000003 RDI: ffff96848811cae0 [14472.276280] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [14472.307187] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [14472.334240] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14472.367954] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [14472.375550] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14472.388566] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [14472.393661] Call Trace: [14472.400647] [14472.403479] ? show_regs.cold.9+0x22/0x2f [14472.408710] ? watchdog_timer_fn.cold.6+0x62/0x10a [14472.413921] ? __hrtimer_run_queues+0x1c2/0x4d0 [14472.420199] ? watchdog+0x40/0x40 [14472.424759] ? hrtimer_interrupt+0x128/0x2c0 [14472.430546] ? smp_apic_timer_interrupt+0xa6/0x270 [14472.433120] ? apic_timer_interrupt+0xf/0x20 [14472.439661] [14472.440636] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [14472.444949] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14472.446871] do_raw_spin_lock+0x177/0x190 [14472.448521] _raw_spin_lock+0x21/0x30 [14472.449778] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14472.452487] obd_mod_free+0x46/0x140 [ptlrpc] [14472.454230] mdc_commit_open+0x91/0xb0 [mdc] [14472.455895] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14472.458808] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14472.461613] after_reply+0xe2c/0x1780 [ptlrpc] [14472.465529] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14472.476473] ptlrpcd+0xa82/0xe40 [ptlrpc] [14472.478953] ? do_wait_intr+0xf0/0xf0 [14472.480589] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14472.484059] kthread+0x1d1/0x200 [14472.485617] ? set_kthread_struct+0x70/0x70 [14472.487099] ret_from_fork+0x1f/0x30 [14474.724265] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12515s! [14474.737124] Showing busy workqueues and worker pools: [14474.740941] workqueue events: flags=0x0 [14474.745814] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14474.756713] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [14474.777090] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [14474.791227] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [14474.825968] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14474.850086] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [14474.857157] workqueue events_long: flags=0x0 [14474.872312] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14474.875354] pending: lu_objects_flush [obdclass] [14474.880094] workqueue events_unbound: flags=0x2 [14474.888131] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [14474.892311] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [14474.900135] workqueue events_power_efficient: flags=0x80 [14474.902246] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14474.908394] pending: neigh_periodic_work [14474.912779] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14474.922269] pending: do_cache_clean, check_lifetime [14474.927454] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14474.931282] pending: neigh_periodic_work [14474.933836] workqueue writeback: flags=0x4a [14474.935234] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [14474.936882] pending: wb_workfn, wb_workfn [14474.951407] workqueue xprtiod: flags=0xa [14474.957835] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [14474.962388] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [14474.965899] workqueue ipv6_addrconf: flags=0x40008 [14474.967050] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14474.974044] pending: addrconf_verify_work [14474.991468] workqueue knbd0-recv: flags=0x1a [14474.993472] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14475.005378] in-flight: 657:recv_work [14475.009185] workqueue ptlrpc_pinger: flags=0x2 [14475.014378] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14475.022925] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [14475.033221] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12516s workers=4 idle: 20831 12642 28859 [14475.038780] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14465s workers=2 idle: 659 [14488.062067] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [14488.064681] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14488.075377] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [14488.104526] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14488.104545] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14488.104548] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14488.104564] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14488.104569] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14488.104575] RAX: 0000000000007a9f RBX: ffff96848811cae0 RCX: 0000000000000001 [14488.104577] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [14488.104579] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [14488.104581] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [14488.104583] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [14488.104587] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [14488.104590] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14488.104593] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [14488.104959] Call Trace: [14488.104966] [14488.104972] ? show_regs.cold.9+0x22/0x2f [14488.104984] ? watchdog_timer_fn.cold.6+0x62/0x10a [14488.104991] ? __hrtimer_run_queues+0x1c2/0x4d0 [14488.104995] ? watchdog+0x40/0x40 [14488.104999] ? hrtimer_interrupt+0x128/0x2c0 [14488.105003] ? smp_apic_timer_interrupt+0xa6/0x270 [14488.105012] ? apic_timer_interrupt+0xf/0x20 [14488.105017] [14488.105020] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14488.105029] do_raw_spin_lock+0x177/0x190 [14488.105034] _raw_spin_lock+0x21/0x30 [14488.105039] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14488.105340] ? do_raw_spin_unlock+0x75/0x190 [14488.105346] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14488.109213] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14488.109295] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14488.109302] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14488.109308] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14488.111103] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14488.114989] ? do_wait_intr+0xf0/0xf0 [14488.115016] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14488.115320] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [14488.115532] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14488.115886] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14488.116015] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14488.116052] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [14488.116081] mdc_intent_lock+0x353/0x7e0 [mdc] [14488.116110] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14488.116195] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14488.116412] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14488.116440] lmv_intent_open+0x338/0x1150 [lmv] [14488.116471] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14488.116554] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14488.116867] lmv_intent_lock+0x1dd/0x4c0 [lmv] [14488.116938] ll_intent_lock+0xc5/0xaa0 [lustre] [14488.117026] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14488.117108] ll_lookup_it+0xe0e/0x2270 [lustre] [14488.117190] ? do_raw_spin_unlock+0x75/0x190 [14488.117198] ? _raw_spin_unlock+0x12/0x30 [14488.117204] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [14488.117234] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [14488.117253] ll_atomic_open+0x36a/0xd70 [lustre] [14488.117340] lookup_open+0x356/0xb40 [14488.117349] do_last+0x313/0xfc0 [14488.117354] ? nd_jump_root+0xe5/0x160 [14488.117359] ? path_init+0x437/0x520 [14488.117363] path_openat+0xf7/0x500 [14488.117368] ? do_raw_spin_unlock+0x75/0x190 [14488.117372] do_filp_open+0x99/0x140 [14488.117379] ? getname_flags+0x6e/0x330 [14488.117383] ? __check_object_size+0xff/0x256 [14488.117391] ? do_raw_spin_unlock+0x75/0x190 [14488.117394] ? _raw_spin_unlock+0x12/0x30 [14488.117398] do_sys_openat2+0x2b4/0x410 [14488.117402] do_sys_open+0x73/0xa0 [14488.117405] __x64_sys_openat+0x24/0x30 [14488.117409] do_syscall_64+0xc1/0x440 [14488.117416] entry_SYSCALL_64_after_hwframe+0x49/0xae [14488.117423] RIP: 0033:0x7ff54af45332 [14488.117431] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [14488.117436] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [14488.117441] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [14488.117443] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [14488.117445] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [14488.117446] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [14488.117448] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [14488.580536] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14488.591136] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14488.595301] RAX: 0000000000004290 RBX: ffff96848811cae0 RCX: 0000000000000001 [14488.599632] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [14488.603007] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [14488.606089] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [14488.609346] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [14488.612361] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [14488.615818] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14488.618399] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [14488.621333] Call Trace: [14488.622402] [14488.623086] ? show_regs.cold.9+0x22/0x2f [14488.624666] ? watchdog_timer_fn.cold.6+0x62/0x10a [14488.626819] ? __hrtimer_run_queues+0x1c2/0x4d0 [14488.629017] ? watchdog+0x40/0x40 [14488.630457] ? hrtimer_interrupt+0x128/0x2c0 [14488.632487] ? smp_apic_timer_interrupt+0xa6/0x270 [14488.634670] ? apic_timer_interrupt+0xf/0x20 [14488.636484] [14488.637143] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14488.640647] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14488.642491] do_raw_spin_lock+0x177/0x190 [14488.644460] _raw_spin_lock+0x21/0x30 [14488.645761] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14488.649578] process_one_work+0x2c8/0x700 [14488.651410] worker_thread+0x296/0x6e0 [14488.653588] ? rescuer_thread+0x570/0x570 [14488.655368] kthread+0x1d1/0x200 [14488.656615] ? set_kthread_struct+0x70/0x70 [14488.658515] ret_from_fork+0x1f/0x30 [14492.058649] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [14492.066675] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14492.104635] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14492.112270] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14492.122600] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14492.135162] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14492.163486] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14492.178521] RAX: 0000000000007e91 RBX: ffff96848811cae0 RCX: 0000000000000001 [14492.186628] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [14492.195509] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [14492.200453] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [14492.205194] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [14492.208616] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [14492.216501] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14492.219124] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [14492.223132] Call Trace: [14492.224264] [14492.225482] ? show_regs.cold.9+0x22/0x2f [14492.227589] ? watchdog_timer_fn.cold.6+0x62/0x10a [14492.233458] ? __hrtimer_run_queues+0x1c2/0x4d0 [14492.237079] ? watchdog+0x40/0x40 [14492.238429] ? hrtimer_interrupt+0x128/0x2c0 [14492.243306] ? smp_apic_timer_interrupt+0xa6/0x270 [14492.248696] ? apic_timer_interrupt+0xf/0x20 [14492.253331] [14492.255345] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14492.259319] do_raw_spin_lock+0x177/0x190 [14492.261474] _raw_spin_lock+0x21/0x30 [14492.263558] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14492.268000] ? do_raw_spin_unlock+0x75/0x190 [14492.270052] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14492.273116] ? do_wait_intr+0xf0/0xf0 [14492.275187] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14492.278196] mdc_close+0x53b/0xd70 [mdc] [14492.286551] lmv_close+0x18d/0x490 [lmv] [14492.291970] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [14492.300430] ll_close_inode_openhandle+0x493/0xea0 [lustre] [14492.305355] ll_md_real_close+0xa5/0x2a0 [lustre] [14492.309647] ll_md_close+0x274/0x800 [lustre] [14492.312650] ll_file_release+0x128/0xc50 [lustre] [14492.314320] __fput+0x10d/0x440 [14492.316863] ____fput+0x12/0x20 [14492.319451] task_work_run+0xc6/0x110 [14492.321548] exit_to_usermode_loop+0x1dd/0x1f0 [14492.328577] do_syscall_64+0x3d6/0x440 [14492.334138] entry_SYSCALL_64_after_hwframe+0x49/0xae [14492.341385] RIP: 0033:0x7ff54af45cd8 [14492.345486] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [14492.358080] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [14492.363296] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [14492.365732] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [14492.371008] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [14492.378421] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [14492.386427] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14500.055048] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [14500.063254] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14500.094586] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14500.108206] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14500.113448] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [14500.120183] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [14500.140558] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14500.145742] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [14500.149707] RDX: 0000000000000192 RSI: 0000000000000003 RDI: ffff96848811cae0 [14500.160094] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [14500.166507] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [14500.174482] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14500.183965] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [14500.193015] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14500.198595] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [14500.205682] Call Trace: [14500.210667] [14500.211955] ? show_regs.cold.9+0x22/0x2f [14500.217347] ? watchdog_timer_fn.cold.6+0x62/0x10a [14500.220456] ? __hrtimer_run_queues+0x1c2/0x4d0 [14500.228394] ? watchdog+0x40/0x40 [14500.235195] ? hrtimer_interrupt+0x128/0x2c0 [14500.237706] ? smp_apic_timer_interrupt+0xa6/0x270 [14500.243873] ? apic_timer_interrupt+0xf/0x20 [14500.246177] [14500.247027] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [14500.253730] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14500.256607] do_raw_spin_lock+0x177/0x190 [14500.264503] _raw_spin_lock+0x21/0x30 [14500.271381] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14500.275084] obd_mod_free+0x46/0x140 [ptlrpc] [14500.280186] mdc_commit_open+0x91/0xb0 [mdc] [14500.285921] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14500.290215] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14500.295432] after_reply+0xe2c/0x1780 [ptlrpc] [14500.298846] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14500.304042] ptlrpcd+0xa82/0xe40 [ptlrpc] [14500.309196] ? do_wait_intr+0xf0/0xf0 [14500.312108] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14500.319497] kthread+0x1d1/0x200 [14500.323098] ? set_kthread_struct+0x70/0x70 [14500.325897] ret_from_fork+0x1f/0x30 [14505.442284] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12546s! [14505.448170] Showing busy workqueues and worker pools: [14505.452241] workqueue events: flags=0x0 [14505.455465] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14505.461051] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [14505.474168] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [14505.486209] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [14505.499462] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14505.507123] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [14505.524113] workqueue events_long: flags=0x0 [14505.529384] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14505.535933] pending: lu_objects_flush [obdclass] [14505.538606] workqueue events_unbound: flags=0x2 [14505.545940] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [14505.554788] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [14505.565553] workqueue events_power_efficient: flags=0x80 [14505.581167] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14505.586186] pending: neigh_periodic_work [14505.589463] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14505.596540] pending: do_cache_clean, check_lifetime [14505.604648] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14505.614154] pending: neigh_periodic_work [14505.616941] workqueue writeback: flags=0x4a [14505.622469] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [14505.628642] pending: wb_workfn, wb_workfn [14505.630301] workqueue xprtiod: flags=0xa [14505.651692] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [14505.663047] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [14505.678159] workqueue ipv6_addrconf: flags=0x40008 [14505.681363] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14505.686328] pending: addrconf_verify_work [14505.689438] workqueue knbd0-recv: flags=0x1a [14505.692658] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14505.696359] in-flight: 657:recv_work [14505.700061] workqueue ptlrpc_pinger: flags=0x2 [14505.706387] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14505.711458] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [14505.717650] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12546s workers=4 idle: 20831 12642 28859 [14505.726352] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14495s workers=2 idle: 659 [14516.059866] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [14516.069167] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [14516.071714] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14516.071803] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14516.071810] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14516.071817] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14516.083948] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14516.085731] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14516.085744] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14516.085747] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14516.085769] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14516.085775] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14516.085779] RAX: 0000000000007d5d RBX: ffff96848811cae0 RCX: 0000000000000001 [14516.085833] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [14516.085836] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [14516.085838] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [14516.085842] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [14516.085847] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [14516.085850] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14516.085853] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [14516.085862] Call Trace: [14516.085872] [14516.085881] ? show_regs.cold.9+0x22/0x2f [14516.085896] ? watchdog_timer_fn.cold.6+0x62/0x10a [14516.085904] ? __hrtimer_run_queues+0x1c2/0x4d0 [14516.085909] ? watchdog+0x40/0x40 [14516.085913] ? hrtimer_interrupt+0x128/0x2c0 [14516.085918] ? smp_apic_timer_interrupt+0xa6/0x270 [14516.085925] ? apic_timer_interrupt+0xf/0x20 [14516.085929] [14516.085932] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14516.085940] do_raw_spin_lock+0x177/0x190 [14516.085945] _raw_spin_lock+0x21/0x30 [14516.085949] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14516.086337] ? do_raw_spin_unlock+0x75/0x190 [14516.086343] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14516.087775] ? do_wait_intr+0xf0/0xf0 [14516.088132] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14516.088417] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [14516.089090] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14516.089315] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14516.089587] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14516.089721] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [14516.089754] mdc_intent_lock+0x353/0x7e0 [mdc] [14516.089785] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14516.089954] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14516.130371] [14516.135601] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14516.135674] lmv_intent_open+0x338/0x1150 [lmv] [14516.135714] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14516.135841] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14516.135931] lmv_intent_lock+0x1dd/0x4c0 [lmv] [14516.135956] ll_intent_lock+0xc5/0xaa0 [lustre] [14516.136056] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14516.136140] ll_lookup_it+0xe0e/0x2270 [lustre] [14516.147175] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14516.147197] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14516.147201] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14516.147208] RAX: 0000000000007f3b RBX: ffff96848811cae0 RCX: 0000000000000001 [14516.147210] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [14516.147212] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [14516.147213] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [14516.147216] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [14516.147219] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [14516.147221] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14516.147223] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [14516.147231] Call Trace: [14516.147237] [14516.147243] ? show_regs.cold.9+0x22/0x2f [14516.147254] ? watchdog_timer_fn.cold.6+0x62/0x10a [14516.147260] ? __hrtimer_run_queues+0x1c2/0x4d0 [14516.147264] ? watchdog+0x40/0x40 [14516.147269] ? hrtimer_interrupt+0x128/0x2c0 [14516.147273] ? smp_apic_timer_interrupt+0xa6/0x270 [14516.147282] ? apic_timer_interrupt+0xf/0x20 [14516.147287] [14516.147290] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14516.147295] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14516.147302] do_raw_spin_lock+0x177/0x190 [14516.147307] _raw_spin_lock+0x21/0x30 [14516.147313] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14516.147598] process_one_work+0x2c8/0x700 [14516.147629] worker_thread+0x296/0x6e0 [14516.147634] ? rescuer_thread+0x570/0x570 [14516.147637] kthread+0x1d1/0x200 [14516.147642] ? set_kthread_struct+0x70/0x70 [14516.147646] ret_from_fork+0x1f/0x30 [14516.601354] ? do_raw_spin_unlock+0x75/0x190 [14516.603914] ? _raw_spin_unlock+0x12/0x30 [14516.610010] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [14516.616183] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [14516.628678] ll_atomic_open+0x36a/0xd70 [lustre] [14516.633182] lookup_open+0x356/0xb40 [14516.635674] do_last+0x313/0xfc0 [14516.637361] ? nd_jump_root+0xe5/0x160 [14516.642351] ? path_init+0x437/0x520 [14516.645408] path_openat+0xf7/0x500 [14516.650947] ? do_raw_spin_unlock+0x75/0x190 [14516.654884] do_filp_open+0x99/0x140 [14516.657052] ? getname_flags+0x6e/0x330 [14516.661459] ? __check_object_size+0xff/0x256 [14516.663625] ? do_raw_spin_unlock+0x75/0x190 [14516.667152] ? _raw_spin_unlock+0x12/0x30 [14516.670316] do_sys_openat2+0x2b4/0x410 [14516.671762] do_sys_open+0x73/0xa0 [14516.672927] __x64_sys_openat+0x24/0x30 [14516.677817] do_syscall_64+0xc1/0x440 [14516.679580] entry_SYSCALL_64_after_hwframe+0x49/0xae [14516.682768] RIP: 0033:0x7ff54af45332 [14516.690734] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [14516.773458] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [14516.792317] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [14516.803421] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [14516.810498] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [14516.819093] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [14516.823984] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [14520.059976] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [14520.064911] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14520.103056] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14520.114994] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14520.120229] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14520.122317] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14520.131527] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14520.144036] RAX: 0000000000007ac1 RBX: ffff96848811cae0 RCX: 0000000000000001 [14520.146826] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [14520.148635] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [14520.153806] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [14520.158096] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [14520.163026] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [14520.166443] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14520.168802] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [14520.172037] Call Trace: [14520.172563] [14520.172944] ? show_regs.cold.9+0x22/0x2f [14520.174030] ? watchdog_timer_fn.cold.6+0x62/0x10a [14520.176340] ? __hrtimer_run_queues+0x1c2/0x4d0 [14520.178315] ? watchdog+0x40/0x40 [14520.180169] ? hrtimer_interrupt+0x128/0x2c0 [14520.193870] ? smp_apic_timer_interrupt+0xa6/0x270 [14520.199969] ? apic_timer_interrupt+0xf/0x20 [14520.203374] [14520.204274] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14520.207898] do_raw_spin_lock+0x177/0x190 [14520.211139] _raw_spin_lock+0x21/0x30 [14520.216175] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14520.220134] ? do_raw_spin_unlock+0x75/0x190 [14520.223849] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14520.227862] ? do_wait_intr+0xf0/0xf0 [14520.229089] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14520.232937] mdc_close+0x53b/0xd70 [mdc] [14520.237531] lmv_close+0x18d/0x490 [lmv] [14520.239076] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [14520.244686] ll_close_inode_openhandle+0x493/0xea0 [lustre] [14520.248971] ll_md_real_close+0xa5/0x2a0 [lustre] [14520.252054] ll_md_close+0x274/0x800 [lustre] [14520.257849] ll_file_release+0x128/0xc50 [lustre] [14520.264943] __fput+0x10d/0x440 [14520.268807] ____fput+0x12/0x20 [14520.272082] task_work_run+0xc6/0x110 [14520.275061] exit_to_usermode_loop+0x1dd/0x1f0 [14520.278168] do_syscall_64+0x3d6/0x440 [14520.280994] entry_SYSCALL_64_after_hwframe+0x49/0xae [14520.284355] RIP: 0033:0x7ff54af45cd8 [14520.286808] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [14520.314925] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [14520.319898] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [14520.329920] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [14520.338981] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [14520.350794] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [14520.374820] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14532.057049] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [14532.066061] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14532.117726] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14532.131184] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14532.135922] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [14532.138933] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [14532.151333] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14532.158151] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [14532.162023] RDX: 00000000000013f8 RSI: 0000000000000003 RDI: ffff96848811cae0 [14532.167701] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [14532.174544] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [14532.180169] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14532.186666] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [14532.191203] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14532.194685] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [14532.200941] Call Trace: [14532.202277] [14532.204090] ? show_regs.cold.9+0x22/0x2f [14532.206379] ? watchdog_timer_fn.cold.6+0x62/0x10a [14532.209064] ? __hrtimer_run_queues+0x1c2/0x4d0 [14532.213917] ? watchdog+0x40/0x40 [14532.216041] ? hrtimer_interrupt+0x128/0x2c0 [14532.219222] ? smp_apic_timer_interrupt+0xa6/0x270 [14532.222078] ? apic_timer_interrupt+0xf/0x20 [14532.224442] [14532.225721] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [14532.229336] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14532.232651] do_raw_spin_lock+0x177/0x190 [14532.235234] _raw_spin_lock+0x21/0x30 [14532.237278] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14532.242806] obd_mod_free+0x46/0x140 [ptlrpc] [14532.248766] mdc_commit_open+0x91/0xb0 [mdc] [14532.260170] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14532.266520] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14532.276313] after_reply+0xe2c/0x1780 [ptlrpc] [14532.287233] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14532.299676] ptlrpcd+0xa82/0xe40 [ptlrpc] [14532.313676] ? do_wait_intr+0xf0/0xf0 [14532.314987] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14532.322860] kthread+0x1d1/0x200 [14532.324636] ? set_kthread_struct+0x70/0x70 [14532.331035] ret_from_fork+0x1f/0x30 [14536.162699] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12577s! [14536.168170] Showing busy workqueues and worker pools: [14536.175711] workqueue events: flags=0x0 [14536.179468] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14536.190774] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [14536.200591] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [14536.205068] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [14536.210757] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14536.214785] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [14536.217989] workqueue events_long: flags=0x0 [14536.220711] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14536.229054] pending: lu_objects_flush [obdclass] [14536.237119] workqueue events_unbound: flags=0x2 [14536.245001] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [14536.248658] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [14536.262258] workqueue events_power_efficient: flags=0x80 [14536.272730] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14536.277645] pending: neigh_periodic_work [14536.280610] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14536.284316] pending: do_cache_clean, check_lifetime [14536.289883] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14536.297244] pending: neigh_periodic_work [14536.303412] workqueue writeback: flags=0x4a [14536.305356] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [14536.313539] pending: wb_workfn, wb_workfn [14536.315777] workqueue xprtiod: flags=0xa [14536.318332] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [14536.322030] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [14536.326639] workqueue ipv6_addrconf: flags=0x40008 [14536.330271] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14536.334494] pending: addrconf_verify_work [14536.338051] workqueue knbd0-recv: flags=0x1a [14536.341725] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14536.346079] in-flight: 657:recv_work [14536.348402] workqueue ptlrpc_pinger: flags=0x2 [14536.351752] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14536.358891] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [14536.368523] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12577s workers=4 idle: 20831 12642 28859 [14536.379207] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14526s workers=2 idle: 659 [14544.059128] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [14544.061072] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [14544.062753] Modules linked in: [14544.075579] Modules linked in: [14544.075582] lustre(O) osp(O) ofd(O) [14544.078728] lustre(O) [14544.079507] lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) [14544.084918] osp(O) [14544.085515] obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14544.085555] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14544.085560] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14544.085565] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14544.085839] RIP: 0010:native_safe_halt+0x2e/0x50 [14544.085850] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14544.085855] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14544.085860] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [14544.085863] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c2134154 [14544.085865] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [14544.085868] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [14544.085871] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [14544.085874] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [14544.085877] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14544.085880] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [14544.085888] Call Trace: [14544.085893] [14544.085897] ? show_regs.cold.9+0x22/0x2f [14544.085907] ? watchdog_timer_fn.cold.6+0x62/0x10a [14544.085914] ? __hrtimer_run_queues+0x1c2/0x4d0 [14544.085917] ? watchdog+0x40/0x40 [14544.085921] ? hrtimer_interrupt+0x128/0x2c0 [14544.085925] ? smp_apic_timer_interrupt+0xa6/0x270 [14544.085931] ? apic_timer_interrupt+0xf/0x20 [14544.085936] [14544.085939] ? native_safe_halt+0x2e/0x50 [14544.085943] kvm_wait+0xcd/0xe0 [14544.085949] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14544.085956] do_raw_spin_lock+0x177/0x190 [14544.085960] _raw_spin_lock+0x21/0x30 [14544.085964] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14544.095248] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14544.095336] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14544.095343] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14544.095356] RIP: 0010:native_safe_halt+0x2e/0x50 [14544.095369] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14544.095373] RSP: 0000:ffffaf9fc15234f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14544.095377] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [14544.095379] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c21b4154 [14544.095382] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [14544.095384] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [14544.095386] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [14544.095389] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [14544.095396] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14544.095398] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [14544.095406] Call Trace: [14544.095411] [14544.095417] ? show_regs.cold.9+0x22/0x2f [14544.095427] ? watchdog_timer_fn.cold.6+0x62/0x10a [14544.095433] ? __hrtimer_run_queues+0x1c2/0x4d0 [14544.095437] ? watchdog+0x40/0x40 [14544.095441] ? hrtimer_interrupt+0x128/0x2c0 [14544.095446] ? smp_apic_timer_interrupt+0xa6/0x270 [14544.095453] ? apic_timer_interrupt+0xf/0x20 [14544.095458] [14544.095461] ? native_safe_halt+0x2e/0x50 [14544.095465] kvm_wait+0xcd/0xe0 [14544.095472] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14544.095482] do_raw_spin_lock+0x177/0x190 [14544.095486] _raw_spin_lock+0x21/0x30 [14544.095491] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14544.095800] ? do_raw_spin_unlock+0x75/0x190 [14544.095807] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14544.096157] ? do_wait_intr+0xf0/0xf0 [14544.096165] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14544.096450] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [14544.096672] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14544.098801] process_one_work+0x2c8/0x700 [14544.098815] worker_thread+0x296/0x6e0 [14544.098820] ? rescuer_thread+0x570/0x570 [14544.098825] kthread+0x1d1/0x200 [14544.098829] ? set_kthread_struct+0x70/0x70 [14544.098834] ret_from_fork+0x1f/0x30 [14544.402611] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14544.405557] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14544.407828] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [14544.411326] mdc_intent_lock+0x353/0x7e0 [mdc] [14544.413403] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14544.416029] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14544.419603] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14544.422382] lmv_intent_open+0x338/0x1150 [lmv] [14544.425224] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14544.427845] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14544.430655] lmv_intent_lock+0x1dd/0x4c0 [lmv] [14544.432781] ll_intent_lock+0xc5/0xaa0 [lustre] [14544.437570] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14544.443247] ll_lookup_it+0xe0e/0x2270 [lustre] [14544.448042] ? do_raw_spin_unlock+0x75/0x190 [14544.449457] ? _raw_spin_unlock+0x12/0x30 [14544.459859] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [14544.463717] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [14544.471531] ll_atomic_open+0x36a/0xd70 [lustre] [14544.475438] lookup_open+0x356/0xb40 [14544.476760] do_last+0x313/0xfc0 [14544.479132] ? nd_jump_root+0xe5/0x160 [14544.480274] ? path_init+0x437/0x520 [14544.481161] path_openat+0xf7/0x500 [14544.482048] ? do_raw_spin_unlock+0x75/0x190 [14544.483722] do_filp_open+0x99/0x140 [14544.484803] ? getname_flags+0x6e/0x330 [14544.485927] ? __check_object_size+0xff/0x256 [14544.487134] ? do_raw_spin_unlock+0x75/0x190 [14544.488533] ? _raw_spin_unlock+0x12/0x30 [14544.489719] do_sys_openat2+0x2b4/0x410 [14544.491017] do_sys_open+0x73/0xa0 [14544.491952] __x64_sys_openat+0x24/0x30 [14544.493115] do_syscall_64+0xc1/0x440 [14544.494532] entry_SYSCALL_64_after_hwframe+0x49/0xae [14544.496238] RIP: 0033:0x7ff54af45332 [14544.497245] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [14544.504787] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [14544.507431] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [14544.509681] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [14544.511917] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [14544.515127] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [14544.521228] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [14548.058409] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [14548.068952] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14548.111325] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14548.118475] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14548.123349] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14548.127292] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14548.140937] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14548.145072] RAX: 0000000000007a98 RBX: ffff96848811cae0 RCX: 0000000000000001 [14548.150219] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [14548.158247] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [14548.169410] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [14548.174390] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [14548.179540] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [14548.183830] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14548.186952] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [14548.191262] Call Trace: [14548.192713] [14548.194336] ? show_regs.cold.9+0x22/0x2f [14548.196800] ? watchdog_timer_fn.cold.6+0x62/0x10a [14548.199717] ? __hrtimer_run_queues+0x1c2/0x4d0 [14548.202785] ? watchdog+0x40/0x40 [14548.204927] ? hrtimer_interrupt+0x128/0x2c0 [14548.207694] ? smp_apic_timer_interrupt+0xa6/0x270 [14548.210714] ? apic_timer_interrupt+0xf/0x20 [14548.217847] [14548.219356] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14548.225939] do_raw_spin_lock+0x177/0x190 [14548.228610] _raw_spin_lock+0x21/0x30 [14548.231268] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14548.234016] ? do_raw_spin_unlock+0x75/0x190 [14548.237346] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14548.240980] ? do_wait_intr+0xf0/0xf0 [14548.243631] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14548.247397] mdc_close+0x53b/0xd70 [mdc] [14548.250512] lmv_close+0x18d/0x490 [lmv] [14548.254546] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [14548.266358] ll_close_inode_openhandle+0x493/0xea0 [lustre] [14548.271294] ll_md_real_close+0xa5/0x2a0 [lustre] [14548.274834] ll_md_close+0x274/0x800 [lustre] [14548.277913] ll_file_release+0x128/0xc50 [lustre] [14548.281333] __fput+0x10d/0x440 [14548.283636] ____fput+0x12/0x20 [14548.286322] task_work_run+0xc6/0x110 [14548.288647] exit_to_usermode_loop+0x1dd/0x1f0 [14548.291727] do_syscall_64+0x3d6/0x440 [14548.294167] entry_SYSCALL_64_after_hwframe+0x49/0xae [14548.297821] RIP: 0033:0x7ff54af45cd8 [14548.300459] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [14548.313840] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [14548.318034] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [14548.322376] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [14548.329504] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [14548.336896] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [14548.346363] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14564.083759] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [14564.086385] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14564.102386] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14564.106585] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14564.111407] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [14564.115571] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [14564.123055] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14564.125786] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [14564.128357] RDX: 00000000000015f8 RSI: 0000000000000003 RDI: ffff96848811cae0 [14564.130897] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [14564.133512] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [14564.135909] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14564.138405] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [14564.141075] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14564.143097] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [14564.145460] Call Trace: [14564.146239] [14564.146861] ? show_regs.cold.9+0x22/0x2f [14564.148088] ? watchdog_timer_fn.cold.6+0x62/0x10a [14564.149728] ? __hrtimer_run_queues+0x1c2/0x4d0 [14564.151219] ? watchdog+0x40/0x40 [14564.152332] ? hrtimer_interrupt+0x128/0x2c0 [14564.153626] ? smp_apic_timer_interrupt+0xa6/0x270 [14564.155205] ? apic_timer_interrupt+0xf/0x20 [14564.156686] [14564.157391] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [14564.159192] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14564.160914] do_raw_spin_lock+0x177/0x190 [14564.162407] _raw_spin_lock+0x21/0x30 [14564.163583] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14564.165553] obd_mod_free+0x46/0x140 [ptlrpc] [14564.167320] mdc_commit_open+0x91/0xb0 [mdc] [14564.168784] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14564.170798] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14564.172971] after_reply+0xe2c/0x1780 [ptlrpc] [14564.174880] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14564.176882] ptlrpcd+0xa82/0xe40 [ptlrpc] [14564.179520] ? do_wait_intr+0xf0/0xf0 [14564.181987] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14564.186840] kthread+0x1d1/0x200 [14564.188032] ? set_kthread_struct+0x70/0x70 [14564.189508] ret_from_fork+0x1f/0x30 [14566.879076] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12607s! [14566.885524] Showing busy workqueues and worker pools: [14566.889882] workqueue events: flags=0x0 [14566.895216] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14566.897625] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [14566.910775] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [14566.918906] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [14566.926403] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14566.936623] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [14566.948633] workqueue events_long: flags=0x0 [14566.957586] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14566.965767] pending: lu_objects_flush [obdclass] [14566.972810] workqueue events_unbound: flags=0x2 [14566.979440] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [14566.984840] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [14567.002043] workqueue events_power_efficient: flags=0x80 [14567.004429] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14567.010877] pending: neigh_periodic_work [14567.018456] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14567.021012] pending: do_cache_clean, check_lifetime [14567.039634] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14567.049409] pending: neigh_periodic_work [14567.056441] workqueue writeback: flags=0x4a [14567.062949] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [14567.073247] pending: wb_workfn, wb_workfn [14567.079687] workqueue xprtiod: flags=0xa [14567.089585] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [14567.092244] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [14567.104228] workqueue ipv6_addrconf: flags=0x40008 [14567.106791] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14567.115339] pending: addrconf_verify_work [14567.120197] workqueue knbd0-recv: flags=0x1a [14567.125474] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14567.131677] in-flight: 657:recv_work [14567.134102] workqueue ptlrpc_pinger: flags=0x2 [14567.140752] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14567.144625] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [14567.155315] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12608s workers=4 idle: 20831 12642 28859 [14567.166636] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14557s workers=2 idle: 659 [14572.063022] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [14572.066596] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [14572.077708] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14572.077806] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14572.077815] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14572.077821] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14572.081344] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14572.081366] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14572.081371] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14572.081376] RAX: 0000000000001227 RBX: ffff96848811cae0 RCX: 0000000000000001 [14572.081379] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [14572.081381] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [14572.081383] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [14572.081385] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [14572.081388] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [14572.081391] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14572.081393] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [14572.081400] Call Trace: [14572.081404] [14572.081409] ? show_regs.cold.9+0x22/0x2f [14572.081418] ? watchdog_timer_fn.cold.6+0x62/0x10a [14572.081424] ? __hrtimer_run_queues+0x1c2/0x4d0 [14572.081427] ? watchdog+0x40/0x40 [14572.081432] ? hrtimer_interrupt+0x128/0x2c0 [14572.081435] ? smp_apic_timer_interrupt+0xa6/0x270 [14572.081442] ? apic_timer_interrupt+0xf/0x20 [14572.081446] [14572.081449] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14572.081453] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14572.081466] do_raw_spin_lock+0x177/0x190 [14572.081471] _raw_spin_lock+0x21/0x30 [14572.081475] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14572.081735] process_one_work+0x2c8/0x700 [14572.081742] worker_thread+0x296/0x6e0 [14572.081746] ? rescuer_thread+0x570/0x570 [14572.081749] kthread+0x1d1/0x200 [14572.081754] ? set_kthread_struct+0x70/0x70 [14572.081757] ret_from_fork+0x1f/0x30 [14572.413780] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14572.504436] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14572.527839] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14572.541751] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14572.567370] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14572.602712] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [14572.606339] RAX: 000000000000710d RBX: ffff96848811cae0 RCX: 0000000000000001 [14572.614037] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [14572.626852] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [14572.649149] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [14572.658477] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [14572.672916] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [14572.707393] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14572.729832] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [14572.758526] Call Trace: [14572.765180] [14572.766365] ? show_regs.cold.9+0x22/0x2f [14572.780845] ? watchdog_timer_fn.cold.6+0x62/0x10a [14572.796340] ? __hrtimer_run_queues+0x1c2/0x4d0 [14572.801549] ? watchdog+0x40/0x40 [14572.803019] ? hrtimer_interrupt+0x128/0x2c0 [14572.805020] ? smp_apic_timer_interrupt+0xa6/0x270 [14572.807271] ? apic_timer_interrupt+0xf/0x20 [14572.814521] [14572.817195] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14572.824991] do_raw_spin_lock+0x177/0x190 [14572.828422] _raw_spin_lock+0x21/0x30 [14572.832792] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14572.847312] ? do_raw_spin_unlock+0x75/0x190 [14572.852188] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14572.875655] ? do_wait_intr+0xf0/0xf0 [14572.876443] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14572.877673] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [14572.878926] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14572.917151] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14572.937094] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14572.944835] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [14572.949847] mdc_intent_lock+0x353/0x7e0 [mdc] [14572.959348] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14572.967225] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14572.989858] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14573.005850] lmv_intent_open+0x338/0x1150 [lmv] [14573.007667] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14573.017023] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14573.029549] lmv_intent_lock+0x1dd/0x4c0 [lmv] [14573.037667] ll_intent_lock+0xc5/0xaa0 [lustre] [14573.048256] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14573.061596] ll_lookup_it+0xe0e/0x2270 [lustre] [14573.063307] ? do_raw_spin_unlock+0x75/0x190 [14573.074561] ? _raw_spin_unlock+0x12/0x30 [14573.076002] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [14573.088559] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [14573.105801] ll_atomic_open+0x36a/0xd70 [lustre] [14573.119724] lookup_open+0x356/0xb40 [14573.129983] do_last+0x313/0xfc0 [14573.130899] ? nd_jump_root+0xe5/0x160 [14573.132062] ? path_init+0x437/0x520 [14573.142217] path_openat+0xf7/0x500 [14573.143196] ? do_raw_spin_unlock+0x75/0x190 [14573.151746] do_filp_open+0x99/0x140 [14573.155689] ? getname_flags+0x6e/0x330 [14573.159324] ? __check_object_size+0xff/0x256 [14573.169775] ? do_raw_spin_unlock+0x75/0x190 [14573.174924] ? _raw_spin_unlock+0x12/0x30 [14573.177358] do_sys_openat2+0x2b4/0x410 [14573.184350] do_sys_open+0x73/0xa0 [14573.187615] __x64_sys_openat+0x24/0x30 [14573.194648] do_syscall_64+0xc1/0x440 [14573.201515] entry_SYSCALL_64_after_hwframe+0x49/0xae [14573.211411] RIP: 0033:0x7ff54af45332 [14573.212472] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [14573.243927] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [14573.261663] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [14573.271344] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [14573.277897] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [14573.282665] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [14573.286831] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [14576.061533] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [14576.069667] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14576.095315] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14576.099850] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14576.110498] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14576.116461] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14576.131071] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14576.134625] RAX: 0000000000002c10 RBX: ffff96848811cae0 RCX: 0000000000000001 [14576.143362] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [14576.150339] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [14576.155596] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [14576.161281] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [14576.168512] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [14576.177579] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14576.188348] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [14576.228126] Call Trace: [14576.231660] [14576.236760] ? show_regs.cold.9+0x22/0x2f [14576.240371] ? watchdog_timer_fn.cold.6+0x62/0x10a [14576.255601] ? __hrtimer_run_queues+0x1c2/0x4d0 [14576.258028] ? watchdog+0x40/0x40 [14576.263181] ? hrtimer_interrupt+0x128/0x2c0 [14576.263993] ? smp_apic_timer_interrupt+0xa6/0x270 [14576.269322] ? apic_timer_interrupt+0xf/0x20 [14576.270636] [14576.271267] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14576.277314] do_raw_spin_lock+0x177/0x190 [14576.285496] _raw_spin_lock+0x21/0x30 [14576.290226] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14576.310638] ? do_raw_spin_unlock+0x75/0x190 [14576.315747] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14576.320497] ? do_wait_intr+0xf0/0xf0 [14576.321802] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14576.327133] mdc_close+0x53b/0xd70 [mdc] [14576.331434] lmv_close+0x18d/0x490 [lmv] [14576.333018] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [14576.337231] ll_close_inode_openhandle+0x493/0xea0 [lustre] [14576.340350] ll_md_real_close+0xa5/0x2a0 [lustre] [14576.342680] ll_md_close+0x274/0x800 [lustre] [14576.344116] ll_file_release+0x128/0xc50 [lustre] [14576.347198] __fput+0x10d/0x440 [14576.348782] ____fput+0x12/0x20 [14576.352255] task_work_run+0xc6/0x110 [14576.354238] exit_to_usermode_loop+0x1dd/0x1f0 [14576.361382] do_syscall_64+0x3d6/0x440 [14576.363447] entry_SYSCALL_64_after_hwframe+0x49/0xae [14576.365972] RIP: 0033:0x7ff54af45cd8 [14576.367716] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [14576.380302] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [14576.400634] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [14576.416089] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [14576.418069] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [14576.445510] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [14576.483293] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14592.059080] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [14592.061867] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14592.096888] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14592.103281] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14592.107546] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [14592.110423] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [14592.120571] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14592.123069] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [14592.125234] RDX: 0000000000007ea7 RSI: 0000000000000003 RDI: ffff96848811cae0 [14592.129035] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [14592.133222] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [14592.136847] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14592.141023] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [14592.146151] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14592.151235] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [14592.161200] Call Trace: [14592.162448] [14592.163775] ? show_regs.cold.9+0x22/0x2f [14592.169447] ? watchdog_timer_fn.cold.6+0x62/0x10a [14592.173112] ? __hrtimer_run_queues+0x1c2/0x4d0 [14592.175005] ? watchdog+0x40/0x40 [14592.179234] ? hrtimer_interrupt+0x128/0x2c0 [14592.181341] ? smp_apic_timer_interrupt+0xa6/0x270 [14592.185951] ? apic_timer_interrupt+0xf/0x20 [14592.188527] [14592.189674] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [14592.194386] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14592.198921] do_raw_spin_lock+0x177/0x190 [14592.200644] _raw_spin_lock+0x21/0x30 [14592.211975] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14592.221104] obd_mod_free+0x46/0x140 [ptlrpc] [14592.230088] mdc_commit_open+0x91/0xb0 [mdc] [14592.239473] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14592.246288] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14592.255225] after_reply+0xe2c/0x1780 [ptlrpc] [14592.270391] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14592.283541] ptlrpcd+0xa82/0xe40 [ptlrpc] [14592.291958] ? do_wait_intr+0xf0/0xf0 [14592.293387] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14592.309258] kthread+0x1d1/0x200 [14592.312681] ? set_kthread_struct+0x70/0x70 [14592.316459] ret_from_fork+0x1f/0x30 [14597.599129] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12638s! [14597.609983] Showing busy workqueues and worker pools: [14597.622045] workqueue events: flags=0x0 [14597.626533] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14597.636623] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [14597.653333] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [14597.657271] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [14597.667217] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14597.674245] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [14597.681252] workqueue events_long: flags=0x0 [14597.683639] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14597.690524] pending: lu_objects_flush [obdclass] [14597.697012] workqueue events_unbound: flags=0x2 [14597.704332] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [14597.710266] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [14597.724963] workqueue events_power_efficient: flags=0x80 [14597.726253] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14597.735258] pending: neigh_periodic_work [14597.736947] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14597.746992] pending: do_cache_clean, check_lifetime [14597.754438] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14597.771362] pending: neigh_periodic_work [14597.779764] workqueue writeback: flags=0x4a [14597.789624] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [14597.802343] pending: wb_workfn, wb_workfn [14597.810394] workqueue xprtiod: flags=0xa [14597.820268] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [14597.822564] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [14597.834379] workqueue ipv6_addrconf: flags=0x40008 [14597.846416] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14597.861232] pending: addrconf_verify_work [14597.862888] workqueue knbd0-recv: flags=0x1a [14597.870872] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14597.874670] in-flight: 657:recv_work [14597.886712] workqueue ptlrpc_pinger: flags=0x2 [14597.888361] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14597.899496] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [14597.909288] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12639s workers=4 idle: 20831 12642 28859 [14597.921316] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14588s workers=2 idle: 659 [14600.062988] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [14600.069197] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [14600.069386] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14600.069538] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14600.069544] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14600.069551] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14600.069875] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14600.069887] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14600.069891] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14600.069896] RAX: 00000000000038a5 RBX: ffff96848811cae0 RCX: 0000000000000001 [14600.069897] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [14600.069900] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [14600.069902] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [14600.069904] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [14600.069907] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [14600.069909] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14600.069912] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [14600.069919] Call Trace: [14600.069925] [14600.069930] ? show_regs.cold.9+0x22/0x2f [14600.069941] ? watchdog_timer_fn.cold.6+0x62/0x10a [14600.069947] ? __hrtimer_run_queues+0x1c2/0x4d0 [14600.069952] ? watchdog+0x40/0x40 [14600.069956] ? hrtimer_interrupt+0x128/0x2c0 [14600.069960] ? smp_apic_timer_interrupt+0xa6/0x270 [14600.069969] ? apic_timer_interrupt+0xf/0x20 [14600.069973] [14600.069976] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14600.069981] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14600.069986] do_raw_spin_lock+0x177/0x190 [14600.069992] _raw_spin_lock+0x21/0x30 [14600.069997] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14600.070346] process_one_work+0x2c8/0x700 [14600.070354] worker_thread+0x296/0x6e0 [14600.070358] ? rescuer_thread+0x570/0x570 [14600.070363] kthread+0x1d1/0x200 [14600.070368] ? set_kthread_struct+0x70/0x70 [14600.070372] ret_from_fork+0x1f/0x30 [14600.374225] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14600.446884] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14600.474487] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14600.494981] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14600.506504] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14600.533410] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14600.544956] RAX: 000000000000730a RBX: ffff96848811cae0 RCX: 0000000000000001 [14600.554963] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [14600.560497] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [14600.564338] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [14600.570611] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [14600.576189] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [14600.583025] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14600.585030] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [14600.589354] Call Trace: [14600.591159] [14600.595195] ? show_regs.cold.9+0x22/0x2f [14600.598017] ? watchdog_timer_fn.cold.6+0x62/0x10a [14600.603752] ? __hrtimer_run_queues+0x1c2/0x4d0 [14600.609581] ? watchdog+0x40/0x40 [14600.620356] ? hrtimer_interrupt+0x128/0x2c0 [14600.623448] ? smp_apic_timer_interrupt+0xa6/0x270 [14600.631466] ? apic_timer_interrupt+0xf/0x20 [14600.636267] [14600.641130] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14600.645320] do_raw_spin_lock+0x177/0x190 [14600.648919] _raw_spin_lock+0x21/0x30 [14600.660536] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14600.672859] ? do_raw_spin_unlock+0x75/0x190 [14600.683061] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14600.695235] ? do_wait_intr+0xf0/0xf0 [14600.706125] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14600.715905] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [14600.723549] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14600.738206] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14600.749255] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14600.753337] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [14600.772628] mdc_intent_lock+0x353/0x7e0 [mdc] [14600.781913] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14600.791375] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14600.801337] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14600.810346] lmv_intent_open+0x338/0x1150 [lmv] [14600.812904] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14600.819524] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14600.827349] lmv_intent_lock+0x1dd/0x4c0 [lmv] [14600.833625] ll_intent_lock+0xc5/0xaa0 [lustre] [14600.842772] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14600.852544] ll_lookup_it+0xe0e/0x2270 [lustre] [14600.859148] ? do_raw_spin_unlock+0x75/0x190 [14600.863041] ? _raw_spin_unlock+0x12/0x30 [14600.868193] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [14600.871310] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [14600.875204] ll_atomic_open+0x36a/0xd70 [lustre] [14600.879324] lookup_open+0x356/0xb40 [14600.886245] do_last+0x313/0xfc0 [14600.895851] ? nd_jump_root+0xe5/0x160 [14600.904159] ? path_init+0x437/0x520 [14600.906366] path_openat+0xf7/0x500 [14600.908327] ? do_raw_spin_unlock+0x75/0x190 [14600.914939] do_filp_open+0x99/0x140 [14600.917473] ? getname_flags+0x6e/0x330 [14600.921446] ? __check_object_size+0xff/0x256 [14600.926114] ? do_raw_spin_unlock+0x75/0x190 [14600.930495] ? _raw_spin_unlock+0x12/0x30 [14600.934170] do_sys_openat2+0x2b4/0x410 [14600.937248] do_sys_open+0x73/0xa0 [14600.944840] __x64_sys_openat+0x24/0x30 [14600.951957] do_syscall_64+0xc1/0x440 [14600.958026] entry_SYSCALL_64_after_hwframe+0x49/0xae [14600.963057] RIP: 0033:0x7ff54af45332 [14600.966223] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [14600.987844] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [14600.994003] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [14600.998912] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [14601.010470] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [14601.019299] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [14601.022203] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [14604.059094] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [14604.080131] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14604.116885] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14604.131330] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14604.143087] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14604.151113] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14604.170181] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14604.175943] RAX: 0000000000007eae RBX: ffff96848811cae0 RCX: 0000000000000001 [14604.184893] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [14604.196323] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [14604.198239] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [14604.215894] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [14604.221046] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [14604.226109] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14604.237573] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [14604.257819] Call Trace: [14604.261935] [14604.265891] ? show_regs.cold.9+0x22/0x2f [14604.272002] ? watchdog_timer_fn.cold.6+0x62/0x10a [14604.280122] ? __hrtimer_run_queues+0x1c2/0x4d0 [14604.288145] ? watchdog+0x40/0x40 [14604.294154] ? hrtimer_interrupt+0x128/0x2c0 [14604.310115] ? smp_apic_timer_interrupt+0xa6/0x270 [14604.329909] ? apic_timer_interrupt+0xf/0x20 [14604.338581] [14604.341725] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14604.346216] do_raw_spin_lock+0x177/0x190 [14604.351603] _raw_spin_lock+0x21/0x30 [14604.355361] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14604.381164] ? do_raw_spin_unlock+0x75/0x190 [14604.395106] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14604.407788] ? do_wait_intr+0xf0/0xf0 [14604.415354] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14604.427029] mdc_close+0x53b/0xd70 [mdc] [14604.431862] lmv_close+0x18d/0x490 [lmv] [14604.439339] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [14604.444922] ll_close_inode_openhandle+0x493/0xea0 [lustre] [14604.452561] ll_md_real_close+0xa5/0x2a0 [lustre] [14604.454683] ll_md_close+0x274/0x800 [lustre] [14604.466866] ll_file_release+0x128/0xc50 [lustre] [14604.474930] __fput+0x10d/0x440 [14604.481738] ____fput+0x12/0x20 [14604.485497] task_work_run+0xc6/0x110 [14604.495331] exit_to_usermode_loop+0x1dd/0x1f0 [14604.499790] do_syscall_64+0x3d6/0x440 [14604.506990] entry_SYSCALL_64_after_hwframe+0x49/0xae [14604.511991] RIP: 0033:0x7ff54af45cd8 [14604.515346] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [14604.541490] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [14604.552374] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [14604.565122] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [14604.574660] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [14604.592312] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [14604.606204] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14614.182538] rcu: INFO: rcu_sched self-detected stall on CPU [14614.189990] rcu: 0-...!: (6738064 ticks this GP) idle=98a/1/0x4000000000000002 softirq=125154/125154 fqs=959 [14614.212199] (t=12660342 jiffies g=253585 q=7036) [14614.216298] rcu: rcu_sched kthread starved for 12655302 jiffies! g253585 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=3 [14614.224733] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [14614.228987] rcu: RCU grace-period kthread stack dump: [14614.231589] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [14614.245659] Call Trace: [14614.246930] __schedule+0x351/0xcb0 [14614.257530] schedule+0xc0/0x180 [14614.258587] schedule_timeout+0xb4/0x190 [14614.259953] ? do_raw_spin_unlock+0x75/0x190 [14614.263651] ? __next_timer_interrupt+0x160/0x160 [14614.270660] ? prepare_to_swait_event+0xf3/0x140 [14614.279250] rcu_gp_kthread+0x8dd/0x15b0 [14614.283223] ? force_qs_rnp+0x2c0/0x2c0 [14614.287453] kthread+0x1d1/0x200 [14614.289952] ? set_kthread_struct+0x70/0x70 [14614.293991] ret_from_fork+0x1f/0x30 [14614.296119] rcu: Stack dump where RCU GP kthread last ran: [14614.301740] Sending NMI from CPU 0 to CPUs 3: [14614.308924] NMI backtrace for cpu 3 [14614.308933] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14614.308936] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14614.308937] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14614.308939] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14614.308942] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 [14614.308944] RAX: 0000000000002982 RBX: ffff96848811cae0 RCX: 0000000000000001 [14614.308946] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [14614.308947] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [14614.308949] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [14614.308951] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [14614.308952] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [14614.308954] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14614.308955] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [14614.308957] Call Trace: [14614.308958] [14614.308959] ? show_regs.cold.9+0x22/0x2f [14614.308960] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [14614.308961] ? nmi_cpu_backtrace_handler+0x15/0x40 [14614.308962] ? nmi_handle+0xbd/0x210 [14614.308963] ? default_do_nmi+0x4b/0x1b0 [14614.308964] ? do_nmi+0x280/0x380 [14614.308966] ? end_repeat_nmi+0x16/0x1f [14614.308967] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14614.308968] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14614.308969] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14614.308971] [14614.308971] do_raw_spin_lock+0x177/0x190 [14614.308972] _raw_spin_lock+0x21/0x30 [14614.308973] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14614.308975] ? do_raw_spin_unlock+0x75/0x190 [14614.308976] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14614.308977] ? do_wait_intr+0xf0/0xf0 [14614.308978] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14614.308979] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [14614.308980] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14614.308981] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14614.308982] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14614.308983] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [14614.308984] mdc_intent_lock+0x353/0x7e0 [mdc] [14614.308985] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14614.308986] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14614.308987] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14614.308988] lmv_intent_open+0x338/0x1150 [lmv] [14614.308990] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14614.308991] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14614.308992] lmv_intent_lock+0x1dd/0x4c0 [lmv] [14614.308993] ll_intent_lock+0xc5/0xaa0 [lustre] [14614.308995] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14614.308996] ll_lookup_it+0xe0e/0x2270 [lustre] [14614.308997] ? do_raw_spin_unlock+0x75/0x190 [14614.308999] ? _raw_spin_unlock+0x12/0x30 [14614.309000] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [14614.309001] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [14614.309002] ll_atomic_open+0x36a/0xd70 [lustre] [14614.309003] lookup_open+0x356/0xb40 [14614.309004] do_last+0x313/0xfc0 [14614.309005] ? nd_jump_root+0xe5/0x160 [14614.309006] ? path_init+0x437/0x520 [14614.309006] path_openat+0xf7/0x500 [14614.309007] ? do_raw_spin_unlock+0x75/0x190 [14614.309008] do_filp_open+0x99/0x140 [14614.309009] ? getname_flags+0x6e/0x330 [14614.309010] ? __check_object_size+0xff/0x256 [14614.309011] ? do_raw_spin_unlock+0x75/0x190 [14614.309012] ? _raw_spin_unlock+0x12/0x30 [14614.309013] do_sys_openat2+0x2b4/0x410 [14614.309013] do_sys_open+0x73/0xa0 [14614.309014] __x64_sys_openat+0x24/0x30 [14614.309015] do_syscall_64+0xc1/0x440 [14614.309016] entry_SYSCALL_64_after_hwframe+0x49/0xae [14614.309017] RIP: 0033:0x7ff54af45332 [14614.309019] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [14614.309021] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [14614.309024] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [14614.309025] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [14614.309027] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [14614.309028] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [14614.309029] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [14614.309709] NMI backtrace for cpu 0 [14614.643371] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14614.654137] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14614.658909] Call Trace: [14614.659996] [14614.660878] ? dump_stack+0xbb/0x10e [14614.662220] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [14614.664025] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [14614.666111] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [14614.667852] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [14614.669651] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [14614.671381] ? rcu_dump_cpu_stacks+0x11b/0x171 [14614.673872] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [14614.675681] ? update_process_times+0x6d/0xc0 [14614.677260] ? tick_sched_handle.isra.18+0x27/0x90 [14614.679944] ? tick_sched_timer+0x53/0xb0 [14614.683350] ? __hrtimer_run_queues+0x1c2/0x4d0 [14614.688252] ? tick_sched_handle.isra.18+0x90/0x90 [14614.690682] ? hrtimer_interrupt+0x128/0x2c0 [14614.692861] ? smp_apic_timer_interrupt+0xa6/0x270 [14614.695200] ? apic_timer_interrupt+0xf/0x20 [14614.697299] [14614.698369] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [14614.701060] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14614.703893] ? do_raw_spin_lock+0x177/0x190 [14614.705614] ? _raw_spin_lock+0x21/0x30 [14614.708248] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14614.711450] ? obd_mod_free+0x46/0x140 [ptlrpc] [14614.714636] ? mdc_commit_open+0x91/0xb0 [mdc] [14614.716673] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14614.719915] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14614.723991] ? after_reply+0xe2c/0x1780 [ptlrpc] [14614.726348] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14614.728890] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [14614.730823] ? do_wait_intr+0xf0/0xf0 [14614.732221] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14614.735252] ? kthread+0x1d1/0x200 [14614.737284] ? set_kthread_struct+0x70/0x70 [14614.738457] ? ret_from_fork+0x1f/0x30 [14628.059012] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [14628.062939] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14628.116630] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14628.136654] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14628.143746] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14628.149270] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14628.155782] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14628.175697] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14628.182525] RAX: 0000000000000c8e RBX: ffff96848811cae0 RCX: 0000000000000001 [14628.186123] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [14628.189742] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [14628.196584] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [14628.202141] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [14628.210474] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [14628.217536] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14628.223775] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [14628.227647] Call Trace: [14628.230726] [14628.231761] ? show_regs.cold.9+0x22/0x2f [14628.237777] ? watchdog_timer_fn.cold.6+0x62/0x10a [14628.245293] ? __hrtimer_run_queues+0x1c2/0x4d0 [14628.251822] ? watchdog+0x40/0x40 [14628.256781] ? hrtimer_interrupt+0x128/0x2c0 [14628.258361] ? smp_apic_timer_interrupt+0xa6/0x270 [14628.263543] ? apic_timer_interrupt+0xf/0x20 [14628.268949] [14628.270434] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14628.278906] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14628.283492] do_raw_spin_lock+0x177/0x190 [14628.289977] _raw_spin_lock+0x21/0x30 [14628.296155] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14628.299280] process_one_work+0x2c8/0x700 [14628.307202] worker_thread+0x296/0x6e0 [14628.313021] ? rescuer_thread+0x570/0x570 [14628.314417] kthread+0x1d1/0x200 [14628.319104] BUG: workqueue lockup - pool [14628.323220] ? set_kthread_struct+0x70/0x70 [14628.323241] ret_from_fork+0x1f/0x30 [14628.326214] cpus=0-3 flags=0x4 nice=0 stuck for 12669s! [14628.326235] Showing busy workqueues and worker pools: [14628.326238] workqueue events: flags=0x0 [14628.326241] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14628.326250] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [14628.327589] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [14628.327603] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [14628.361952] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14628.365503] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [14628.369348] workqueue events_long: flags=0x0 [14628.371367] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14628.374521] pending: lu_objects_flush [obdclass] [14628.377742] workqueue events_unbound: flags=0x2 [14628.381116] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [14628.384792] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [14628.391473] workqueue events_power_efficient: flags=0x80 [14628.393733] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14628.396190] pending: neigh_periodic_work [14628.398909] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14628.403165] pending: do_cache_clean, check_lifetime [14628.406337] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14628.409158] pending: neigh_periodic_work [14628.412272] workqueue writeback: flags=0x4a [14628.414841] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [14628.419275] pending: wb_workfn, wb_workfn [14628.422240] workqueue xprtiod: flags=0xa [14628.424335] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [14628.428206] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [14628.434571] workqueue ipv6_addrconf: flags=0x40008 [14628.443525] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14628.448065] pending: addrconf_verify_work [14628.460553] workqueue knbd0-recv: flags=0x1a [14628.465750] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14628.479240] in-flight: 657:recv_work [14628.485441] workqueue ptlrpc_pinger: flags=0x2 [14628.489486] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14628.495233] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [14628.500123] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12669s workers=4 idle: 20831 12642 28859 [14628.505221] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14618s workers=2 idle: 659 [14632.058053] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [14632.062544] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14632.087688] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14632.093161] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14632.097657] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14632.099712] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14632.107686] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14632.111705] RAX: 000000000000161c RBX: ffff96848811cae0 RCX: 0000000000000001 [14632.114771] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [14632.118581] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [14632.121127] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [14632.123728] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [14632.126316] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [14632.129895] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14632.131895] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [14632.134842] Call Trace: [14632.135898] [14632.136808] ? show_regs.cold.9+0x22/0x2f [14632.138470] ? watchdog_timer_fn.cold.6+0x62/0x10a [14632.140221] ? __hrtimer_run_queues+0x1c2/0x4d0 [14632.141874] ? watchdog+0x40/0x40 [14632.143231] ? hrtimer_interrupt+0x128/0x2c0 [14632.144729] ? smp_apic_timer_interrupt+0xa6/0x270 [14632.146261] ? apic_timer_interrupt+0xf/0x20 [14632.147515] [14632.148648] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14632.150839] do_raw_spin_lock+0x177/0x190 [14632.152341] _raw_spin_lock+0x21/0x30 [14632.154039] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14632.157004] ? do_raw_spin_unlock+0x75/0x190 [14632.159184] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14632.161519] ? do_wait_intr+0xf0/0xf0 [14632.162910] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14632.165925] mdc_close+0x53b/0xd70 [mdc] [14632.167416] lmv_close+0x18d/0x490 [lmv] [14632.168735] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [14632.171231] ll_close_inode_openhandle+0x493/0xea0 [lustre] [14632.173998] ll_md_real_close+0xa5/0x2a0 [lustre] [14632.175770] ll_md_close+0x274/0x800 [lustre] [14632.177420] ll_file_release+0x128/0xc50 [lustre] [14632.179681] __fput+0x10d/0x440 [14632.180859] ____fput+0x12/0x20 [14632.182055] task_work_run+0xc6/0x110 [14632.183732] exit_to_usermode_loop+0x1dd/0x1f0 [14632.185333] do_syscall_64+0x3d6/0x440 [14632.186389] entry_SYSCALL_64_after_hwframe+0x49/0xae [14632.188229] RIP: 0033:0x7ff54af45cd8 [14632.189312] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [14632.195008] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [14632.198138] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [14632.200651] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [14632.202833] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [14632.204758] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [14632.207629] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14640.065411] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [14640.088888] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14640.148630] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14640.158126] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14640.168110] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14640.175951] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14640.187364] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14640.197937] RAX: 0000000000003f85 RBX: ffff96848811cae0 RCX: 0000000000000001 [14640.202598] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [14640.210339] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [14640.214521] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [14640.220090] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [14640.228067] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [14640.233249] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14640.236224] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [14640.240371] Call Trace: [14640.243052] [14640.246121] ? show_regs.cold.9+0x22/0x2f [14640.251005] ? watchdog_timer_fn.cold.6+0x62/0x10a [14640.255537] ? __hrtimer_run_queues+0x1c2/0x4d0 [14640.258594] ? watchdog+0x40/0x40 [14640.260518] ? hrtimer_interrupt+0x128/0x2c0 [14640.262129] ? smp_apic_timer_interrupt+0xa6/0x270 [14640.264365] ? apic_timer_interrupt+0xf/0x20 [14640.268567] [14640.270943] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14640.278504] do_raw_spin_lock+0x177/0x190 [14640.283506] _raw_spin_lock+0x21/0x30 [14640.286317] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14640.288858] ? do_raw_spin_unlock+0x75/0x190 [14640.291487] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14640.295049] ? do_wait_intr+0xf0/0xf0 [14640.297100] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14640.302217] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [14640.305440] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14640.308727] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14640.312373] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14640.315504] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [14640.319126] mdc_intent_lock+0x353/0x7e0 [mdc] [14640.322804] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14640.325034] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14640.330847] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14640.336782] lmv_intent_open+0x338/0x1150 [lmv] [14640.343044] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14640.353605] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14640.358823] lmv_intent_lock+0x1dd/0x4c0 [lmv] [14640.362895] ll_intent_lock+0xc5/0xaa0 [lustre] [14640.365029] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14640.370799] ll_lookup_it+0xe0e/0x2270 [lustre] [14640.374704] ? do_raw_spin_unlock+0x75/0x190 [14640.376524] ? _raw_spin_unlock+0x12/0x30 [14640.378555] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [14640.382567] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [14640.385389] ll_atomic_open+0x36a/0xd70 [lustre] [14640.388374] lookup_open+0x356/0xb40 [14640.390721] do_last+0x313/0xfc0 [14640.393439] ? nd_jump_root+0xe5/0x160 [14640.397486] ? path_init+0x437/0x520 [14640.401857] path_openat+0xf7/0x500 [14640.410789] ? do_raw_spin_unlock+0x75/0x190 [14640.413380] do_filp_open+0x99/0x140 [14640.415898] ? getname_flags+0x6e/0x330 [14640.417632] ? __check_object_size+0xff/0x256 [14640.423528] ? do_raw_spin_unlock+0x75/0x190 [14640.428900] ? _raw_spin_unlock+0x12/0x30 [14640.435599] do_sys_openat2+0x2b4/0x410 [14640.441189] do_sys_open+0x73/0xa0 [14640.442331] __x64_sys_openat+0x24/0x30 [14640.443561] do_syscall_64+0xc1/0x440 [14640.447505] entry_SYSCALL_64_after_hwframe+0x49/0xae [14640.453868] RIP: 0033:0x7ff54af45332 [14640.456400] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [14640.480515] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [14640.491296] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [14640.495791] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [14640.510562] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [14640.525457] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [14640.532749] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [14656.055086] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [14656.059065] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [14656.059361] Modules linked in: [14656.062137] Modules linked in: [14656.062140] lustre(O) osp(O) [14656.063470] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14656.063539] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14656.063547] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14656.063553] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14656.063829] RIP: 0010:native_safe_halt+0x2e/0x50 [14656.063840] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14656.063844] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14656.063848] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [14656.063851] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c2134154 [14656.063853] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [14656.063855] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [14656.063858] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [14656.063862] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [14656.063864] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14656.063866] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [14656.063876] Call Trace: [14656.063881] [14656.063886] ? show_regs.cold.9+0x22/0x2f [14656.063897] ? watchdog_timer_fn.cold.6+0x62/0x10a [14656.063903] ? __hrtimer_run_queues+0x1c2/0x4d0 [14656.063908] ? watchdog+0x40/0x40 [14656.063912] ? hrtimer_interrupt+0x128/0x2c0 [14656.063915] ? smp_apic_timer_interrupt+0xa6/0x270 [14656.063924] ? apic_timer_interrupt+0xf/0x20 [14656.063929] [14656.063932] ? native_safe_halt+0x2e/0x50 [14656.063936] kvm_wait+0xcd/0xe0 [14656.063943] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14656.063951] do_raw_spin_lock+0x177/0x190 [14656.063955] _raw_spin_lock+0x21/0x30 [14656.063960] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14656.065701] ofd(O) [14656.066936] process_one_work+0x2c8/0x700 [14656.100940] lod(O) [14656.105717] worker_thread+0x296/0x6e0 [14656.105741] ? rescuer_thread+0x570/0x570 [14656.112358] mdt(O) [14656.114665] kthread+0x1d1/0x200 [14656.114683] ? set_kthread_struct+0x70/0x70 [14656.114688] ret_from_fork+0x1f/0x30 [14656.255109] mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14656.289761] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14656.298609] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14656.304540] RIP: 0010:native_safe_halt+0x2e/0x50 [14656.310806] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14656.327287] RSP: 0000:ffffaf9fc1b97bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14656.333057] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [14656.337269] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff96848811cae0 [14656.343247] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [14656.355127] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [14656.360295] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14656.365841] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [14656.372021] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14656.377019] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [14656.381998] Call Trace: [14656.384396] [14656.385472] ? show_regs.cold.9+0x22/0x2f [14656.388376] ? watchdog_timer_fn.cold.6+0x62/0x10a [14656.391541] ? __hrtimer_run_queues+0x1c2/0x4d0 [14656.394920] ? watchdog+0x40/0x40 [14656.398693] ? hrtimer_interrupt+0x128/0x2c0 [14656.402568] ? smp_apic_timer_interrupt+0xa6/0x270 [14656.407554] ? apic_timer_interrupt+0xf/0x20 [14656.410897] [14656.413037] ? native_safe_halt+0x2e/0x50 [14656.415856] kvm_wait+0xcd/0xe0 [14656.418541] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14656.422303] do_raw_spin_lock+0x177/0x190 [14656.425437] _raw_spin_lock+0x21/0x30 [14656.428022] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14656.431772] obd_mod_free+0x46/0x140 [ptlrpc] [14656.435530] mdc_commit_open+0x91/0xb0 [mdc] [14656.439553] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14656.445413] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14656.451222] after_reply+0xe2c/0x1780 [ptlrpc] [14656.454745] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14656.458547] ptlrpcd+0xa82/0xe40 [ptlrpc] [14656.461808] ? do_wait_intr+0xf0/0xf0 [14656.470509] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14656.477794] kthread+0x1d1/0x200 [14656.480573] ? set_kthread_struct+0x70/0x70 [14656.483575] ret_from_fork+0x1f/0x30 [14658.527201] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12699s! [14658.531442] Showing busy workqueues and worker pools: [14658.535214] workqueue events: flags=0x0 [14658.536987] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14658.545385] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [14658.554021] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [14658.557325] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [14658.572736] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14658.578200] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [14658.583501] workqueue events_long: flags=0x0 [14658.585909] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14658.593050] pending: lu_objects_flush [obdclass] [14658.600501] workqueue events_unbound: flags=0x2 [14658.602669] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [14658.605715] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [14658.610948] workqueue events_power_efficient: flags=0x80 [14658.614049] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14658.618566] pending: neigh_periodic_work [14658.621189] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14658.625050] pending: do_cache_clean, check_lifetime [14658.627768] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14658.631214] pending: neigh_periodic_work [14658.633611] workqueue writeback: flags=0x4a [14658.636195] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [14658.638535] pending: wb_workfn, wb_workfn [14658.640029] workqueue xprtiod: flags=0xa [14658.641851] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [14658.644747] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [14658.647964] workqueue ipv6_addrconf: flags=0x40008 [14658.649980] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14658.652936] pending: addrconf_verify_work [14658.655314] workqueue knbd0-recv: flags=0x1a [14658.657141] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14658.659917] in-flight: 657:recv_work [14658.661424] workqueue ptlrpc_pinger: flags=0x2 [14658.663023] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14658.665556] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [14658.668179] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12699s workers=4 idle: 20831 12642 28859 [14658.671700] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14648s workers=2 idle: 659 [14660.063662] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [14660.074932] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14660.104585] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14660.116090] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14660.129453] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14660.135722] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14660.160233] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14660.172412] RAX: 00000000000020ff RBX: ffff96848811cae0 RCX: 0000000000000001 [14660.189432] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [14660.191817] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [14660.205538] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [14660.213566] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [14660.218346] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [14660.234563] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14660.239515] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [14660.247579] Call Trace: [14660.248713] [14660.249717] ? show_regs.cold.9+0x22/0x2f [14660.251345] ? watchdog_timer_fn.cold.6+0x62/0x10a [14660.253717] ? __hrtimer_run_queues+0x1c2/0x4d0 [14660.255746] ? watchdog+0x40/0x40 [14660.257341] ? hrtimer_interrupt+0x128/0x2c0 [14660.258755] ? smp_apic_timer_interrupt+0xa6/0x270 [14660.261042] ? apic_timer_interrupt+0xf/0x20 [14660.262747] [14660.263595] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14660.266339] do_raw_spin_lock+0x177/0x190 [14660.267960] _raw_spin_lock+0x21/0x30 [14660.269470] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14660.271810] ? do_raw_spin_unlock+0x75/0x190 [14660.274459] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14660.277004] ? do_wait_intr+0xf0/0xf0 [14660.278455] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14660.281649] mdc_close+0x53b/0xd70 [mdc] [14660.283592] lmv_close+0x18d/0x490 [lmv] [14660.285541] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [14660.288674] ll_close_inode_openhandle+0x493/0xea0 [lustre] [14660.291972] ll_md_real_close+0xa5/0x2a0 [lustre] [14660.294935] ll_md_close+0x274/0x800 [lustre] [14660.297649] ll_file_release+0x128/0xc50 [lustre] [14660.300800] __fput+0x10d/0x440 [14660.304416] ____fput+0x12/0x20 [14660.306037] task_work_run+0xc6/0x110 [14660.309720] exit_to_usermode_loop+0x1dd/0x1f0 [14660.315448] do_syscall_64+0x3d6/0x440 [14660.317446] entry_SYSCALL_64_after_hwframe+0x49/0xae [14660.324674] RIP: 0033:0x7ff54af45cd8 [14660.326492] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [14660.339533] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [14660.343432] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [14660.347599] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [14660.351160] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [14660.354746] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [14660.358523] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14668.068517] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [14668.082857] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14668.140264] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14668.150373] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14668.158257] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14668.171978] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14668.189036] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14668.202883] RAX: 0000000000004683 RBX: ffff96848811cae0 RCX: 0000000000000001 [14668.205513] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [14668.222912] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [14668.234246] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [14668.241212] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [14668.246337] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [14668.250324] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14668.254300] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [14668.261889] Call Trace: [14668.269899] [14668.271792] ? show_regs.cold.9+0x22/0x2f [14668.277342] ? watchdog_timer_fn.cold.6+0x62/0x10a [14668.278906] ? __hrtimer_run_queues+0x1c2/0x4d0 [14668.285504] ? watchdog+0x40/0x40 [14668.288512] ? hrtimer_interrupt+0x128/0x2c0 [14668.296328] ? smp_apic_timer_interrupt+0xa6/0x270 [14668.300463] ? apic_timer_interrupt+0xf/0x20 [14668.305787] [14668.306564] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14668.313069] do_raw_spin_lock+0x177/0x190 [14668.317343] _raw_spin_lock+0x21/0x30 [14668.321655] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14668.329661] ? do_raw_spin_unlock+0x75/0x190 [14668.335269] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14668.340341] ? do_wait_intr+0xf0/0xf0 [14668.348060] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14668.350043] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [14668.356171] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14668.368910] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14668.378176] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14668.386254] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [14668.391227] mdc_intent_lock+0x353/0x7e0 [mdc] [14668.394899] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14668.397737] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14668.410942] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14668.424536] lmv_intent_open+0x338/0x1150 [lmv] [14668.433229] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14668.443508] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14668.445437] lmv_intent_lock+0x1dd/0x4c0 [lmv] [14668.453282] ll_intent_lock+0xc5/0xaa0 [lustre] [14668.458797] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14668.466540] ll_lookup_it+0xe0e/0x2270 [lustre] [14668.472955] ? do_raw_spin_unlock+0x75/0x190 [14668.475209] ? _raw_spin_unlock+0x12/0x30 [14668.477867] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [14668.481877] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [14668.486835] ll_atomic_open+0x36a/0xd70 [lustre] [14668.490986] lookup_open+0x356/0xb40 [14668.495534] do_last+0x313/0xfc0 [14668.497844] ? nd_jump_root+0xe5/0x160 [14668.499152] ? path_init+0x437/0x520 [14668.500889] path_openat+0xf7/0x500 [14668.502890] ? do_raw_spin_unlock+0x75/0x190 [14668.508775] do_filp_open+0x99/0x140 [14668.513644] ? getname_flags+0x6e/0x330 [14668.515967] ? __check_object_size+0xff/0x256 [14668.518128] ? do_raw_spin_unlock+0x75/0x190 [14668.520834] ? _raw_spin_unlock+0x12/0x30 [14668.523699] do_sys_openat2+0x2b4/0x410 [14668.527951] do_sys_open+0x73/0xa0 [14668.530182] __x64_sys_openat+0x24/0x30 [14668.532642] do_syscall_64+0xc1/0x440 [14668.534784] entry_SYSCALL_64_after_hwframe+0x49/0xae [14668.541622] RIP: 0033:0x7ff54af45332 [14668.547830] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [14668.570541] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [14668.574518] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [14668.579538] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [14668.584560] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [14668.588470] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [14668.591034] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [14684.055564] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [14684.068608] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [14684.069785] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14684.070032] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14684.070040] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14684.070043] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [14684.070059] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [14684.070063] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14684.070067] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [14684.070070] RDX: 00000000000013e2 RSI: 0000000000000003 RDI: ffff96848811cae0 [14684.070072] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [14684.070074] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [14684.070076] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14684.070079] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [14684.070082] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14684.070084] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [14684.070092] Call Trace: [14684.070098] [14684.070104] ? show_regs.cold.9+0x22/0x2f [14684.070114] ? watchdog_timer_fn.cold.6+0x62/0x10a [14684.070121] ? __hrtimer_run_queues+0x1c2/0x4d0 [14684.070125] ? watchdog+0x40/0x40 [14684.070129] ? hrtimer_interrupt+0x128/0x2c0 [14684.070133] ? smp_apic_timer_interrupt+0xa6/0x270 [14684.070142] ? apic_timer_interrupt+0xf/0x20 [14684.070147] [14684.070149] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [14684.070154] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14684.070159] do_raw_spin_lock+0x177/0x190 [14684.070163] _raw_spin_lock+0x21/0x30 [14684.070169] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14684.070569] obd_mod_free+0x46/0x140 [ptlrpc] [14684.070759] mdc_commit_open+0x91/0xb0 [mdc] [14684.070801] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14684.071413] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14684.071616] after_reply+0xe2c/0x1780 [ptlrpc] [14684.071808] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14684.072045] ptlrpcd+0xa82/0xe40 [ptlrpc] [14684.072512] ? do_wait_intr+0xf0/0xf0 [14684.072523] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14684.072735] kthread+0x1d1/0x200 [14684.072741] ? set_kthread_struct+0x70/0x70 [14684.072746] ret_from_fork+0x1f/0x30 [14684.353278] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14684.402727] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14684.411362] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14684.421580] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14684.425574] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14684.432212] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14684.443645] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14684.453759] RAX: 0000000000005bb3 RBX: ffff96848811cae0 RCX: 0000000000000001 [14684.456958] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [14684.461362] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [14684.466031] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [14684.476751] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [14684.487666] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [14684.496488] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14684.502584] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [14684.507753] Call Trace: [14684.513276] [14684.515603] ? show_regs.cold.9+0x22/0x2f [14684.516820] ? watchdog_timer_fn.cold.6+0x62/0x10a [14684.522417] ? __hrtimer_run_queues+0x1c2/0x4d0 [14684.526286] ? watchdog+0x40/0x40 [14684.535487] ? hrtimer_interrupt+0x128/0x2c0 [14684.538735] ? smp_apic_timer_interrupt+0xa6/0x270 [14684.543225] ? apic_timer_interrupt+0xf/0x20 [14684.546590] [14684.551344] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14684.566284] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14684.569811] do_raw_spin_lock+0x177/0x190 [14684.575600] _raw_spin_lock+0x21/0x30 [14684.581087] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14684.593194] process_one_work+0x2c8/0x700 [14684.602763] worker_thread+0x296/0x6e0 [14684.607483] ? rescuer_thread+0x570/0x570 [14684.614138] kthread+0x1d1/0x200 [14684.618823] ? set_kthread_struct+0x70/0x70 [14684.620214] ret_from_fork+0x1f/0x30 [14688.058062] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [14688.064824] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14688.112788] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14688.134767] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14688.141007] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14688.146819] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14688.162641] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14688.168722] RAX: 0000000000002586 RBX: ffff96848811cae0 RCX: 0000000000000001 [14688.173613] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [14688.180822] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [14688.186064] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [14688.191739] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [14688.197067] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [14688.204465] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14688.208754] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [14688.213863] Call Trace: [14688.215646] [14688.217444] ? show_regs.cold.9+0x22/0x2f [14688.220525] ? watchdog_timer_fn.cold.6+0x62/0x10a [14688.223806] ? __hrtimer_run_queues+0x1c2/0x4d0 [14688.227037] ? watchdog+0x40/0x40 [14688.230549] ? hrtimer_interrupt+0x128/0x2c0 [14688.233789] ? smp_apic_timer_interrupt+0xa6/0x270 [14688.237336] ? apic_timer_interrupt+0xf/0x20 [14688.240483] [14688.242264] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14688.246628] do_raw_spin_lock+0x177/0x190 [14688.249516] _raw_spin_lock+0x21/0x30 [14688.252617] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14688.257518] ? do_raw_spin_unlock+0x75/0x190 [14688.260806] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14688.264900] ? do_wait_intr+0xf0/0xf0 [14688.267904] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14688.275429] mdc_close+0x53b/0xd70 [mdc] [14688.277646] lmv_close+0x18d/0x490 [lmv] [14688.280941] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [14688.285381] ll_close_inode_openhandle+0x493/0xea0 [lustre] [14688.289479] ll_md_real_close+0xa5/0x2a0 [lustre] [14688.292464] ll_md_close+0x274/0x800 [lustre] [14688.295075] ll_file_release+0x128/0xc50 [lustre] [14688.298871] __fput+0x10d/0x440 [14688.300858] ____fput+0x12/0x20 [14688.302630] task_work_run+0xc6/0x110 [14688.304738] exit_to_usermode_loop+0x1dd/0x1f0 [14688.307403] do_syscall_64+0x3d6/0x440 [14688.309441] entry_SYSCALL_64_after_hwframe+0x49/0xae [14688.312028] RIP: 0033:0x7ff54af45cd8 [14688.313937] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [14688.323051] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [14688.327843] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [14688.333330] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [14688.337576] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [14688.341326] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [14688.345520] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14688.736538] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12729s! [14688.743535] Showing busy workqueues and worker pools: [14688.762106] workqueue events: flags=0x0 [14688.771460] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14688.785584] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [14688.795278] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [14688.817460] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [14688.830558] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14688.838386] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [14688.843669] workqueue events_long: flags=0x0 [14688.849611] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14688.854572] pending: lu_objects_flush [obdclass] [14688.865751] workqueue events_unbound: flags=0x2 [14688.870461] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [14688.875452] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [14688.894363] workqueue events_power_efficient: flags=0x80 [14688.897663] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14688.902101] pending: neigh_periodic_work [14688.906602] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14688.918389] pending: do_cache_clean, check_lifetime [14688.922593] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14688.937926] pending: neigh_periodic_work [14688.951507] workqueue writeback: flags=0x4a [14688.953031] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [14688.975465] pending: wb_workfn, wb_workfn [14688.981294] workqueue xprtiod: flags=0xa [14688.985511] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [14688.992991] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [14689.005189] workqueue ipv6_addrconf: flags=0x40008 [14689.008500] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14689.012819] pending: addrconf_verify_work [14689.019241] workqueue knbd0-recv: flags=0x1a [14689.021813] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14689.028111] in-flight: 657:recv_work [14689.029349] workqueue ptlrpc_pinger: flags=0x2 [14689.045220] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14689.049268] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [14689.052851] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12730s workers=4 idle: 20831 12642 28859 [14689.058211] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14679s workers=2 idle: 659 [14696.061086] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [14696.078309] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14696.148294] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14696.164260] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14696.176467] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14696.189674] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14696.240041] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [14696.251279] RAX: 0000000000007e0b RBX: ffff96848811cae0 RCX: 0000000000000001 [14696.264534] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [14696.268617] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [14696.272153] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [14696.277194] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [14696.281263] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [14696.290964] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14696.297134] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [14696.306446] Call Trace: [14696.311320] [14696.315258] ? show_regs.cold.9+0x22/0x2f [14696.321051] ? watchdog_timer_fn.cold.6+0x62/0x10a [14696.328037] ? __hrtimer_run_queues+0x1c2/0x4d0 [14696.333299] ? watchdog+0x40/0x40 [14696.339640] ? hrtimer_interrupt+0x128/0x2c0 [14696.342483] ? smp_apic_timer_interrupt+0xa6/0x270 [14696.346627] ? apic_timer_interrupt+0xf/0x20 [14696.349166] [14696.352279] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14696.354013] do_raw_spin_lock+0x177/0x190 [14696.359333] _raw_spin_lock+0x21/0x30 [14696.364295] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14696.371483] ? do_raw_spin_unlock+0x75/0x190 [14696.378095] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14696.385910] ? do_wait_intr+0xf0/0xf0 [14696.387556] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14696.391818] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [14696.395277] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14696.399409] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14696.402933] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14696.406343] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [14696.409323] mdc_intent_lock+0x353/0x7e0 [mdc] [14696.412117] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14696.415613] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14696.420156] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14696.424261] lmv_intent_open+0x338/0x1150 [lmv] [14696.427953] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14696.433559] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14696.438419] lmv_intent_lock+0x1dd/0x4c0 [lmv] [14696.447536] ll_intent_lock+0xc5/0xaa0 [lustre] [14696.455400] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14696.496339] ll_lookup_it+0xe0e/0x2270 [lustre] [14696.504195] ? do_raw_spin_unlock+0x75/0x190 [14696.507953] ? _raw_spin_unlock+0x12/0x30 [14696.519519] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [14696.525500] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [14696.537356] ll_atomic_open+0x36a/0xd70 [lustre] [14696.551565] lookup_open+0x356/0xb40 [14696.571191] do_last+0x313/0xfc0 [14696.592469] ? nd_jump_root+0xe5/0x160 [14696.610560] ? path_init+0x437/0x520 [14696.614040] path_openat+0xf7/0x500 [14696.618501] ? do_raw_spin_unlock+0x75/0x190 [14696.625591] do_filp_open+0x99/0x140 [14696.632510] ? getname_flags+0x6e/0x330 [14696.636616] ? __check_object_size+0xff/0x256 [14696.641453] ? do_raw_spin_unlock+0x75/0x190 [14696.642859] ? _raw_spin_unlock+0x12/0x30 [14696.650712] do_sys_openat2+0x2b4/0x410 [14696.656179] do_sys_open+0x73/0xa0 [14696.665590] __x64_sys_openat+0x24/0x30 [14696.666807] do_syscall_64+0xc1/0x440 [14696.670405] entry_SYSCALL_64_after_hwframe+0x49/0xae [14696.675184] RIP: 0033:0x7ff54af45332 [14696.679083] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [14696.693749] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [14696.702962] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [14696.710416] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [14696.719226] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [14696.729183] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [14696.731643] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [14712.070679] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [14712.084335] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14712.138788] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14712.149498] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14712.160429] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14712.163810] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14712.168179] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14712.196139] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14712.207996] RAX: 0000000000005d8c RBX: ffff96848811cae0 RCX: 0000000000000001 [14712.220339] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [14712.224673] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [14712.235807] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [14712.239532] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [14712.244235] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [14712.248685] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14712.252467] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [14712.258819] Call Trace: [14712.260226] [14712.261263] ? show_regs.cold.9+0x22/0x2f [14712.263683] ? watchdog_timer_fn.cold.6+0x62/0x10a [14712.266532] ? __hrtimer_run_queues+0x1c2/0x4d0 [14712.269821] ? watchdog+0x40/0x40 [14712.272379] ? hrtimer_interrupt+0x128/0x2c0 [14712.275106] ? smp_apic_timer_interrupt+0xa6/0x270 [14712.277470] ? apic_timer_interrupt+0xf/0x20 [14712.279698] [14712.280833] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14712.283650] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14712.286394] do_raw_spin_lock+0x177/0x190 [14712.288260] _raw_spin_lock+0x21/0x30 [14712.290227] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14712.293482] process_one_work+0x2c8/0x700 [14712.295663] worker_thread+0x296/0x6e0 [14712.297313] ? rescuer_thread+0x570/0x570 [14712.300256] kthread+0x1d1/0x200 [14712.301865] ? set_kthread_struct+0x70/0x70 [14712.304477] ret_from_fork+0x1f/0x30 [14716.055024] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [14716.057239] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14716.058024] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [14716.070723] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14716.073093] Modules linked in: [14716.075320] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14716.075323] lustre(O) osp(O) [14716.075858] RIP: 0010:native_safe_halt+0x2e/0x50 [14716.078233] ofd(O) [14716.079264] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14716.080794] lod(O) [14716.082131] RSP: 0000:ffffaf9fc1b97bb8 EFLAGS: 00000212 [14716.086501] mdt(O) [14716.087607] ORIG_RAX: ffffffffffffff13 [14716.089248] mdd(O) [14716.090143] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [14716.091073] mgs(O) [14716.091460] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff96848811cae0 [14716.092984] lquota(O) [14716.093459] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [14716.094810] lfsck(O) [14716.095193] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [14716.096781] obdecho(O) [14716.097668] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14716.100052] mgc(O) [14716.101387] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [14716.103778] mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14716.103839] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14716.103846] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14716.103849] RIP: 0010:native_safe_halt+0x2e/0x50 [14716.103863] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14716.103868] RSP: 0018:ffffaf9fc1543ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14716.103872] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [14716.103874] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c20b4154 [14716.103876] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [14716.103878] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [14716.103880] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [14716.103883] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [14716.103886] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14716.103888] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [14716.103896] Call Trace: [14716.103902] [14716.103908] ? show_regs.cold.9+0x22/0x2f [14716.103918] ? watchdog_timer_fn.cold.6+0x62/0x10a [14716.103925] ? __hrtimer_run_queues+0x1c2/0x4d0 [14716.103929] ? watchdog+0x40/0x40 [14716.103934] ? hrtimer_interrupt+0x128/0x2c0 [14716.103938] ? smp_apic_timer_interrupt+0xa6/0x270 [14716.103945] ? apic_timer_interrupt+0xf/0x20 [14716.103950] [14716.103952] ? native_safe_halt+0x2e/0x50 [14716.103956] kvm_wait+0xcd/0xe0 [14716.103964] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14716.103972] do_raw_spin_lock+0x177/0x190 [14716.103977] _raw_spin_lock+0x21/0x30 [14716.103981] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14716.104259] ? do_raw_spin_unlock+0x75/0x190 [14716.104265] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14716.104463] ? do_wait_intr+0xf0/0xf0 [14716.104471] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14716.106153] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14716.106160] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [14716.106180] Call Trace: [14716.106185] [14716.106188] ? show_regs.cold.9+0x22/0x2f [14716.106206] ? watchdog_timer_fn.cold.6+0x62/0x10a [14716.106211] ? __hrtimer_run_queues+0x1c2/0x4d0 [14716.106214] ? watchdog+0x40/0x40 [14716.106216] ? hrtimer_interrupt+0x128/0x2c0 [14716.106219] ? smp_apic_timer_interrupt+0xa6/0x270 [14716.106225] ? apic_timer_interrupt+0xf/0x20 [14716.106229] [14716.106231] ? native_safe_halt+0x2e/0x50 [14716.106234] kvm_wait+0xcd/0xe0 [14716.106240] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14716.106245] do_raw_spin_lock+0x177/0x190 [14716.108780] mdc_close+0x53b/0xd70 [mdc] [14716.134936] _raw_spin_lock+0x21/0x30 [14716.139843] lmv_close+0x18d/0x490 [lmv] [14716.143296] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14716.158346] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [14716.166370] obd_mod_free+0x46/0x140 [ptlrpc] [14716.169078] ll_close_inode_openhandle+0x493/0xea0 [lustre] [14716.172919] mdc_commit_open+0x91/0xb0 [mdc] [14716.175151] ll_md_real_close+0xa5/0x2a0 [lustre] [14716.178455] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14716.182740] ll_md_close+0x274/0x800 [lustre] [14716.186482] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14716.186794] after_reply+0xe2c/0x1780 [ptlrpc] [14716.193960] ll_file_release+0x128/0xc50 [lustre] [14716.196641] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14716.197423] __fput+0x10d/0x440 [14716.198264] ptlrpcd+0xa82/0xe40 [ptlrpc] [14716.199743] ____fput+0x12/0x20 [14716.201739] ? do_wait_intr+0xf0/0xf0 [14716.203060] task_work_run+0xc6/0x110 [14716.203077] exit_to_usermode_loop+0x1dd/0x1f0 [14716.203085] do_syscall_64+0x3d6/0x440 [14716.203090] entry_SYSCALL_64_after_hwframe+0x49/0xae [14716.203098] RIP: 0033:0x7ff54af45cd8 [14716.204310] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14716.205988] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [14716.207768] kthread+0x1d1/0x200 [14716.209046] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [14716.209057] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [14716.209059] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [14716.209062] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [14716.209064] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [14716.209066] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14716.289005] ? set_kthread_struct+0x70/0x70 [14716.290033] ret_from_fork+0x1f/0x30 [14719.461022] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12760s! [14719.472961] Showing busy workqueues and worker pools: [14719.485563] workqueue events: flags=0x0 [14719.491165] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14719.502933] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [14719.514273] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [14719.520381] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [14719.529666] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14719.533680] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [14719.541396] workqueue events_long: flags=0x0 [14719.544850] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14719.549619] pending: lu_objects_flush [obdclass] [14719.554405] workqueue events_unbound: flags=0x2 [14719.556962] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [14719.560931] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [14719.569102] workqueue events_power_efficient: flags=0x80 [14719.572299] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14719.576100] pending: neigh_periodic_work [14719.579382] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14719.583399] pending: do_cache_clean, check_lifetime [14719.586628] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14719.590236] pending: neigh_periodic_work [14719.592412] workqueue writeback: flags=0x4a [14719.595046] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [14719.599724] pending: wb_workfn, wb_workfn [14719.602840] workqueue xprtiod: flags=0xa [14719.604966] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [14719.608969] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [14719.614240] workqueue ipv6_addrconf: flags=0x40008 [14719.616895] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14719.621024] pending: addrconf_verify_work [14719.622232] workqueue knbd0-recv: flags=0x1a [14719.625147] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14719.629510] in-flight: 657:recv_work [14719.631901] workqueue ptlrpc_pinger: flags=0x2 [14719.634891] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14719.638169] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [14719.641681] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12760s workers=4 idle: 20831 12642 28859 [14719.646962] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14709s workers=2 idle: 659 [14724.063645] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [14724.095776] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14724.142320] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14724.158850] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14724.170531] RIP: 0010:native_safe_halt+0x2e/0x50 [14724.176204] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14724.220591] RSP: 0000:ffffaf9fc15234f0 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [14724.229532] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [14724.235392] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c21b4154 [14724.239189] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [14724.245984] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [14724.249897] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [14724.258158] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [14724.267826] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14724.274718] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [14724.279758] Call Trace: [14724.280900] [14724.282608] ? show_regs.cold.9+0x22/0x2f [14724.293463] ? watchdog_timer_fn.cold.6+0x62/0x10a [14724.295115] ? __hrtimer_run_queues+0x1c2/0x4d0 [14724.305391] ? watchdog+0x40/0x40 [14724.311204] ? hrtimer_interrupt+0x128/0x2c0 [14724.320990] ? smp_apic_timer_interrupt+0xa6/0x270 [14724.326806] ? apic_timer_interrupt+0xf/0x20 [14724.330812] [14724.331556] ? native_safe_halt+0x2e/0x50 [14724.340686] kvm_wait+0xcd/0xe0 [14724.342747] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14724.345626] do_raw_spin_lock+0x177/0x190 [14724.356308] _raw_spin_lock+0x21/0x30 [14724.359813] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14724.365289] ? do_raw_spin_unlock+0x75/0x190 [14724.366768] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14724.374870] ? do_wait_intr+0xf0/0xf0 [14724.376835] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14724.384840] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [14724.394044] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14724.396255] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14724.410688] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14724.418221] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [14724.421576] mdc_intent_lock+0x353/0x7e0 [mdc] [14724.425615] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14724.431385] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14724.435693] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14724.437665] lmv_intent_open+0x338/0x1150 [lmv] [14724.443572] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14724.449061] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14724.459718] lmv_intent_lock+0x1dd/0x4c0 [lmv] [14724.468646] ll_intent_lock+0xc5/0xaa0 [lustre] [14724.476411] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14724.484518] ll_lookup_it+0xe0e/0x2270 [lustre] [14724.494008] ? do_raw_spin_unlock+0x75/0x190 [14724.497494] ? _raw_spin_unlock+0x12/0x30 [14724.504937] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [14724.506726] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [14724.512555] ll_atomic_open+0x36a/0xd70 [lustre] [14724.516483] lookup_open+0x356/0xb40 [14724.521619] do_last+0x313/0xfc0 [14724.522961] ? nd_jump_root+0xe5/0x160 [14724.524628] ? path_init+0x437/0x520 [14724.527157] path_openat+0xf7/0x500 [14724.529839] ? do_raw_spin_unlock+0x75/0x190 [14724.531880] do_filp_open+0x99/0x140 [14724.533190] ? getname_flags+0x6e/0x330 [14724.534896] ? __check_object_size+0xff/0x256 [14724.536447] ? do_raw_spin_unlock+0x75/0x190 [14724.538685] ? _raw_spin_unlock+0x12/0x30 [14724.541290] do_sys_openat2+0x2b4/0x410 [14724.547776] do_sys_open+0x73/0xa0 [14724.552010] __x64_sys_openat+0x24/0x30 [14724.554760] do_syscall_64+0xc1/0x440 [14724.556290] entry_SYSCALL_64_after_hwframe+0x49/0xae [14724.558511] RIP: 0033:0x7ff54af45332 [14724.560762] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [14724.573282] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [14724.580178] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [14724.584071] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [14724.593310] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [14724.596427] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [14724.599823] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [14740.063742] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [14740.072234] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14740.102199] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14740.121989] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14740.129629] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14740.139110] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14740.148204] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14740.157780] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [14740.163732] RAX: 000000000000478f RBX: ffff96848811cae0 RCX: 0000000000000001 [14740.166637] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [14740.180760] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [14740.186306] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [14740.191428] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [14740.198641] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [14740.205932] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14740.213416] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [14740.223865] Call Trace: [14740.225140] [14740.227546] ? show_regs.cold.9+0x22/0x2f [14740.233624] ? watchdog_timer_fn.cold.6+0x62/0x10a [14740.238523] ? __hrtimer_run_queues+0x1c2/0x4d0 [14740.244780] ? watchdog+0x40/0x40 [14740.245924] ? hrtimer_interrupt+0x128/0x2c0 [14740.249381] ? smp_apic_timer_interrupt+0xa6/0x270 [14740.256251] ? apic_timer_interrupt+0xf/0x20 [14740.260721] [14740.263796] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14740.266696] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14740.270064] do_raw_spin_lock+0x177/0x190 [14740.272176] _raw_spin_lock+0x21/0x30 [14740.274793] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14740.279565] process_one_work+0x2c8/0x700 [14740.287667] worker_thread+0x296/0x6e0 [14740.290093] ? rescuer_thread+0x570/0x570 [14740.294261] kthread+0x1d1/0x200 [14740.296047] ? set_kthread_struct+0x70/0x70 [14740.297370] ret_from_fork+0x1f/0x30 [14744.055173] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [14744.058014] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [14744.060934] Modules linked in: [14744.065615] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14744.067902] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) [14744.106213] [14744.106229] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14744.116614] fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14744.116676] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14744.116685] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14744.116688] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [14744.117185] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14744.117190] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14744.117200] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14744.117205] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14744.117210] RAX: 0000000000001aac RBX: ffff96848811cae0 RCX: 0000000000000001 [14744.117212] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [14744.117215] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [14744.117217] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [14744.117220] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [14744.117223] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [14744.117226] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14744.117229] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [14744.117237] Call Trace: [14744.117243] [14744.117248] ? show_regs.cold.9+0x22/0x2f [14744.117258] ? watchdog_timer_fn.cold.6+0x62/0x10a [14744.117264] ? __hrtimer_run_queues+0x1c2/0x4d0 [14744.117268] ? watchdog+0x40/0x40 [14744.117272] ? hrtimer_interrupt+0x128/0x2c0 [14744.117277] ? smp_apic_timer_interrupt+0xa6/0x270 [14744.117284] ? apic_timer_interrupt+0xf/0x20 [14744.117289] [14744.117292] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14744.117299] do_raw_spin_lock+0x177/0x190 [14744.117304] _raw_spin_lock+0x21/0x30 [14744.117308] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14744.117873] ? do_raw_spin_unlock+0x75/0x190 [14744.117879] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14744.123918] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [14744.123939] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14744.123946] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [14744.123949] RDX: 00000000000067fc RSI: 0000000000000003 RDI: ffff96848811cae0 [14744.123951] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [14744.123954] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [14744.123956] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14744.123960] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [14744.123963] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14744.124047] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [14744.124057] Call Trace: [14744.124066] [14744.124075] ? show_regs.cold.9+0x22/0x2f [14744.124096] ? watchdog_timer_fn.cold.6+0x62/0x10a [14744.124103] ? __hrtimer_run_queues+0x1c2/0x4d0 [14744.124108] ? watchdog+0x40/0x40 [14744.124112] ? hrtimer_interrupt+0x128/0x2c0 [14744.124116] ? smp_apic_timer_interrupt+0xa6/0x270 [14744.124127] ? apic_timer_interrupt+0xf/0x20 [14744.124132] [14744.124135] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [14744.124145] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14744.124149] do_raw_spin_lock+0x177/0x190 [14744.124155] _raw_spin_lock+0x21/0x30 [14744.124165] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14744.159955] ? do_wait_intr+0xf0/0xf0 [14744.159981] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14744.172494] obd_mod_free+0x46/0x140 [ptlrpc] [14744.172896] mdc_commit_open+0x91/0xb0 [mdc] [14744.172948] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14744.173157] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14744.173810] after_reply+0xe2c/0x1780 [ptlrpc] [14744.174123] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14744.174455] ptlrpcd+0xa82/0xe40 [ptlrpc] [14744.174701] ? do_wait_intr+0xf0/0xf0 [14744.174713] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14744.195130] mdc_close+0x53b/0xd70 [mdc] [14744.199637] kthread+0x1d1/0x200 [14744.199661] ? set_kthread_struct+0x70/0x70 [14744.199666] ret_from_fork+0x1f/0x30 [14744.604955] lmv_close+0x18d/0x490 [lmv] [14744.610063] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [14744.613337] ll_close_inode_openhandle+0x493/0xea0 [lustre] [14744.615384] ll_md_real_close+0xa5/0x2a0 [lustre] [14744.619552] ll_md_close+0x274/0x800 [lustre] [14744.621163] ll_file_release+0x128/0xc50 [lustre] [14744.626917] __fput+0x10d/0x440 [14744.628800] ____fput+0x12/0x20 [14744.630424] task_work_run+0xc6/0x110 [14744.633766] exit_to_usermode_loop+0x1dd/0x1f0 [14744.638581] do_syscall_64+0x3d6/0x440 [14744.640829] entry_SYSCALL_64_after_hwframe+0x49/0xae [14744.644294] RIP: 0033:0x7ff54af45cd8 [14744.645579] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [14744.657515] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [14744.661711] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [14744.668577] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [14744.672762] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [14744.676211] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [14744.684867] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14749.665791] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12790s! [14749.678749] Showing busy workqueues and worker pools: [14749.681439] workqueue events: flags=0x0 [14749.684397] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14749.692986] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [14749.702906] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [14749.714314] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [14749.720682] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14749.726631] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [14749.731752] workqueue events_long: flags=0x0 [14749.734862] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14749.739597] pending: lu_objects_flush [obdclass] [14749.743839] workqueue events_unbound: flags=0x2 [14749.751110] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [14749.760091] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [14749.767939] workqueue events_power_efficient: flags=0x80 [14749.773424] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14749.783157] pending: neigh_periodic_work [14749.787685] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14749.792206] pending: do_cache_clean, check_lifetime [14749.794619] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14749.801050] pending: neigh_periodic_work [14749.803684] workqueue writeback: flags=0x4a [14749.806880] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [14749.812931] pending: wb_workfn, wb_workfn [14749.818109] workqueue xprtiod: flags=0xa [14749.820931] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [14749.823695] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [14749.831945] workqueue ipv6_addrconf: flags=0x40008 [14749.837643] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14749.847164] pending: addrconf_verify_work [14749.852923] workqueue knbd0-recv: flags=0x1a [14749.858585] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14749.866410] in-flight: 657:recv_work [14749.868107] workqueue ptlrpc_pinger: flags=0x2 [14749.871620] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14749.881561] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [14749.890694] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12791s workers=4 idle: 20831 12642 28859 [14749.898832] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14740s workers=2 idle: 659 [14752.060070] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [14752.065722] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14752.116476] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14752.133869] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14752.146749] RIP: 0010:native_safe_halt+0x2e/0x50 [14752.153299] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14752.192127] RSP: 0000:ffffaf9fc15234f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14752.207303] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [14752.223498] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c21b4154 [14752.232403] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [14752.238425] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [14752.242559] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [14752.251683] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [14752.264606] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14752.270438] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [14752.279485] Call Trace: [14752.281668] [14752.283940] ? show_regs.cold.9+0x22/0x2f [14752.287873] ? watchdog_timer_fn.cold.6+0x62/0x10a [14752.294937] ? __hrtimer_run_queues+0x1c2/0x4d0 [14752.298465] ? watchdog+0x40/0x40 [14752.303536] ? hrtimer_interrupt+0x128/0x2c0 [14752.306783] ? smp_apic_timer_interrupt+0xa6/0x270 [14752.309557] ? apic_timer_interrupt+0xf/0x20 [14752.311070] [14752.311888] ? native_safe_halt+0x2e/0x50 [14752.320674] kvm_wait+0xcd/0xe0 [14752.324135] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14752.328839] do_raw_spin_lock+0x177/0x190 [14752.335305] _raw_spin_lock+0x21/0x30 [14752.339783] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14752.353446] ? do_raw_spin_unlock+0x75/0x190 [14752.363346] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14752.372250] ? do_wait_intr+0xf0/0xf0 [14752.375129] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14752.389931] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [14752.397847] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14752.407900] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14752.418531] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14752.429477] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [14752.449884] mdc_intent_lock+0x353/0x7e0 [mdc] [14752.460188] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14752.473600] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14752.483369] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14752.490496] lmv_intent_open+0x338/0x1150 [lmv] [14752.497414] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14752.501396] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14752.509434] lmv_intent_lock+0x1dd/0x4c0 [lmv] [14752.516467] ll_intent_lock+0xc5/0xaa0 [lustre] [14752.520411] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14752.524348] ll_lookup_it+0xe0e/0x2270 [lustre] [14752.527865] ? do_raw_spin_unlock+0x75/0x190 [14752.532183] ? _raw_spin_unlock+0x12/0x30 [14752.539173] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [14752.542813] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [14752.544549] ll_atomic_open+0x36a/0xd70 [lustre] [14752.547972] lookup_open+0x356/0xb40 [14752.551290] do_last+0x313/0xfc0 [14752.553431] ? nd_jump_root+0xe5/0x160 [14752.556135] ? path_init+0x437/0x520 [14752.558374] path_openat+0xf7/0x500 [14752.561504] ? do_raw_spin_unlock+0x75/0x190 [14752.564946] do_filp_open+0x99/0x140 [14752.570560] ? getname_flags+0x6e/0x330 [14752.576738] ? __check_object_size+0xff/0x256 [14752.585776] ? do_raw_spin_unlock+0x75/0x190 [14752.589840] ? _raw_spin_unlock+0x12/0x30 [14752.592545] do_sys_openat2+0x2b4/0x410 [14752.596540] do_sys_open+0x73/0xa0 [14752.599558] __x64_sys_openat+0x24/0x30 [14752.601877] do_syscall_64+0xc1/0x440 [14752.603169] entry_SYSCALL_64_after_hwframe+0x49/0xae [14752.607788] RIP: 0033:0x7ff54af45332 [14752.610424] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [14752.619571] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [14752.626140] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [14752.638902] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [14752.648442] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [14752.656576] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [14752.664050] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [14768.059170] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [14768.064624] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14768.093838] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14768.100682] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14768.106160] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14768.110411] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14768.114246] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14768.126255] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14768.131993] RAX: 000000000000130e RBX: ffff96848811cae0 RCX: 0000000000000001 [14768.136121] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [14768.140887] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [14768.146405] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [14768.151891] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [14768.157806] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [14768.161128] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14768.164034] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [14768.167681] Call Trace: [14768.168868] [14768.169877] ? show_regs.cold.9+0x22/0x2f [14768.171950] ? watchdog_timer_fn.cold.6+0x62/0x10a [14768.174536] ? __hrtimer_run_queues+0x1c2/0x4d0 [14768.176753] ? watchdog+0x40/0x40 [14768.178538] ? hrtimer_interrupt+0x128/0x2c0 [14768.180597] ? smp_apic_timer_interrupt+0xa6/0x270 [14768.182832] ? apic_timer_interrupt+0xf/0x20 [14768.184864] [14768.186269] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14768.189054] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14768.191856] do_raw_spin_lock+0x177/0x190 [14768.194521] _raw_spin_lock+0x21/0x30 [14768.196208] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14768.199826] process_one_work+0x2c8/0x700 [14768.201834] worker_thread+0x296/0x6e0 [14768.204184] ? rescuer_thread+0x570/0x570 [14768.206358] kthread+0x1d1/0x200 [14768.207720] ? set_kthread_struct+0x70/0x70 [14768.209705] ret_from_fork+0x1f/0x30 [14772.058124] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [14772.064887] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14772.113297] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14772.131005] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14772.146599] RIP: 0010:native_safe_halt+0x2e/0x50 [14772.151955] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14772.168630] RSP: 0018:ffffaf9fc1543ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14772.178100] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [14772.194091] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c20b4154 [14772.208656] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [14772.224626] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [14772.238582] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [14772.241783] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [14772.247758] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14772.250618] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [14772.255734] Call Trace: [14772.257367] [14772.258713] ? show_regs.cold.9+0x22/0x2f [14772.261625] ? watchdog_timer_fn.cold.6+0x62/0x10a [14772.265095] ? __hrtimer_run_queues+0x1c2/0x4d0 [14772.268374] ? watchdog+0x40/0x40 [14772.270626] ? hrtimer_interrupt+0x128/0x2c0 [14772.273468] ? smp_apic_timer_interrupt+0xa6/0x270 [14772.277357] ? apic_timer_interrupt+0xf/0x20 [14772.280824] [14772.282152] ? native_safe_halt+0x2e/0x50 [14772.284775] kvm_wait+0xcd/0xe0 [14772.286637] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14772.291735] do_raw_spin_lock+0x177/0x190 [14772.294646] _raw_spin_lock+0x21/0x30 [14772.297091] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14772.301318] ? do_raw_spin_unlock+0x75/0x190 [14772.304196] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14772.308855] ? do_wait_intr+0xf0/0xf0 [14772.311938] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14772.316441] mdc_close+0x53b/0xd70 [mdc] [14772.319301] lmv_close+0x18d/0x490 [lmv] [14772.324790] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [14772.331755] ll_close_inode_openhandle+0x493/0xea0 [lustre] [14772.339123] ll_md_real_close+0xa5/0x2a0 [lustre] [14772.346213] ll_md_close+0x274/0x800 [lustre] [14772.351205] ll_file_release+0x128/0xc50 [lustre] [14772.357229] __fput+0x10d/0x440 [14772.359822] ____fput+0x12/0x20 [14772.362504] task_work_run+0xc6/0x110 [14772.365025] exit_to_usermode_loop+0x1dd/0x1f0 [14772.368627] do_syscall_64+0x3d6/0x440 [14772.371788] entry_SYSCALL_64_after_hwframe+0x49/0xae [14772.375798] RIP: 0033:0x7ff54af45cd8 [14772.379081] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [14772.392188] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [14772.397396] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [14772.407392] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [14772.416639] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [14772.423306] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [14772.428803] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14776.055107] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [14776.069157] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14776.122149] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14776.128472] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14776.135185] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [14776.139907] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [14776.151750] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14776.157056] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [14776.165702] RDX: 000000000000308d RSI: 0000000000000003 RDI: ffff96848811cae0 [14776.175901] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [14776.186399] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [14776.195229] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14776.201593] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [14776.208603] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14776.213410] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [14776.219883] Call Trace: [14776.222284] [14776.223675] ? show_regs.cold.9+0x22/0x2f [14776.227214] ? watchdog_timer_fn.cold.6+0x62/0x10a [14776.231099] ? __hrtimer_run_queues+0x1c2/0x4d0 [14776.234937] ? watchdog+0x40/0x40 [14776.238300] ? hrtimer_interrupt+0x128/0x2c0 [14776.246337] ? smp_apic_timer_interrupt+0xa6/0x270 [14776.252007] ? apic_timer_interrupt+0xf/0x20 [14776.255885] [14776.258644] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [14776.263157] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14776.266260] do_raw_spin_lock+0x177/0x190 [14776.269578] _raw_spin_lock+0x21/0x30 [14776.277629] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14776.281513] obd_mod_free+0x46/0x140 [ptlrpc] [14776.289210] mdc_commit_open+0x91/0xb0 [mdc] [14776.294520] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14776.299073] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14776.302388] after_reply+0xe2c/0x1780 [ptlrpc] [14776.305301] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14776.307402] ptlrpcd+0xa82/0xe40 [ptlrpc] [14776.312680] ? do_wait_intr+0xf0/0xf0 [14776.314617] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14776.320533] kthread+0x1d1/0x200 [14776.322556] ? set_kthread_struct+0x70/0x70 [14776.326161] ret_from_fork+0x1f/0x30 [14780.060069] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [14780.064610] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14780.100743] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14780.107225] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14780.112213] RIP: 0010:native_safe_halt+0x2e/0x50 [14780.114676] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14780.137891] RSP: 0000:ffffaf9fc15234f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14780.143199] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [14780.146107] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [14780.153717] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [14780.164558] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [14780.176234] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [14780.180228] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [14780.185468] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14780.188405] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [14780.192212] Call Trace: [14780.193532] [14780.194637] ? show_regs.cold.9+0x22/0x2f [14780.196699] ? watchdog_timer_fn.cold.6+0x62/0x10a [14780.199174] ? __hrtimer_run_queues+0x1c2/0x4d0 [14780.200659] ? watchdog+0x40/0x40 [14780.201548] ? hrtimer_interrupt+0x128/0x2c0 [14780.203284] ? smp_apic_timer_interrupt+0xa6/0x270 [14780.205408] ? apic_timer_interrupt+0xf/0x20 [14780.207281] [14780.208503] ? native_safe_halt+0x2e/0x50 [14780.210485] kvm_wait+0xcd/0xe0 [14780.212834] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14780.215630] do_raw_spin_lock+0x177/0x190 [14780.217625] _raw_spin_lock+0x21/0x30 [14780.219806] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14780.222940] ? do_raw_spin_unlock+0x75/0x190 [14780.229691] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14780.234188] ? do_wait_intr+0xf0/0xf0 [14780.236115] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14780.238262] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [14780.241159] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14780.243811] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14780.245802] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14780.247729] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [14780.249558] mdc_intent_lock+0x353/0x7e0 [mdc] [14780.251315] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14780.253259] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14780.255908] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14780.257290] lmv_intent_open+0x338/0x1150 [lmv] [14780.258862] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14780.264722] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14780.269667] lmv_intent_lock+0x1dd/0x4c0 [lmv] [14780.273539] ll_intent_lock+0xc5/0xaa0 [lustre] [14780.283712] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14780.294186] ll_lookup_it+0xe0e/0x2270 [lustre] [14780.295892] ? do_raw_spin_unlock+0x75/0x190 [14780.300807] ? _raw_spin_unlock+0x12/0x30 [14780.302686] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [14780.304414] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [14780.305668] ll_atomic_open+0x36a/0xd70 [lustre] [14780.306952] lookup_open+0x356/0xb40 [14780.310020] do_last+0x313/0xfc0 [14780.311691] ? nd_jump_root+0xe5/0x160 [14780.314274] ? path_init+0x437/0x520 [14780.316408] path_openat+0xf7/0x500 [14780.318550] ? do_raw_spin_unlock+0x75/0x190 [14780.321320] do_filp_open+0x99/0x140 [14780.323296] ? getname_flags+0x6e/0x330 [14780.325720] ? __check_object_size+0xff/0x256 [14780.328698] ? do_raw_spin_unlock+0x75/0x190 [14780.332531] ? _raw_spin_unlock+0x12/0x30 [14780.343607] do_sys_openat2+0x2b4/0x410 [14780.350427] do_sys_open+0x73/0xa0 [14780.356719] __x64_sys_openat+0x24/0x30 [14780.359538] do_syscall_64+0xc1/0x440 [14780.364746] entry_SYSCALL_64_after_hwframe+0x49/0xae [14780.370483] RIP: 0033:0x7ff54af45332 [14780.371594] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [14780.382527] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [14780.385183] BUG: workqueue lockup - pool [14780.390523] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [14780.390535] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [14780.390538] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [14780.390541] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [14780.390543] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [14780.458074] cpus=0-3 flags=0x4 nice=0 stuck for 12821s! [14780.468280] Showing busy workqueues and worker pools: [14780.478068] workqueue events: flags=0x0 [14780.486543] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14780.498741] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [14780.509736] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [14780.521466] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [14780.536509] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14780.546644] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [14780.553513] workqueue events_long: flags=0x0 [14780.556876] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14780.567820] pending: lu_objects_flush [obdclass] [14780.577680] workqueue events_unbound: flags=0x2 [14780.585237] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [14780.589413] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [14780.612499] workqueue events_power_efficient: flags=0x80 [14780.622789] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14780.633210] pending: neigh_periodic_work [14780.642483] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14780.652633] pending: do_cache_clean, check_lifetime [14780.659502] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14780.667887] pending: neigh_periodic_work [14780.672505] workqueue writeback: flags=0x4a [14780.675521] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [14780.683647] pending: wb_workfn, wb_workfn [14780.689827] workqueue xprtiod: flags=0xa [14780.697867] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [14780.709549] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [14780.722699] workqueue ipv6_addrconf: flags=0x40008 [14780.725104] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14780.734677] pending: addrconf_verify_work [14780.741968] workqueue knbd0-recv: flags=0x1a [14780.748441] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14780.762867] in-flight: 657:recv_work [14780.774166] workqueue ptlrpc_pinger: flags=0x2 [14780.786188] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14780.798553] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [14780.811561] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12821s workers=4 idle: 20831 12642 28859 [14780.819854] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14771s workers=2 idle: 659 [14794.185652] rcu: INFO: rcu_sched self-detected stall on CPU [14794.196431] rcu: 0-...!: (6831582 ticks this GP) idle=98a/1/0x4000000000000002 softirq=125154/125154 fqs=959 [14794.213514] (t=12840345 jiffies g=253585 q=7056) [14794.216178] rcu: rcu_sched kthread starved for 12835305 jiffies! g253585 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [14794.230815] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [14794.242075] rcu: RCU grace-period kthread stack dump: [14794.249807] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [14794.266647] Call Trace: [14794.274075] __schedule+0x351/0xcb0 [14794.283487] schedule+0xc0/0x180 [14794.284556] schedule_timeout+0xb4/0x190 [14794.285806] ? do_raw_spin_unlock+0x75/0x190 [14794.304297] ? __next_timer_interrupt+0x160/0x160 [14794.311547] ? prepare_to_swait_event+0xf3/0x140 [14794.319167] rcu_gp_kthread+0x8dd/0x15b0 [14794.320524] ? force_qs_rnp+0x2c0/0x2c0 [14794.330661] kthread+0x1d1/0x200 [14794.332998] ? set_kthread_struct+0x70/0x70 [14794.335644] ret_from_fork+0x1f/0x30 [14794.337690] rcu: Stack dump where RCU GP kthread last ran: [14794.341591] Sending NMI from CPU 0 to CPUs 2: [14794.348552] NMI backtrace for cpu 2 [14794.348560] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14794.348563] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14794.348564] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14794.348568] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14794.348569] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14794.348572] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 [14794.348574] RAX: 0000000000006e8d RBX: ffff96848811cae0 RCX: 0000000000000001 [14794.348575] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [14794.348577] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [14794.348578] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [14794.348579] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [14794.348581] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [14794.348582] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14794.348583] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [14794.348584] Call Trace: [14794.348585] [14794.348586] ? show_regs.cold.9+0x22/0x2f [14794.348587] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [14794.348588] ? nmi_cpu_backtrace_handler+0x15/0x40 [14794.348590] ? nmi_handle+0xbd/0x210 [14794.348591] ? default_do_nmi+0x4b/0x1b0 [14794.348593] ? do_nmi+0x280/0x380 [14794.348594] ? end_repeat_nmi+0x16/0x1f [14794.348595] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14794.348596] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14794.348598] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14794.348599] [14794.348600] do_raw_spin_lock+0x177/0x190 [14794.348601] _raw_spin_lock+0x21/0x30 [14794.348602] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14794.348603] process_one_work+0x2c8/0x700 [14794.348604] worker_thread+0x296/0x6e0 [14794.348605] ? rescuer_thread+0x570/0x570 [14794.348606] kthread+0x1d1/0x200 [14794.348607] ? set_kthread_struct+0x70/0x70 [14794.348688] ret_from_fork+0x1f/0x30 [14794.350397] NMI backtrace for cpu 0 [14794.505737] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14794.514773] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14794.524466] Call Trace: [14794.525693] [14794.528017] ? dump_stack+0xbb/0x10e [14794.532687] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [14794.535931] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [14794.540683] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [14794.545420] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [14794.549942] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [14794.555474] ? rcu_dump_cpu_stacks+0x11b/0x171 [14794.557296] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [14794.562538] ? update_process_times+0x6d/0xc0 [14794.563900] ? tick_sched_handle.isra.18+0x27/0x90 [14794.569487] ? tick_sched_timer+0x53/0xb0 [14794.572263] ? __hrtimer_run_queues+0x1c2/0x4d0 [14794.577215] ? tick_sched_handle.isra.18+0x90/0x90 [14794.580857] ? hrtimer_interrupt+0x128/0x2c0 [14794.582635] ? smp_apic_timer_interrupt+0xa6/0x270 [14794.584872] ? apic_timer_interrupt+0xf/0x20 [14794.586812] [14794.588033] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [14794.590497] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14794.592832] ? do_raw_spin_lock+0x177/0x190 [14794.594721] ? _raw_spin_lock+0x21/0x30 [14794.596461] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14794.602100] ? obd_mod_free+0x46/0x140 [ptlrpc] [14794.608255] ? mdc_commit_open+0x91/0xb0 [mdc] [14794.614701] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14794.630408] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14794.647283] ? after_reply+0xe2c/0x1780 [ptlrpc] [14794.655625] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14794.665082] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [14794.668517] ? do_wait_intr+0xf0/0xf0 [14794.672235] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14794.676602] ? kthread+0x1d1/0x200 [14794.680100] ? set_kthread_struct+0x70/0x70 [14794.683211] ? ret_from_fork+0x1f/0x30 [14800.068263] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [14800.078237] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14800.122531] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14800.137075] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14800.146225] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14800.158477] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14800.168322] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14800.171045] RAX: 0000000000007c55 RBX: ffff96848811cae0 RCX: 0000000000000001 [14800.173410] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [14800.175463] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [14800.178524] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [14800.180283] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [14800.184494] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [14800.188404] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14800.192571] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [14800.196942] Call Trace: [14800.197689] [14800.198346] ? show_regs.cold.9+0x22/0x2f [14800.199490] ? watchdog_timer_fn.cold.6+0x62/0x10a [14800.204119] ? __hrtimer_run_queues+0x1c2/0x4d0 [14800.208146] ? watchdog+0x40/0x40 [14800.209469] ? hrtimer_interrupt+0x128/0x2c0 [14800.211572] ? smp_apic_timer_interrupt+0xa6/0x270 [14800.214370] ? apic_timer_interrupt+0xf/0x20 [14800.216943] [14800.217465] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14800.221287] do_raw_spin_lock+0x177/0x190 [14800.223256] _raw_spin_lock+0x21/0x30 [14800.224513] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14800.227628] ? do_raw_spin_unlock+0x75/0x190 [14800.231310] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14800.236007] ? do_wait_intr+0xf0/0xf0 [14800.239055] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14800.241256] mdc_close+0x53b/0xd70 [mdc] [14800.245596] lmv_close+0x18d/0x490 [lmv] [14800.249307] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [14800.252650] ll_close_inode_openhandle+0x493/0xea0 [lustre] [14800.255697] ll_md_real_close+0xa5/0x2a0 [lustre] [14800.257118] ll_md_close+0x274/0x800 [lustre] [14800.258470] ll_file_release+0x128/0xc50 [lustre] [14800.264349] __fput+0x10d/0x440 [14800.269075] ____fput+0x12/0x20 [14800.270657] task_work_run+0xc6/0x110 [14800.272518] exit_to_usermode_loop+0x1dd/0x1f0 [14800.274343] do_syscall_64+0x3d6/0x440 [14800.275933] entry_SYSCALL_64_after_hwframe+0x49/0xae [14800.277308] RIP: 0033:0x7ff54af45cd8 [14800.278493] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [14800.298661] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [14800.304768] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [14800.312221] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [14800.321212] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [14800.328606] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [14800.342654] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14808.062900] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [14808.071660] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14808.121320] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14808.130438] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14808.141112] RIP: 0010:native_safe_halt+0x2e/0x50 [14808.143790] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14808.157441] RSP: 0000:ffffaf9fc15234f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14808.162739] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [14808.167282] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c21b4154 [14808.173760] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [14808.178176] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [14808.183143] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [14808.189151] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [14808.194838] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14808.199377] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [14808.204115] Call Trace: [14808.205732] [14808.207107] ? show_regs.cold.9+0x22/0x2f [14808.209877] ? watchdog_timer_fn.cold.6+0x62/0x10a [14808.213768] ? __hrtimer_run_queues+0x1c2/0x4d0 [14808.218044] ? watchdog+0x40/0x40 [14808.219728] ? hrtimer_interrupt+0x128/0x2c0 [14808.222885] ? smp_apic_timer_interrupt+0xa6/0x270 [14808.226495] ? apic_timer_interrupt+0xf/0x20 [14808.228651] [14808.229735] ? native_safe_halt+0x2e/0x50 [14808.232658] kvm_wait+0xcd/0xe0 [14808.234254] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14808.237860] do_raw_spin_lock+0x177/0x190 [14808.240472] _raw_spin_lock+0x21/0x30 [14808.242281] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14808.247371] ? do_raw_spin_unlock+0x75/0x190 [14808.250100] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14808.253046] ? do_wait_intr+0xf0/0xf0 [14808.255423] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14808.259189] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [14808.264122] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14808.268081] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14808.271455] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14808.278925] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [14808.283412] mdc_intent_lock+0x353/0x7e0 [mdc] [14808.286416] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14808.290601] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14808.294799] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14808.298493] lmv_intent_open+0x338/0x1150 [lmv] [14808.300686] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14808.304502] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14808.308160] lmv_intent_lock+0x1dd/0x4c0 [lmv] [14808.310943] ll_intent_lock+0xc5/0xaa0 [lustre] [14808.314568] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14808.317590] ll_lookup_it+0xe0e/0x2270 [lustre] [14808.320581] ? do_raw_spin_unlock+0x75/0x190 [14808.329184] ? _raw_spin_unlock+0x12/0x30 [14808.332123] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [14808.340032] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [14808.343681] ll_atomic_open+0x36a/0xd70 [lustre] [14808.346104] lookup_open+0x356/0xb40 [14808.348686] do_last+0x313/0xfc0 [14808.351164] ? nd_jump_root+0xe5/0x160 [14808.355878] ? path_init+0x437/0x520 [14808.359863] path_openat+0xf7/0x500 [14808.366156] ? do_raw_spin_unlock+0x75/0x190 [14808.368410] do_filp_open+0x99/0x140 [14808.370698] ? getname_flags+0x6e/0x330 [14808.373090] ? __check_object_size+0xff/0x256 [14808.375227] ? do_raw_spin_unlock+0x75/0x190 [14808.379574] ? _raw_spin_unlock+0x12/0x30 [14808.384362] do_sys_openat2+0x2b4/0x410 [14808.388773] do_sys_open+0x73/0xa0 [14808.391373] __x64_sys_openat+0x24/0x30 [14808.395566] do_syscall_64+0xc1/0x440 [14808.398575] entry_SYSCALL_64_after_hwframe+0x49/0xae [14808.402833] RIP: 0033:0x7ff54af45332 [14808.411440] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [14808.431041] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [14808.440447] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [14808.450459] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [14808.463938] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [14808.471701] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [14808.478309] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [14811.104479] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12852s! [14811.109861] Showing busy workqueues and worker pools: [14811.112365] workqueue events: flags=0x0 [14811.117589] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14811.132127] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [14811.140467] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [14811.149483] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [14811.174712] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14811.177804] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [14811.182546] workqueue events_long: flags=0x0 [14811.190580] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14811.198370] pending: lu_objects_flush [obdclass] [14811.206512] workqueue events_unbound: flags=0x2 [14811.209384] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [14811.215014] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [14811.232889] workqueue events_power_efficient: flags=0x80 [14811.234608] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14811.239873] pending: neigh_periodic_work [14811.242513] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14811.246113] pending: do_cache_clean, check_lifetime [14811.250797] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14811.255261] pending: neigh_periodic_work [14811.257568] workqueue writeback: flags=0x4a [14811.260035] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [14811.264132] pending: wb_workfn, wb_workfn [14811.269059] workqueue xprtiod: flags=0xa [14811.270823] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [14811.272751] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [14811.276403] workqueue ipv6_addrconf: flags=0x40008 [14811.278099] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14811.284403] pending: addrconf_verify_work [14811.288012] workqueue knbd0-recv: flags=0x1a [14811.292410] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14811.296266] in-flight: 657:recv_work [14811.298366] workqueue ptlrpc_pinger: flags=0x2 [14811.300960] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14811.306265] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [14811.311330] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12852s workers=4 idle: 20831 12642 28859 [14811.319583] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14801s workers=2 idle: 659 [14820.066475] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [14820.069820] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14820.135175] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14820.151882] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14820.165705] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14820.171469] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14820.179617] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14820.198958] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14820.204218] RAX: 000000000000286e RBX: ffff96848811cae0 RCX: 0000000000000001 [14820.211921] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [14820.218381] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [14820.228746] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [14820.235416] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [14820.244359] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [14820.250144] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14820.257161] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [14820.261958] Call Trace: [14820.266049] [14820.271051] ? show_regs.cold.9+0x22/0x2f [14820.272647] ? watchdog_timer_fn.cold.6+0x62/0x10a [14820.280761] ? __hrtimer_run_queues+0x1c2/0x4d0 [14820.284675] ? watchdog+0x40/0x40 [14820.286227] ? hrtimer_interrupt+0x128/0x2c0 [14820.293700] ? smp_apic_timer_interrupt+0xa6/0x270 [14820.298019] ? apic_timer_interrupt+0xf/0x20 [14820.303778] [14820.304445] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14820.310853] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14820.316736] do_raw_spin_lock+0x177/0x190 [14820.319181] _raw_spin_lock+0x21/0x30 [14820.321006] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14820.327087] process_one_work+0x2c8/0x700 [14820.330190] worker_thread+0x296/0x6e0 [14820.331658] ? rescuer_thread+0x570/0x570 [14820.333555] kthread+0x1d1/0x200 [14820.336993] ? set_kthread_struct+0x70/0x70 [14820.341083] ret_from_fork+0x1f/0x30 [14828.061626] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [14828.073128] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14828.112949] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14828.128943] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14828.137154] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14828.140407] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14828.158159] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14828.164759] RAX: 0000000000007d08 RBX: ffff96848811cae0 RCX: 0000000000000001 [14828.167501] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [14828.172535] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [14828.176262] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [14828.180355] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [14828.184571] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [14828.189389] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14828.194276] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [14828.199491] Call Trace: [14828.201213] [14828.202226] ? show_regs.cold.9+0x22/0x2f [14828.204311] ? watchdog_timer_fn.cold.6+0x62/0x10a [14828.206904] ? __hrtimer_run_queues+0x1c2/0x4d0 [14828.209296] ? watchdog+0x40/0x40 [14828.212190] ? hrtimer_interrupt+0x128/0x2c0 [14828.214343] ? smp_apic_timer_interrupt+0xa6/0x270 [14828.223101] ? apic_timer_interrupt+0xf/0x20 [14828.226564] [14828.227583] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14828.233489] do_raw_spin_lock+0x177/0x190 [14828.235421] _raw_spin_lock+0x21/0x30 [14828.240905] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14828.247378] ? do_raw_spin_unlock+0x75/0x190 [14828.249261] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14828.252979] ? do_wait_intr+0xf0/0xf0 [14828.254920] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14828.257847] mdc_close+0x53b/0xd70 [mdc] [14828.260022] lmv_close+0x18d/0x490 [lmv] [14828.261908] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [14828.265137] ll_close_inode_openhandle+0x493/0xea0 [lustre] [14828.268657] ll_md_real_close+0xa5/0x2a0 [lustre] [14828.271212] ll_md_close+0x274/0x800 [lustre] [14828.273693] ll_file_release+0x128/0xc50 [lustre] [14828.276228] __fput+0x10d/0x440 [14828.277662] ____fput+0x12/0x20 [14828.279659] task_work_run+0xc6/0x110 [14828.281330] exit_to_usermode_loop+0x1dd/0x1f0 [14828.283460] do_syscall_64+0x3d6/0x440 [14828.285239] entry_SYSCALL_64_after_hwframe+0x49/0xae [14828.287633] RIP: 0033:0x7ff54af45cd8 [14828.289847] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [14828.303585] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [14828.307980] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [14828.311561] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [14828.315392] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [14828.320258] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [14828.323733] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14836.056761] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [14836.060014] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [14836.065870] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14836.065949] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14836.065957] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14836.065960] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [14836.065973] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [14836.065978] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14836.065983] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [14836.065985] RDX: 000000000000458f RSI: 0000000000000003 RDI: ffff96848811cae0 [14836.065987] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [14836.065990] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [14836.065992] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14836.065995] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [14836.065998] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14836.066000] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [14836.066008] Call Trace: [14836.066013] [14836.066019] ? show_regs.cold.9+0x22/0x2f [14836.066027] ? watchdog_timer_fn.cold.6+0x62/0x10a [14836.066033] ? __hrtimer_run_queues+0x1c2/0x4d0 [14836.066037] ? watchdog+0x40/0x40 [14836.066041] ? hrtimer_interrupt+0x128/0x2c0 [14836.066045] ? smp_apic_timer_interrupt+0xa6/0x270 [14836.066053] ? apic_timer_interrupt+0xf/0x20 [14836.066057] [14836.066060] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [14836.066064] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14836.066068] do_raw_spin_lock+0x177/0x190 [14836.066072] _raw_spin_lock+0x21/0x30 [14836.066077] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14836.079833] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14836.079917] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14836.079926] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14836.079929] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14836.079944] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14836.079949] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14836.079953] RAX: 0000000000004e5c RBX: ffff96848811cae0 RCX: 0000000000000001 [14836.079956] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [14836.079958] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [14836.079960] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [14836.079962] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [14836.079965] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [14836.079968] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14836.079970] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [14836.079979] Call Trace: [14836.079985] [14836.132956] obd_mod_free+0x46/0x140 [ptlrpc] [14836.134649] mdc_commit_open+0x91/0xb0 [mdc] [14836.134691] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14836.134930] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14836.137332] after_reply+0xe2c/0x1780 [ptlrpc] [14836.137876] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14836.138077] ptlrpcd+0xa82/0xe40 [ptlrpc] [14836.158768] ? show_regs.cold.9+0x22/0x2f [14836.162012] ? watchdog_timer_fn.cold.6+0x62/0x10a [14836.162020] ? __hrtimer_run_queues+0x1c2/0x4d0 [14836.162025] ? watchdog+0x40/0x40 [14836.162029] ? hrtimer_interrupt+0x128/0x2c0 [14836.162033] ? smp_apic_timer_interrupt+0xa6/0x270 [14836.162042] ? apic_timer_interrupt+0xf/0x20 [14836.162047] [14836.162053] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14836.162063] do_raw_spin_lock+0x177/0x190 [14836.162071] _raw_spin_lock+0x21/0x30 [14836.162075] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14836.166974] ? do_wait_intr+0xf0/0xf0 [14836.166995] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14836.167591] kthread+0x1d1/0x200 [14836.167599] ? set_kthread_struct+0x70/0x70 [14836.167603] ret_from_fork+0x1f/0x30 [14836.617729] ? do_raw_spin_unlock+0x75/0x190 [14836.621022] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14836.624396] ? do_wait_intr+0xf0/0xf0 [14836.627294] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14836.630908] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [14836.635072] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14836.639205] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14836.644959] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14836.651059] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [14836.657491] mdc_intent_lock+0x353/0x7e0 [mdc] [14836.661809] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14836.663831] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14836.676452] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14836.686182] lmv_intent_open+0x338/0x1150 [lmv] [14836.690711] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14836.693910] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14836.697065] lmv_intent_lock+0x1dd/0x4c0 [lmv] [14836.699622] ll_intent_lock+0xc5/0xaa0 [lustre] [14836.702176] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14836.704233] ll_lookup_it+0xe0e/0x2270 [lustre] [14836.707736] ? do_raw_spin_unlock+0x75/0x190 [14836.709777] ? _raw_spin_unlock+0x12/0x30 [14836.711707] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [14836.714112] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [14836.716522] ll_atomic_open+0x36a/0xd70 [lustre] [14836.719302] lookup_open+0x356/0xb40 [14836.721088] do_last+0x313/0xfc0 [14836.723083] ? nd_jump_root+0xe5/0x160 [14836.724886] ? path_init+0x437/0x520 [14836.726703] path_openat+0xf7/0x500 [14836.728132] ? do_raw_spin_unlock+0x75/0x190 [14836.734588] do_filp_open+0x99/0x140 [14836.735958] ? getname_flags+0x6e/0x330 [14836.737364] ? __check_object_size+0xff/0x256 [14836.741891] ? do_raw_spin_unlock+0x75/0x190 [14836.744262] ? _raw_spin_unlock+0x12/0x30 [14836.746417] do_sys_openat2+0x2b4/0x410 [14836.748222] do_sys_open+0x73/0xa0 [14836.750074] __x64_sys_openat+0x24/0x30 [14836.751886] do_syscall_64+0xc1/0x440 [14836.754116] entry_SYSCALL_64_after_hwframe+0x49/0xae [14836.758021] RIP: 0033:0x7ff54af45332 [14836.762711] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [14836.772158] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [14836.775922] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [14836.779199] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [14836.782868] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [14836.786869] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [14836.793865] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [14841.823210] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12882s! [14841.828302] Showing busy workqueues and worker pools: [14841.830742] workqueue events: flags=0x0 [14841.833251] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14841.843250] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [14841.851333] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [14841.858033] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [14841.877668] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14841.884038] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [14841.896114] workqueue events_long: flags=0x0 [14841.900708] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14841.905843] pending: lu_objects_flush [obdclass] [14841.910675] workqueue events_unbound: flags=0x2 [14841.913382] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [14841.916909] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [14841.931216] workqueue events_power_efficient: flags=0x80 [14841.937729] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14841.942495] pending: neigh_periodic_work [14841.946777] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14841.950754] pending: do_cache_clean, check_lifetime [14841.955936] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14841.965414] pending: neigh_periodic_work [14841.968816] workqueue writeback: flags=0x4a [14841.973682] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [14841.981270] pending: wb_workfn, wb_workfn [14841.989247] workqueue xprtiod: flags=0xa [14841.995620] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [14842.005588] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [14842.013707] workqueue ipv6_addrconf: flags=0x40008 [14842.017947] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14842.023784] pending: addrconf_verify_work [14842.028617] workqueue knbd0-recv: flags=0x1a [14842.031712] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14842.042059] in-flight: 657:recv_work [14842.045907] workqueue ptlrpc_pinger: flags=0x2 [14842.053778] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14842.058702] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [14842.064778] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12883s workers=4 idle: 20831 12642 28859 [14842.075832] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14832s workers=2 idle: 659 [14848.065502] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [14848.067871] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14848.126601] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14848.135665] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14848.139993] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14848.152045] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14848.172008] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14848.190393] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14848.210756] RAX: 000000000000748a RBX: ffff96848811cae0 RCX: 0000000000000001 [14848.213039] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [14848.221109] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [14848.224823] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [14848.232010] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [14848.242038] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [14848.247699] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14848.257788] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [14848.263036] Call Trace: [14848.263826] [14848.267176] ? show_regs.cold.9+0x22/0x2f [14848.270235] ? watchdog_timer_fn.cold.6+0x62/0x10a [14848.273567] ? __hrtimer_run_queues+0x1c2/0x4d0 [14848.278531] ? watchdog+0x40/0x40 [14848.280750] ? hrtimer_interrupt+0x128/0x2c0 [14848.282065] ? smp_apic_timer_interrupt+0xa6/0x270 [14848.284741] ? apic_timer_interrupt+0xf/0x20 [14848.289123] [14848.289874] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14848.295932] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14848.299439] do_raw_spin_lock+0x177/0x190 [14848.301998] _raw_spin_lock+0x21/0x30 [14848.303775] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14848.306533] process_one_work+0x2c8/0x700 [14848.308779] worker_thread+0x296/0x6e0 [14848.315944] ? rescuer_thread+0x570/0x570 [14848.320024] kthread+0x1d1/0x200 [14848.323112] ? set_kthread_struct+0x70/0x70 [14848.329527] ret_from_fork+0x1f/0x30 [14856.058088] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [14856.068922] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14856.139339] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14856.159483] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14856.170519] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14856.179463] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14856.209489] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14856.220229] RAX: 0000000000002d48 RBX: ffff96848811cae0 RCX: 0000000000000001 [14856.232653] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [14856.238356] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [14856.250401] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [14856.258559] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [14856.268328] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [14856.279168] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14856.283248] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [14856.290611] Call Trace: [14856.291845] [14856.296362] ? show_regs.cold.9+0x22/0x2f [14856.301253] ? watchdog_timer_fn.cold.6+0x62/0x10a [14856.308601] ? __hrtimer_run_queues+0x1c2/0x4d0 [14856.315369] ? watchdog+0x40/0x40 [14856.320778] ? hrtimer_interrupt+0x128/0x2c0 [14856.328517] ? smp_apic_timer_interrupt+0xa6/0x270 [14856.335857] ? apic_timer_interrupt+0xf/0x20 [14856.342338] [14856.344217] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14856.352344] do_raw_spin_lock+0x177/0x190 [14856.355744] _raw_spin_lock+0x21/0x30 [14856.361447] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14856.368552] ? do_raw_spin_unlock+0x75/0x190 [14856.373434] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14856.382289] ? do_wait_intr+0xf0/0xf0 [14856.392676] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14856.404636] mdc_close+0x53b/0xd70 [mdc] [14856.411585] lmv_close+0x18d/0x490 [lmv] [14856.418530] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [14856.425774] ll_close_inode_openhandle+0x493/0xea0 [lustre] [14856.436460] ll_md_real_close+0xa5/0x2a0 [lustre] [14856.445547] ll_md_close+0x274/0x800 [lustre] [14856.451798] ll_file_release+0x128/0xc50 [lustre] [14856.463386] __fput+0x10d/0x440 [14856.467934] ____fput+0x12/0x20 [14856.473766] task_work_run+0xc6/0x110 [14856.477293] exit_to_usermode_loop+0x1dd/0x1f0 [14856.484913] do_syscall_64+0x3d6/0x440 [14856.488395] entry_SYSCALL_64_after_hwframe+0x49/0xae [14856.494383] RIP: 0033:0x7ff54af45cd8 [14856.496765] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [14856.522637] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [14856.533574] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [14856.547696] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [14856.552429] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [14856.563362] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [14856.576388] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14864.060091] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [14864.066615] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14864.111511] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14864.117128] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14864.135696] RIP: 0010:native_safe_halt+0x2e/0x50 [14864.139658] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14864.157716] RSP: 0000:ffffaf9fc15234f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14864.165262] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [14864.169956] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [14864.174674] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [14864.178732] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [14864.191849] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [14864.195255] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [14864.201879] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14864.205265] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [14864.207779] Call Trace: [14864.209975] [14864.211184] ? show_regs.cold.9+0x22/0x2f [14864.213185] ? watchdog_timer_fn.cold.6+0x62/0x10a [14864.216515] ? __hrtimer_run_queues+0x1c2/0x4d0 [14864.220319] ? watchdog+0x40/0x40 [14864.222583] ? hrtimer_interrupt+0x128/0x2c0 [14864.226205] ? smp_apic_timer_interrupt+0xa6/0x270 [14864.227852] ? apic_timer_interrupt+0xf/0x20 [14864.234601] [14864.235089] ? native_safe_halt+0x2e/0x50 [14864.235981] kvm_wait+0xcd/0xe0 [14864.241567] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14864.243927] do_raw_spin_lock+0x177/0x190 [14864.246464] _raw_spin_lock+0x21/0x30 [14864.249446] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14864.251910] ? do_raw_spin_unlock+0x75/0x190 [14864.254296] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14864.256642] ? do_wait_intr+0xf0/0xf0 [14864.263979] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14864.269639] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [14864.274611] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14864.284314] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14864.286214] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14864.290038] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [14864.295006] mdc_intent_lock+0x353/0x7e0 [mdc] [14864.296651] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14864.303712] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14864.307912] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14864.310596] lmv_intent_open+0x338/0x1150 [lmv] [14864.317349] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14864.321464] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14864.324553] lmv_intent_lock+0x1dd/0x4c0 [lmv] [14864.326345] ll_intent_lock+0xc5/0xaa0 [lustre] [14864.328805] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14864.332035] ll_lookup_it+0xe0e/0x2270 [lustre] [14864.337933] ? do_raw_spin_unlock+0x75/0x190 [14864.340150] ? _raw_spin_unlock+0x12/0x30 [14864.341745] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [14864.343405] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [14864.345372] ll_atomic_open+0x36a/0xd70 [lustre] [14864.349826] lookup_open+0x356/0xb40 [14864.351700] do_last+0x313/0xfc0 [14864.354090] ? nd_jump_root+0xe5/0x160 [14864.356692] ? path_init+0x437/0x520 [14864.358691] path_openat+0xf7/0x500 [14864.359912] ? do_raw_spin_unlock+0x75/0x190 [14864.361443] do_filp_open+0x99/0x140 [14864.365032] ? getname_flags+0x6e/0x330 [14864.367345] ? __check_object_size+0xff/0x256 [14864.370021] ? do_raw_spin_unlock+0x75/0x190 [14864.373010] ? _raw_spin_unlock+0x12/0x30 [14864.375311] do_sys_openat2+0x2b4/0x410 [14864.377407] do_sys_open+0x73/0xa0 [14864.379563] __x64_sys_openat+0x24/0x30 [14864.382381] do_syscall_64+0xc1/0x440 [14864.385003] entry_SYSCALL_64_after_hwframe+0x49/0xae [14864.386818] RIP: 0033:0x7ff54af45332 [14864.388113] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [14864.399666] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [14864.404758] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [14864.407777] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [14864.414012] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [14864.425359] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [14864.428441] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [14868.055213] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [14868.060641] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14868.091855] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14868.098485] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14868.102347] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [14868.105652] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [14868.131772] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14868.142825] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [14868.158501] RDX: 000000000000207b RSI: 0000000000000003 RDI: ffff96848811cae0 [14868.163504] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [14868.167823] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [14868.170474] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14868.174164] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [14868.179225] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14868.183310] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [14868.186335] Call Trace: [14868.187454] [14868.188427] ? show_regs.cold.9+0x22/0x2f [14868.190000] ? watchdog_timer_fn.cold.6+0x62/0x10a [14868.192013] ? __hrtimer_run_queues+0x1c2/0x4d0 [14868.193856] ? watchdog+0x40/0x40 [14868.195341] ? hrtimer_interrupt+0x128/0x2c0 [14868.196922] ? smp_apic_timer_interrupt+0xa6/0x270 [14868.199017] ? apic_timer_interrupt+0xf/0x20 [14868.200901] [14868.201857] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [14868.204082] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14868.206360] do_raw_spin_lock+0x177/0x190 [14868.207870] _raw_spin_lock+0x21/0x30 [14868.213481] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14868.219950] obd_mod_free+0x46/0x140 [ptlrpc] [14868.224006] mdc_commit_open+0x91/0xb0 [mdc] [14868.227483] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14868.231234] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14868.237412] after_reply+0xe2c/0x1780 [ptlrpc] [14868.240974] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14868.244299] ptlrpcd+0xa82/0xe40 [ptlrpc] [14868.248789] ? do_wait_intr+0xf0/0xf0 [14868.249977] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14868.255931] kthread+0x1d1/0x200 [14868.259230] ? set_kthread_struct+0x70/0x70 [14868.262133] ret_from_fork+0x1f/0x30 [14872.547698] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12913s! [14872.554610] Showing busy workqueues and worker pools: [14872.562113] workqueue events: flags=0x0 [14872.571434] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14872.576952] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [14872.583518] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [14872.588464] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [14872.596788] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14872.600676] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [14872.606416] workqueue events_long: flags=0x0 [14872.612494] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14872.620166] pending: lu_objects_flush [obdclass] [14872.628823] workqueue events_unbound: flags=0x2 [14872.631550] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [14872.635473] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [14872.646188] workqueue events_power_efficient: flags=0x80 [14872.650479] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14872.657617] pending: neigh_periodic_work [14872.666789] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14872.672522] pending: do_cache_clean, check_lifetime [14872.691199] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14872.699465] pending: neigh_periodic_work [14872.701097] workqueue writeback: flags=0x4a [14872.704856] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [14872.710424] pending: wb_workfn, wb_workfn [14872.712041] workqueue xprtiod: flags=0xa [14872.714730] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [14872.716936] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [14872.733584] workqueue ipv6_addrconf: flags=0x40008 [14872.740459] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14872.744173] pending: addrconf_verify_work [14872.748341] workqueue knbd0-recv: flags=0x1a [14872.755854] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14872.760644] in-flight: 657:recv_work [14872.763703] workqueue ptlrpc_pinger: flags=0x2 [14872.765886] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14872.769702] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [14872.774787] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12913s workers=4 idle: 20831 12642 28859 [14872.782434] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14863s workers=2 idle: 659 [14876.061697] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [14876.074972] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14876.125038] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14876.154437] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14876.166096] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14876.174331] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14876.182350] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14876.206702] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14876.217146] RAX: 0000000000007d84 RBX: ffff96848811cae0 RCX: 0000000000000001 [14876.227490] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [14876.234287] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [14876.245146] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [14876.255330] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [14876.262488] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [14876.275419] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14876.286455] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [14876.292063] Call Trace: [14876.292947] [14876.306441] ? show_regs.cold.9+0x22/0x2f [14876.309802] ? watchdog_timer_fn.cold.6+0x62/0x10a [14876.319264] ? __hrtimer_run_queues+0x1c2/0x4d0 [14876.326355] ? watchdog+0x40/0x40 [14876.332768] ? hrtimer_interrupt+0x128/0x2c0 [14876.345470] ? smp_apic_timer_interrupt+0xa6/0x270 [14876.356830] ? apic_timer_interrupt+0xf/0x20 [14876.365270] [14876.369286] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14876.376660] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14876.385333] do_raw_spin_lock+0x177/0x190 [14876.386912] _raw_spin_lock+0x21/0x30 [14876.393173] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14876.401245] process_one_work+0x2c8/0x700 [14876.404948] worker_thread+0x296/0x6e0 [14876.409934] ? rescuer_thread+0x570/0x570 [14876.414494] kthread+0x1d1/0x200 [14876.415656] ? set_kthread_struct+0x70/0x70 [14876.424632] ret_from_fork+0x1f/0x30 [14884.058026] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [14884.073895] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14884.124238] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14884.143117] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14884.151421] RIP: 0010:native_safe_halt+0x2e/0x50 [14884.161412] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14884.191622] RSP: 0018:ffffaf9fc1543ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14884.202403] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [14884.216401] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [14884.233968] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [14884.245543] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [14884.255407] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [14884.274998] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [14884.286090] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14884.295843] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [14884.298974] Call Trace: [14884.300248] [14884.301185] ? show_regs.cold.9+0x22/0x2f [14884.303271] ? watchdog_timer_fn.cold.6+0x62/0x10a [14884.305487] ? __hrtimer_run_queues+0x1c2/0x4d0 [14884.307652] ? watchdog+0x40/0x40 [14884.309549] ? hrtimer_interrupt+0x128/0x2c0 [14884.312005] ? smp_apic_timer_interrupt+0xa6/0x270 [14884.315620] ? apic_timer_interrupt+0xf/0x20 [14884.318351] [14884.319791] ? native_safe_halt+0x2e/0x50 [14884.321814] kvm_wait+0xcd/0xe0 [14884.324398] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14884.327312] do_raw_spin_lock+0x177/0x190 [14884.329940] _raw_spin_lock+0x21/0x30 [14884.331340] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14884.333372] ? do_raw_spin_unlock+0x75/0x190 [14884.334846] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14884.337345] ? do_wait_intr+0xf0/0xf0 [14884.338643] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14884.340801] mdc_close+0x53b/0xd70 [mdc] [14884.343376] lmv_close+0x18d/0x490 [lmv] [14884.345502] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [14884.349220] ll_close_inode_openhandle+0x493/0xea0 [lustre] [14884.352649] ll_md_real_close+0xa5/0x2a0 [lustre] [14884.357705] ll_md_close+0x274/0x800 [lustre] [14884.363829] ll_file_release+0x128/0xc50 [lustre] [14884.366726] __fput+0x10d/0x440 [14884.367878] ____fput+0x12/0x20 [14884.371691] task_work_run+0xc6/0x110 [14884.375715] exit_to_usermode_loop+0x1dd/0x1f0 [14884.380664] do_syscall_64+0x3d6/0x440 [14884.384332] entry_SYSCALL_64_after_hwframe+0x49/0xae [14884.390244] RIP: 0033:0x7ff54af45cd8 [14884.394525] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [14884.408299] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [14884.414693] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [14884.421428] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [14884.424376] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [14884.432009] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [14884.435680] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14892.061158] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [14892.067660] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14892.107603] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14892.121334] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14892.127113] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14892.134299] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14892.153713] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14892.164791] RAX: 0000000000006e91 RBX: ffff96848811cae0 RCX: 0000000000000001 [14892.168928] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [14892.174734] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [14892.182975] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [14892.190470] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [14892.195564] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [14892.200120] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14892.202998] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [14892.205723] Call Trace: [14892.206523] [14892.212146] ? show_regs.cold.9+0x22/0x2f [14892.214549] ? watchdog_timer_fn.cold.6+0x62/0x10a [14892.217850] ? __hrtimer_run_queues+0x1c2/0x4d0 [14892.222076] ? watchdog+0x40/0x40 [14892.224510] ? hrtimer_interrupt+0x128/0x2c0 [14892.227324] ? smp_apic_timer_interrupt+0xa6/0x270 [14892.229428] ? apic_timer_interrupt+0xf/0x20 [14892.233502] [14892.234347] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14892.236299] do_raw_spin_lock+0x177/0x190 [14892.242500] _raw_spin_lock+0x21/0x30 [14892.245401] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14892.248181] ? do_raw_spin_unlock+0x75/0x190 [14892.255543] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14892.260886] ? do_wait_intr+0xf0/0xf0 [14892.264395] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14892.266434] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [14892.274069] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14892.279152] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14892.283137] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14892.284988] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [14892.288330] mdc_intent_lock+0x353/0x7e0 [mdc] [14892.291236] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14892.295542] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14892.299084] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14892.303510] lmv_intent_open+0x338/0x1150 [lmv] [14892.307158] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14892.312136] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14892.317561] lmv_intent_lock+0x1dd/0x4c0 [lmv] [14892.320263] ll_intent_lock+0xc5/0xaa0 [lustre] [14892.323409] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14892.328783] ll_lookup_it+0xe0e/0x2270 [lustre] [14892.333383] ? do_raw_spin_unlock+0x75/0x190 [14892.337903] ? _raw_spin_unlock+0x12/0x30 [14892.340393] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [14892.343693] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [14892.350219] ll_atomic_open+0x36a/0xd70 [lustre] [14892.353480] lookup_open+0x356/0xb40 [14892.356206] do_last+0x313/0xfc0 [14892.358623] ? nd_jump_root+0xe5/0x160 [14892.360996] ? path_init+0x437/0x520 [14892.362208] path_openat+0xf7/0x500 [14892.366095] ? do_raw_spin_unlock+0x75/0x190 [14892.367672] do_filp_open+0x99/0x140 [14892.370081] ? getname_flags+0x6e/0x330 [14892.372602] ? __check_object_size+0xff/0x256 [14892.376140] ? do_raw_spin_unlock+0x75/0x190 [14892.378464] ? _raw_spin_unlock+0x12/0x30 [14892.380611] do_sys_openat2+0x2b4/0x410 [14892.382476] do_sys_open+0x73/0xa0 [14892.385258] __x64_sys_openat+0x24/0x30 [14892.387980] do_syscall_64+0xc1/0x440 [14892.389957] entry_SYSCALL_64_after_hwframe+0x49/0xae [14892.392948] RIP: 0033:0x7ff54af45332 [14892.395816] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [14892.410223] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [14892.414625] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [14892.421248] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [14892.426325] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [14892.433170] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [14892.438543] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [14900.055018] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [14900.058582] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14900.084735] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14900.090935] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14900.096030] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [14900.099631] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [14900.107408] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14900.111617] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [14900.115175] RDX: 00000000000065c1 RSI: 0000000000000003 RDI: ffff96848811cae0 [14900.118324] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [14900.121906] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [14900.126283] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14900.129984] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [14900.134160] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14900.137210] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [14900.140918] Call Trace: [14900.142089] [14900.144251] ? show_regs.cold.9+0x22/0x2f [14900.147395] ? watchdog_timer_fn.cold.6+0x62/0x10a [14900.152280] ? __hrtimer_run_queues+0x1c2/0x4d0 [14900.154682] ? watchdog+0x40/0x40 [14900.156165] ? hrtimer_interrupt+0x128/0x2c0 [14900.158607] ? smp_apic_timer_interrupt+0xa6/0x270 [14900.160426] ? apic_timer_interrupt+0xf/0x20 [14900.162917] [14900.163966] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [14900.166997] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14900.171205] do_raw_spin_lock+0x177/0x190 [14900.173415] _raw_spin_lock+0x21/0x30 [14900.175725] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14900.180274] obd_mod_free+0x46/0x140 [ptlrpc] [14900.183836] mdc_commit_open+0x91/0xb0 [mdc] [14900.187921] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14900.200581] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14900.206929] after_reply+0xe2c/0x1780 [ptlrpc] [14900.211369] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14900.219454] ptlrpcd+0xa82/0xe40 [ptlrpc] [14900.224467] ? do_wait_intr+0xf0/0xf0 [14900.225728] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14900.233585] kthread+0x1d1/0x200 [14900.236259] ? set_kthread_struct+0x70/0x70 [14900.240487] ret_from_fork+0x1f/0x30 [14903.265106] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12944s! [14903.269625] Showing busy workqueues and worker pools: [14903.271387] workqueue events: flags=0x0 [14903.272870] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14903.274638] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [14903.278497] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [14903.286613] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [14903.295243] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14903.297589] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [14903.302981] workqueue events_long: flags=0x0 [14903.305029] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14903.308444] pending: lu_objects_flush [obdclass] [14903.312397] workqueue events_unbound: flags=0x2 [14903.314949] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [14903.317395] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [14903.321205] workqueue events_power_efficient: flags=0x80 [14903.325109] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14903.332242] pending: neigh_periodic_work [14903.335880] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14903.340904] pending: do_cache_clean, check_lifetime [14903.342611] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14903.346940] pending: neigh_periodic_work [14903.349125] workqueue writeback: flags=0x4a [14903.351190] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [14903.353514] pending: wb_workfn, wb_workfn [14903.356893] workqueue xprtiod: flags=0xa [14903.361366] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [14903.364482] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [14903.372898] workqueue ipv6_addrconf: flags=0x40008 [14903.380201] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14903.382598] pending: addrconf_verify_work [14903.384249] workqueue knbd0-recv: flags=0x1a [14903.387018] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14903.390483] in-flight: 657:recv_work [14903.391593] workqueue ptlrpc_pinger: flags=0x2 [14903.393560] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14903.395513] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [14903.401045] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12944s workers=4 idle: 20831 12642 28859 [14903.409352] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14893s workers=2 idle: 659 [14904.059436] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [14904.074201] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14904.115150] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14904.122456] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14904.128294] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14904.133294] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14904.139310] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14904.157068] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14904.167778] RAX: 0000000000000588 RBX: ffff96848811cae0 RCX: 0000000000000001 [14904.181693] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [14904.184154] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [14904.196881] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [14904.211905] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [14904.219093] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [14904.222442] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14904.227155] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [14904.239977] Call Trace: [14904.240861] [14904.241589] ? show_regs.cold.9+0x22/0x2f [14904.251858] ? watchdog_timer_fn.cold.6+0x62/0x10a [14904.253445] ? __hrtimer_run_queues+0x1c2/0x4d0 [14904.254818] ? watchdog+0x40/0x40 [14904.259511] ? hrtimer_interrupt+0x128/0x2c0 [14904.262139] ? smp_apic_timer_interrupt+0xa6/0x270 [14904.265579] ? apic_timer_interrupt+0xf/0x20 [14904.269224] [14904.271915] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14904.277200] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14904.284720] do_raw_spin_lock+0x177/0x190 [14904.289061] _raw_spin_lock+0x21/0x30 [14904.293269] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14904.297006] process_one_work+0x2c8/0x700 [14904.300959] worker_thread+0x296/0x6e0 [14904.302299] ? rescuer_thread+0x570/0x570 [14904.306642] kthread+0x1d1/0x200 [14904.310012] ? set_kthread_struct+0x70/0x70 [14904.311773] ret_from_fork+0x1f/0x30 [14912.058480] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [14912.067901] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14912.102302] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14912.117832] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14912.127275] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14912.131333] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14912.142288] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14912.147189] RAX: 0000000000005970 RBX: ffff96848811cae0 RCX: 0000000000000001 [14912.153544] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [14912.157576] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [14912.161524] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [14912.166591] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [14912.173183] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [14912.181332] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14912.185558] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [14912.192512] Call Trace: [14912.194397] [14912.195669] ? show_regs.cold.9+0x22/0x2f [14912.201163] ? watchdog_timer_fn.cold.6+0x62/0x10a [14912.205116] ? __hrtimer_run_queues+0x1c2/0x4d0 [14912.209037] ? watchdog+0x40/0x40 [14912.211473] ? hrtimer_interrupt+0x128/0x2c0 [14912.214573] ? smp_apic_timer_interrupt+0xa6/0x270 [14912.218486] ? apic_timer_interrupt+0xf/0x20 [14912.223437] [14912.225195] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14912.228563] do_raw_spin_lock+0x177/0x190 [14912.231410] _raw_spin_lock+0x21/0x30 [14912.233816] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14912.238265] ? do_raw_spin_unlock+0x75/0x190 [14912.241517] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14912.244915] ? do_wait_intr+0xf0/0xf0 [14912.247941] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14912.258207] mdc_close+0x53b/0xd70 [mdc] [14912.265489] lmv_close+0x18d/0x490 [lmv] [14912.273556] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [14912.281214] ll_close_inode_openhandle+0x493/0xea0 [lustre] [14912.287465] ll_md_real_close+0xa5/0x2a0 [lustre] [14912.298942] ll_md_close+0x274/0x800 [lustre] [14912.310782] ll_file_release+0x128/0xc50 [lustre] [14912.312505] __fput+0x10d/0x440 [14912.325870] ____fput+0x12/0x20 [14912.330081] task_work_run+0xc6/0x110 [14912.338024] exit_to_usermode_loop+0x1dd/0x1f0 [14912.341142] do_syscall_64+0x3d6/0x440 [14912.346286] entry_SYSCALL_64_after_hwframe+0x49/0xae [14912.352241] RIP: 0033:0x7ff54af45cd8 [14912.360040] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [14912.376141] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [14912.387167] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [14912.396154] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [14912.402094] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [14912.407450] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [14912.416167] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14920.060024] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [14920.063666] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14920.089780] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14920.096058] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14920.098578] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14920.100684] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14920.111682] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14920.115429] RAX: 0000000000003f85 RBX: ffff96848811cae0 RCX: 0000000000000001 [14920.119838] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [14920.124200] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [14920.127132] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [14920.128963] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [14920.130301] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [14920.132450] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14920.133622] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [14920.137053] Call Trace: [14920.138359] [14920.138985] ? show_regs.cold.9+0x22/0x2f [14920.139854] ? watchdog_timer_fn.cold.6+0x62/0x10a [14920.141233] ? __hrtimer_run_queues+0x1c2/0x4d0 [14920.142442] ? watchdog+0x40/0x40 [14920.144109] ? hrtimer_interrupt+0x128/0x2c0 [14920.145844] ? smp_apic_timer_interrupt+0xa6/0x270 [14920.147427] ? apic_timer_interrupt+0xf/0x20 [14920.149788] [14920.150890] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14920.153079] do_raw_spin_lock+0x177/0x190 [14920.155214] _raw_spin_lock+0x21/0x30 [14920.159886] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14920.163473] ? do_raw_spin_unlock+0x75/0x190 [14920.165590] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14920.170060] ? do_wait_intr+0xf0/0xf0 [14920.172885] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14920.182649] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [14920.184391] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14920.194074] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14920.196107] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14920.200032] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [14920.209080] mdc_intent_lock+0x353/0x7e0 [mdc] [14920.214833] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14920.226162] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14920.232461] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14920.236768] lmv_intent_open+0x338/0x1150 [lmv] [14920.238729] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14920.240672] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14920.244051] lmv_intent_lock+0x1dd/0x4c0 [lmv] [14920.247021] ll_intent_lock+0xc5/0xaa0 [lustre] [14920.256723] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14920.258692] ll_lookup_it+0xe0e/0x2270 [lustre] [14920.267801] ? do_raw_spin_unlock+0x75/0x190 [14920.269366] ? _raw_spin_unlock+0x12/0x30 [14920.274334] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [14920.278022] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [14920.284126] ll_atomic_open+0x36a/0xd70 [lustre] [14920.287154] lookup_open+0x356/0xb40 [14920.289256] do_last+0x313/0xfc0 [14920.294869] ? nd_jump_root+0xe5/0x160 [14920.298938] ? path_init+0x437/0x520 [14920.303052] path_openat+0xf7/0x500 [14920.304986] ? do_raw_spin_unlock+0x75/0x190 [14920.309078] do_filp_open+0x99/0x140 [14920.313147] ? getname_flags+0x6e/0x330 [14920.317378] ? __check_object_size+0xff/0x256 [14920.321967] ? do_raw_spin_unlock+0x75/0x190 [14920.324362] ? _raw_spin_unlock+0x12/0x30 [14920.327263] do_sys_openat2+0x2b4/0x410 [14920.330608] do_sys_open+0x73/0xa0 [14920.331920] __x64_sys_openat+0x24/0x30 [14920.332979] do_syscall_64+0xc1/0x440 [14920.335098] entry_SYSCALL_64_after_hwframe+0x49/0xae [14920.338702] RIP: 0033:0x7ff54af45332 [14920.340465] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [14920.349480] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [14920.353471] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [14920.356276] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [14920.357892] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [14920.360486] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [14920.364303] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [14928.055074] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [14928.058700] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14928.085384] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14928.092130] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14928.097154] RIP: 0010:native_safe_halt+0x2e/0x50 [14928.099902] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14928.110506] RSP: 0000:ffffaf9fc1b97bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14928.116831] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [14928.120353] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff96848811cae0 [14928.130057] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [14928.139109] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [14928.145628] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14928.149908] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [14928.154821] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14928.158694] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [14928.163427] Call Trace: [14928.166356] [14928.167907] ? show_regs.cold.9+0x22/0x2f [14928.170526] ? watchdog_timer_fn.cold.6+0x62/0x10a [14928.173692] ? __hrtimer_run_queues+0x1c2/0x4d0 [14928.175957] ? watchdog+0x40/0x40 [14928.177446] ? hrtimer_interrupt+0x128/0x2c0 [14928.180942] ? smp_apic_timer_interrupt+0xa6/0x270 [14928.183662] ? apic_timer_interrupt+0xf/0x20 [14928.186169] [14928.187411] ? native_safe_halt+0x2e/0x50 [14928.194753] kvm_wait+0xcd/0xe0 [14928.196944] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14928.204163] do_raw_spin_lock+0x177/0x190 [14928.211192] _raw_spin_lock+0x21/0x30 [14928.216394] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14928.220656] obd_mod_free+0x46/0x140 [ptlrpc] [14928.228077] mdc_commit_open+0x91/0xb0 [mdc] [14928.235529] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14928.239116] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14928.243611] after_reply+0xe2c/0x1780 [ptlrpc] [14928.246705] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14928.250497] ptlrpcd+0xa82/0xe40 [ptlrpc] [14928.253526] ? do_wait_intr+0xf0/0xf0 [14928.255834] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14928.258972] kthread+0x1d1/0x200 [14928.260899] ? set_kthread_struct+0x70/0x70 [14928.265034] ret_from_fork+0x1f/0x30 [14932.059996] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [14932.064399] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14932.086748] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14932.097725] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14932.104998] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14932.107797] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14932.110394] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14932.120158] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14932.131078] RAX: 0000000000007f85 RBX: ffff96848811cae0 RCX: 0000000000000001 [14932.136167] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [14932.138334] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [14932.141988] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [14932.150034] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [14932.153727] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [14932.161081] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14932.169777] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [14932.176141] Call Trace: [14932.178085] [14932.182260] ? show_regs.cold.9+0x22/0x2f [14932.192155] ? watchdog_timer_fn.cold.6+0x62/0x10a [14932.193274] ? __hrtimer_run_queues+0x1c2/0x4d0 [14932.194535] ? watchdog+0x40/0x40 [14932.196080] ? hrtimer_interrupt+0x128/0x2c0 [14932.197155] ? smp_apic_timer_interrupt+0xa6/0x270 [14932.199622] ? apic_timer_interrupt+0xf/0x20 [14932.203998] [14932.205932] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14932.211660] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14932.213600] do_raw_spin_lock+0x177/0x190 [14932.215107] _raw_spin_lock+0x21/0x30 [14932.216808] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14932.219175] process_one_work+0x2c8/0x700 [14932.221046] worker_thread+0x296/0x6e0 [14932.226720] ? rescuer_thread+0x570/0x570 [14932.230076] kthread+0x1d1/0x200 [14932.232306] ? set_kthread_struct+0x70/0x70 [14932.238112] ret_from_fork+0x1f/0x30 [14933.471265] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12974s! [14933.476238] Showing busy workqueues and worker pools: [14933.478870] workqueue events: flags=0x0 [14933.482760] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14933.487021] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [14933.494895] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [14933.498903] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [14933.505230] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14933.509291] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [14933.514050] workqueue events_long: flags=0x0 [14933.516098] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14933.519622] pending: lu_objects_flush [obdclass] [14933.524043] workqueue events_unbound: flags=0x2 [14933.526834] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [14933.530967] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [14933.535900] workqueue events_power_efficient: flags=0x80 [14933.538661] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14933.542055] pending: neigh_periodic_work [14933.545577] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14933.549297] pending: do_cache_clean, check_lifetime [14933.552113] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14933.556612] pending: neigh_periodic_work [14933.558811] workqueue writeback: flags=0x4a [14933.560961] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [14933.564121] pending: wb_workfn, wb_workfn [14933.566558] workqueue xprtiod: flags=0xa [14933.569085] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [14933.572427] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [14933.576793] workqueue ipv6_addrconf: flags=0x40008 [14933.579124] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14933.582806] pending: addrconf_verify_work [14933.585213] workqueue knbd0-recv: flags=0x1a [14933.587373] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14933.591382] in-flight: 657:recv_work [14933.593645] workqueue ptlrpc_pinger: flags=0x2 [14933.595859] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14933.598609] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [14933.601894] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12974s workers=4 idle: 20831 12642 28859 [14933.606261] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14923s workers=2 idle: 659 [14940.058813] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [14940.071088] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14940.122059] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14940.127760] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14940.132180] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14940.134852] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14940.143660] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14940.147230] RAX: 00000000000055e2 RBX: ffff96848811cae0 RCX: 0000000000000001 [14940.150740] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [14940.153998] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [14940.157703] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [14940.160940] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [14940.164355] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [14940.167665] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14940.170235] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [14940.173034] Call Trace: [14940.174073] [14940.175110] ? show_regs.cold.9+0x22/0x2f [14940.176818] ? watchdog_timer_fn.cold.6+0x62/0x10a [14940.179081] ? __hrtimer_run_queues+0x1c2/0x4d0 [14940.181248] ? watchdog+0x40/0x40 [14940.182993] ? hrtimer_interrupt+0x128/0x2c0 [14940.184712] ? smp_apic_timer_interrupt+0xa6/0x270 [14940.186729] ? apic_timer_interrupt+0xf/0x20 [14940.188684] [14940.189602] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14940.192057] do_raw_spin_lock+0x177/0x190 [14940.194160] _raw_spin_lock+0x21/0x30 [14940.195869] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14940.198203] ? do_raw_spin_unlock+0x75/0x190 [14940.200063] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14940.202754] ? do_wait_intr+0xf0/0xf0 [14940.204185] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14940.206595] mdc_close+0x53b/0xd70 [mdc] [14940.208679] lmv_close+0x18d/0x490 [lmv] [14940.210666] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [14940.214598] ll_close_inode_openhandle+0x493/0xea0 [lustre] [14940.220682] ll_md_real_close+0xa5/0x2a0 [lustre] [14940.224114] ll_md_close+0x274/0x800 [lustre] [14940.226044] ll_file_release+0x128/0xc50 [lustre] [14940.228300] __fput+0x10d/0x440 [14940.230137] ____fput+0x12/0x20 [14940.231752] task_work_run+0xc6/0x110 [14940.233451] exit_to_usermode_loop+0x1dd/0x1f0 [14940.235542] do_syscall_64+0x3d6/0x440 [14940.237441] entry_SYSCALL_64_after_hwframe+0x49/0xae [14940.244679] RIP: 0033:0x7ff54af45cd8 [14940.247940] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [14940.263314] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [14940.268206] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [14940.273806] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [14940.277626] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [14940.281223] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [14940.285967] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14948.060032] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [14948.065075] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14948.080810] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14948.084844] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14948.087435] RIP: 0010:native_safe_halt+0x2e/0x50 [14948.089114] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14948.095596] RSP: 0000:ffffaf9fc15234f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14948.097617] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [14948.102404] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c21b4154 [14948.106751] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [14948.115852] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [14948.119823] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [14948.126243] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [14948.131559] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14948.138145] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [14948.145093] Call Trace: [14948.146200] [14948.147151] ? show_regs.cold.9+0x22/0x2f [14948.150000] ? watchdog_timer_fn.cold.6+0x62/0x10a [14948.151522] ? __hrtimer_run_queues+0x1c2/0x4d0 [14948.153678] ? watchdog+0x40/0x40 [14948.155767] ? hrtimer_interrupt+0x128/0x2c0 [14948.158231] ? smp_apic_timer_interrupt+0xa6/0x270 [14948.160305] ? apic_timer_interrupt+0xf/0x20 [14948.161943] [14948.162792] ? native_safe_halt+0x2e/0x50 [14948.164358] kvm_wait+0xcd/0xe0 [14948.166336] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14948.168681] do_raw_spin_lock+0x177/0x190 [14948.170460] _raw_spin_lock+0x21/0x30 [14948.171801] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14948.174765] ? do_raw_spin_unlock+0x75/0x190 [14948.177217] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14948.179380] ? do_wait_intr+0xf0/0xf0 [14948.180418] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14948.182303] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [14948.184486] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14948.186584] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14948.188810] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14948.191356] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [14948.193431] mdc_intent_lock+0x353/0x7e0 [mdc] [14948.195069] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14948.197230] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14948.199867] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14948.202797] lmv_intent_open+0x338/0x1150 [lmv] [14948.205518] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14948.208444] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14948.210599] lmv_intent_lock+0x1dd/0x4c0 [lmv] [14948.214709] ll_intent_lock+0xc5/0xaa0 [lustre] [14948.217573] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14948.220804] ll_lookup_it+0xe0e/0x2270 [lustre] [14948.223485] ? do_raw_spin_unlock+0x75/0x190 [14948.225642] ? _raw_spin_unlock+0x12/0x30 [14948.227666] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [14948.231808] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [14948.235747] ll_atomic_open+0x36a/0xd70 [lustre] [14948.237759] lookup_open+0x356/0xb40 [14948.240381] do_last+0x313/0xfc0 [14948.242514] ? nd_jump_root+0xe5/0x160 [14948.244168] ? path_init+0x437/0x520 [14948.245777] path_openat+0xf7/0x500 [14948.248976] ? do_raw_spin_unlock+0x75/0x190 [14948.251052] do_filp_open+0x99/0x140 [14948.254443] ? getname_flags+0x6e/0x330 [14948.260015] ? __check_object_size+0xff/0x256 [14948.262406] ? do_raw_spin_unlock+0x75/0x190 [14948.266553] ? _raw_spin_unlock+0x12/0x30 [14948.268778] do_sys_openat2+0x2b4/0x410 [14948.273181] do_sys_open+0x73/0xa0 [14948.275702] __x64_sys_openat+0x24/0x30 [14948.279157] do_syscall_64+0xc1/0x440 [14948.282665] entry_SYSCALL_64_after_hwframe+0x49/0xae [14948.285923] RIP: 0033:0x7ff54af45332 [14948.288798] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [14948.298459] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [14948.305620] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [14948.310434] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [14948.316023] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [14948.320802] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [14948.328151] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [14960.059501] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [14960.062169] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14960.067378] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [14960.137828] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14960.137843] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14960.137850] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14960.148575] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14960.148670] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14960.148678] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14960.148681] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [14960.148696] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [14960.148700] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14960.148705] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [14960.148708] RDX: 0000000000001ead RSI: 0000000000000003 RDI: ffff96848811cae0 [14960.148711] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [14960.148713] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [14960.148716] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14960.148719] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [14960.148779] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14960.148782] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [14960.148790] Call Trace: [14960.148798] [14960.148804] ? show_regs.cold.9+0x22/0x2f [14960.148817] ? watchdog_timer_fn.cold.6+0x62/0x10a [14960.148824] ? __hrtimer_run_queues+0x1c2/0x4d0 [14960.148828] ? watchdog+0x40/0x40 [14960.148833] ? hrtimer_interrupt+0x128/0x2c0 [14960.149534] ? smp_apic_timer_interrupt+0xa6/0x270 [14960.149547] ? apic_timer_interrupt+0xf/0x20 [14960.149552] [14960.149556] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [14960.149562] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14960.149567] do_raw_spin_lock+0x177/0x190 [14960.149572] _raw_spin_lock+0x21/0x30 [14960.149580] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14960.151720] obd_mod_free+0x46/0x140 [ptlrpc] [14960.170123] [14960.170146] RIP: 0010:native_safe_halt+0x2e/0x50 [14960.170713] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14960.170718] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [14960.170724] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [14960.170726] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c2134154 [14960.170728] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [14960.170730] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [14960.170732] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [14960.170736] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [14960.170738] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14960.170741] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [14960.170749] Call Trace: [14960.170755] [14960.170761] ? show_regs.cold.9+0x22/0x2f [14960.170773] ? watchdog_timer_fn.cold.6+0x62/0x10a [14960.170781] ? __hrtimer_run_queues+0x1c2/0x4d0 [14960.170785] ? watchdog+0x40/0x40 [14960.170789] ? hrtimer_interrupt+0x128/0x2c0 [14960.170794] ? smp_apic_timer_interrupt+0xa6/0x270 [14960.170802] ? apic_timer_interrupt+0xf/0x20 [14960.170808] [14960.170811] ? native_safe_halt+0x2e/0x50 [14960.170815] kvm_wait+0xcd/0xe0 [14960.170824] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14960.170835] do_raw_spin_lock+0x177/0x190 [14960.170841] _raw_spin_lock+0x21/0x30 [14960.170846] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14960.171436] process_one_work+0x2c8/0x700 [14960.171444] worker_thread+0x296/0x6e0 [14960.171448] ? rescuer_thread+0x570/0x570 [14960.171452] kthread+0x1d1/0x200 [14960.171457] ? set_kthread_struct+0x70/0x70 [14960.171461] ret_from_fork+0x1f/0x30 [14960.555121] mdc_commit_open+0x91/0xb0 [mdc] [14960.559530] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14960.562724] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14960.568205] after_reply+0xe2c/0x1780 [ptlrpc] [14960.575089] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14960.583667] ptlrpcd+0xa82/0xe40 [ptlrpc] [14960.590246] ? do_wait_intr+0xf0/0xf0 [14960.592474] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14960.596202] kthread+0x1d1/0x200 [14960.599961] ? set_kthread_struct+0x70/0x70 [14960.603607] ret_from_fork+0x1f/0x30 [14963.679054] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 13004s! [14963.683868] Showing busy workqueues and worker pools: [14963.686797] workqueue events: flags=0x0 [14963.688865] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14963.693710] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [14963.700910] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [14963.706331] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [14963.713643] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14963.718840] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [14963.723653] workqueue events_long: flags=0x0 [14963.726346] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14963.729771] pending: lu_objects_flush [obdclass] [14963.733123] workqueue events_unbound: flags=0x2 [14963.735768] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [14963.739667] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [14963.746073] workqueue events_power_efficient: flags=0x80 [14963.748878] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14963.752380] pending: neigh_periodic_work [14963.754701] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14963.757999] pending: do_cache_clean, check_lifetime [14963.760415] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14963.764076] pending: neigh_periodic_work [14963.766341] workqueue writeback: flags=0x4a [14963.768861] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [14963.771754] pending: wb_workfn, wb_workfn [14963.773823] workqueue xprtiod: flags=0xa [14963.775640] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [14963.777965] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [14963.781687] workqueue ipv6_addrconf: flags=0x40008 [14963.785506] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14963.790474] pending: addrconf_verify_work [14963.792529] workqueue knbd0-recv: flags=0x1a [14963.795758] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14963.800040] in-flight: 657:recv_work [14963.801862] workqueue ptlrpc_pinger: flags=0x2 [14963.804358] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14963.808329] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [14963.811054] pool 8: cpus=0-3 flags=0x4 nice=0 hung=13004s workers=4 idle: 20831 12642 28859 [14963.816706] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14954s workers=2 idle: 659 [14968.061306] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [14968.071663] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14968.131688] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14968.138315] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14968.145474] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14968.149438] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14968.166440] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [14968.171483] RAX: 0000000000004522 RBX: ffff96848811cae0 RCX: 0000000000000001 [14968.176126] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [14968.179344] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [14968.182872] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [14968.188491] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [14968.192267] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [14968.195849] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14968.200693] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [14968.206644] Call Trace: [14968.207797] [14968.208880] ? show_regs.cold.9+0x22/0x2f [14968.211026] ? watchdog_timer_fn.cold.6+0x62/0x10a [14968.216024] ? __hrtimer_run_queues+0x1c2/0x4d0 [14968.224967] ? watchdog+0x40/0x40 [14968.226450] ? hrtimer_interrupt+0x128/0x2c0 [14968.229724] ? smp_apic_timer_interrupt+0xa6/0x270 [14968.231868] ? apic_timer_interrupt+0xf/0x20 [14968.234141] [14968.234968] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14968.240106] do_raw_spin_lock+0x177/0x190 [14968.242189] _raw_spin_lock+0x21/0x30 [14968.243936] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14968.247150] ? do_raw_spin_unlock+0x75/0x190 [14968.248751] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14968.252339] ? do_wait_intr+0xf0/0xf0 [14968.253968] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14968.257599] mdc_close+0x53b/0xd70 [mdc] [14968.259410] lmv_close+0x18d/0x490 [lmv] [14968.262281] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [14968.265494] ll_close_inode_openhandle+0x493/0xea0 [lustre] [14968.269940] ll_md_real_close+0xa5/0x2a0 [lustre] [14968.276478] ll_md_close+0x274/0x800 [lustre] [14968.281470] ll_file_release+0x128/0xc50 [lustre] [14968.283542] __fput+0x10d/0x440 [14968.286453] ____fput+0x12/0x20 [14968.287943] task_work_run+0xc6/0x110 [14968.290391] exit_to_usermode_loop+0x1dd/0x1f0 [14968.292767] do_syscall_64+0x3d6/0x440 [14968.294946] entry_SYSCALL_64_after_hwframe+0x49/0xae [14968.296836] RIP: 0033:0x7ff54af45cd8 [14968.300002] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [14968.322967] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [14968.331565] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [14968.337669] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [14968.345480] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [14968.353362] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [14968.357852] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14974.189414] rcu: INFO: rcu_sched self-detected stall on CPU [14974.194883] rcu: 0-...!: (6919792 ticks this GP) idle=98a/1/0x4000000000000002 softirq=125154/125154 fqs=959 [14974.211926] (t=13020349 jiffies g=253585 q=7056) [14974.219445] rcu: rcu_sched kthread starved for 13015309 jiffies! g253585 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [14974.226940] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [14974.237995] rcu: RCU grace-period kthread stack dump: [14974.239155] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [14974.248783] Call Trace: [14974.249724] __schedule+0x351/0xcb0 [14974.251044] schedule+0xc0/0x180 [14974.252389] schedule_timeout+0xb4/0x190 [14974.253961] ? do_raw_spin_unlock+0x75/0x190 [14974.255912] ? __next_timer_interrupt+0x160/0x160 [14974.257869] ? prepare_to_swait_event+0xf3/0x140 [14974.259984] rcu_gp_kthread+0x8dd/0x15b0 [14974.261665] ? force_qs_rnp+0x2c0/0x2c0 [14974.263216] kthread+0x1d1/0x200 [14974.264421] ? set_kthread_struct+0x70/0x70 [14974.265983] ret_from_fork+0x1f/0x30 [14974.267488] rcu: Stack dump where RCU GP kthread last ran: [14974.269647] NMI backtrace for cpu 0 [14974.270913] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14974.275319] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14974.280267] Call Trace: [14974.281254] [14974.281978] ? dump_stack+0xbb/0x10e [14974.285247] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [14974.287419] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [14974.289568] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [14974.291844] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [14974.294317] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [14974.296429] ? rcu_check_gp_kthread_starvation+0x16f/0x188 [14974.298490] ? rcu_sched_clock_irq.cold.51+0xce/0x307 [14974.300719] ? update_process_times+0x6d/0xc0 [14974.302587] ? tick_sched_handle.isra.18+0x27/0x90 [14974.304670] ? tick_sched_timer+0x53/0xb0 [14974.306405] ? __hrtimer_run_queues+0x1c2/0x4d0 [14974.308387] ? tick_sched_handle.isra.18+0x90/0x90 [14974.310376] ? hrtimer_interrupt+0x128/0x2c0 [14974.312309] ? smp_apic_timer_interrupt+0xa6/0x270 [14974.314319] ? apic_timer_interrupt+0xf/0x20 [14974.315617] [14974.316263] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [14974.318795] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14974.320587] ? do_raw_spin_lock+0x177/0x190 [14974.321810] ? _raw_spin_lock+0x21/0x30 [14974.323431] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14974.325614] ? obd_mod_free+0x46/0x140 [ptlrpc] [14974.327698] ? mdc_commit_open+0x91/0xb0 [mdc] [14974.329686] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14974.331731] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14974.334359] ? after_reply+0xe2c/0x1780 [ptlrpc] [14974.336676] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14974.339623] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [14974.342067] ? do_wait_intr+0xf0/0xf0 [14974.343785] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14974.346641] ? kthread+0x1d1/0x200 [14974.348294] ? set_kthread_struct+0x70/0x70 [14974.350274] ? ret_from_fork+0x1f/0x30 [14974.351915] NMI backtrace for cpu 0 [14974.353451] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14974.362729] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14974.366736] Call Trace: [14974.367617] [14974.368527] ? dump_stack+0xbb/0x10e [14974.370615] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [14974.373441] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [14974.376454] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [14974.382984] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [14974.387850] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [14974.395432] ? rcu_dump_cpu_stacks+0x11b/0x171 [14974.398558] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [14974.404852] ? update_process_times+0x6d/0xc0 [14974.410911] ? tick_sched_handle.isra.18+0x27/0x90 [14974.415959] ? tick_sched_timer+0x53/0xb0 [14974.419859] ? __hrtimer_run_queues+0x1c2/0x4d0 [14974.424287] ? tick_sched_handle.isra.18+0x90/0x90 [14974.428817] ? hrtimer_interrupt+0x128/0x2c0 [14974.432504] ? smp_apic_timer_interrupt+0xa6/0x270 [14974.437854] ? apic_timer_interrupt+0xf/0x20 [14974.441277] [14974.442059] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [14974.445852] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14974.450469] ? do_raw_spin_lock+0x177/0x190 [14974.454725] ? _raw_spin_lock+0x21/0x30 [14974.456544] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14974.459319] ? obd_mod_free+0x46/0x140 [ptlrpc] [14974.465378] ? mdc_commit_open+0x91/0xb0 [mdc] [14974.469451] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14974.475705] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14974.481343] ? after_reply+0xe2c/0x1780 [ptlrpc] [14974.486858] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14974.492422] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [14974.495709] ? do_wait_intr+0xf0/0xf0 [14974.498779] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14974.504205] ? kthread+0x1d1/0x200 [14974.507702] ? set_kthread_struct+0x70/0x70 [14974.510686] ? ret_from_fork+0x1f/0x30 [14976.060221] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [14976.065630] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14976.109383] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14976.120456] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14976.130771] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14976.138586] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14976.163603] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14976.169609] RAX: 0000000000007285 RBX: ffff96848811cae0 RCX: 0000000000000001 [14976.180191] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [14976.190832] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [14976.198694] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [14976.205190] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [14976.208952] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [14976.213596] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14976.217634] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [14976.220703] Call Trace: [14976.224437] [14976.225297] ? show_regs.cold.9+0x22/0x2f [14976.227207] ? watchdog_timer_fn.cold.6+0x62/0x10a [14976.230613] ? __hrtimer_run_queues+0x1c2/0x4d0 [14976.234974] ? watchdog+0x40/0x40 [14976.238635] ? hrtimer_interrupt+0x128/0x2c0 [14976.240525] ? smp_apic_timer_interrupt+0xa6/0x270 [14976.243636] ? apic_timer_interrupt+0xf/0x20 [14976.248515] [14976.249591] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14976.252932] do_raw_spin_lock+0x177/0x190 [14976.255100] _raw_spin_lock+0x21/0x30 [14976.258724] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14976.263835] ? do_raw_spin_unlock+0x75/0x190 [14976.266679] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14976.272817] ? do_wait_intr+0xf0/0xf0 [14976.275805] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14976.281872] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [14976.288470] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14976.297405] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14976.302707] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14976.305789] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [14976.308510] mdc_intent_lock+0x353/0x7e0 [mdc] [14976.314311] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14976.318243] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14976.320189] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14976.323674] lmv_intent_open+0x338/0x1150 [lmv] [14976.325551] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14976.330404] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14976.333269] lmv_intent_lock+0x1dd/0x4c0 [lmv] [14976.337058] ll_intent_lock+0xc5/0xaa0 [lustre] [14976.341454] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14976.344390] ll_lookup_it+0xe0e/0x2270 [lustre] [14976.349674] ? do_raw_spin_unlock+0x75/0x190 [14976.359970] ? _raw_spin_unlock+0x12/0x30 [14976.367819] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [14976.370090] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [14976.391808] ll_atomic_open+0x36a/0xd70 [lustre] [14976.399536] lookup_open+0x356/0xb40 [14976.403616] do_last+0x313/0xfc0 [14976.405659] ? nd_jump_root+0xe5/0x160 [14976.409252] ? path_init+0x437/0x520 [14976.411818] path_openat+0xf7/0x500 [14976.414500] ? do_raw_spin_unlock+0x75/0x190 [14976.418922] do_filp_open+0x99/0x140 [14976.423841] ? getname_flags+0x6e/0x330 [14976.426563] ? __check_object_size+0xff/0x256 [14976.429705] ? do_raw_spin_unlock+0x75/0x190 [14976.432513] ? _raw_spin_unlock+0x12/0x30 [14976.440730] do_sys_openat2+0x2b4/0x410 [14976.445068] do_sys_open+0x73/0xa0 [14976.453646] __x64_sys_openat+0x24/0x30 [14976.455705] do_syscall_64+0xc1/0x440 [14976.466813] entry_SYSCALL_64_after_hwframe+0x49/0xae [14976.470694] RIP: 0033:0x7ff54af45332 [14976.472639] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [14976.499916] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [14976.510493] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [14976.514299] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [14976.519883] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [14976.523761] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [14976.528837] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [14988.065211] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [14988.072122] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14988.132300] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14988.140314] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14988.145733] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14988.150505] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14988.154016] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14988.170493] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14988.179054] RAX: 000000000000748a RBX: ffff96848811cae0 RCX: 0000000000000001 [14988.184095] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [14988.188153] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [14988.191854] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [14988.198601] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [14988.203280] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [14988.209986] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14988.213164] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [14988.216849] Call Trace: [14988.217853] [14988.218549] ? show_regs.cold.9+0x22/0x2f [14988.219952] ? watchdog_timer_fn.cold.6+0x62/0x10a [14988.222639] ? __hrtimer_run_queues+0x1c2/0x4d0 [14988.230327] ? watchdog+0x40/0x40 [14988.234247] ? hrtimer_interrupt+0x128/0x2c0 [14988.239635] ? smp_apic_timer_interrupt+0xa6/0x270 [14988.246851] ? apic_timer_interrupt+0xf/0x20 [14988.255203] [14988.255990] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14988.267282] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14988.271455] do_raw_spin_lock+0x177/0x190 [14988.274309] _raw_spin_lock+0x21/0x30 [14988.283341] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14988.292512] process_one_work+0x2c8/0x700 [14988.294068] worker_thread+0x296/0x6e0 [14988.295486] ? rescuer_thread+0x570/0x570 [14988.301582] kthread+0x1d1/0x200 [14988.303639] ? set_kthread_struct+0x70/0x70 [14988.305754] ret_from_fork+0x1f/0x30 [14993.887048] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 13035s! [14993.894099] Showing busy workqueues and worker pools: [14993.904297] workqueue events: flags=0x0 [14993.910416] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14993.913451] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [14993.919015] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [14993.933596] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [14993.943158] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14993.949171] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [14993.952684] workqueue events_long: flags=0x0 [14993.955466] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14993.963790] pending: lu_objects_flush [obdclass] [14993.974102] workqueue events_unbound: flags=0x2 [14993.975712] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [14993.986711] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [14993.997012] workqueue events_power_efficient: flags=0x80 [14994.002799] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14994.010483] pending: neigh_periodic_work [14994.014633] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14994.020430] pending: do_cache_clean, check_lifetime [14994.023455] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14994.031696] pending: neigh_periodic_work [14994.037302] workqueue writeback: flags=0x4a [14994.042601] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [14994.047333] pending: wb_workfn, wb_workfn [14994.051253] workqueue xprtiod: flags=0xa [14994.057755] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [14994.066592] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [14994.078735] workqueue ipv6_addrconf: flags=0x40008 [14994.085774] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14994.097784] pending: addrconf_verify_work [14994.100260] workqueue knbd0-recv: flags=0x1a [14994.104222] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14994.113433] in-flight: 657:recv_work [14994.115657] workqueue ptlrpc_pinger: flags=0x2 [14994.124461] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14994.128458] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [14994.131341] pool 8: cpus=0-3 flags=0x4 nice=0 hung=13035s workers=4 idle: 20831 12642 28859 [14994.136156] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14984s workers=2 idle: 659 [14996.058062] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [14996.060851] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14996.124681] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14996.134107] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14996.146083] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14996.161179] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14996.185835] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14996.192534] RAX: 0000000000006126 RBX: ffff96848811cae0 RCX: 0000000000000001 [14996.199674] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [14996.202414] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [14996.214593] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [14996.218841] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [14996.226594] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [14996.239948] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14996.245707] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [14996.254588] Call Trace: [14996.257390] [14996.258969] ? show_regs.cold.9+0x22/0x2f [14996.264194] ? watchdog_timer_fn.cold.6+0x62/0x10a [14996.267591] ? __hrtimer_run_queues+0x1c2/0x4d0 [14996.274879] ? watchdog+0x40/0x40 [14996.276696] ? hrtimer_interrupt+0x128/0x2c0 [14996.282042] ? smp_apic_timer_interrupt+0xa6/0x270 [14996.287864] ? apic_timer_interrupt+0xf/0x20 [14996.291068] [14996.295857] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14996.298607] do_raw_spin_lock+0x177/0x190 [14996.303577] _raw_spin_lock+0x21/0x30 [14996.305162] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14996.318497] ? do_raw_spin_unlock+0x75/0x190 [14996.326374] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14996.341350] ? do_wait_intr+0xf0/0xf0 [14996.352048] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14996.361555] mdc_close+0x53b/0xd70 [mdc] [14996.363011] lmv_close+0x18d/0x490 [lmv] [14996.364441] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [14996.392167] ll_close_inode_openhandle+0x493/0xea0 [lustre] [14996.400602] ll_md_real_close+0xa5/0x2a0 [lustre] [14996.403587] ll_md_close+0x274/0x800 [lustre] [14996.407761] ll_file_release+0x128/0xc50 [lustre] [14996.416407] __fput+0x10d/0x440 [14996.422428] ____fput+0x12/0x20 [14996.423592] task_work_run+0xc6/0x110 [14996.431151] exit_to_usermode_loop+0x1dd/0x1f0 [14996.433248] do_syscall_64+0x3d6/0x440 [14996.438317] entry_SYSCALL_64_after_hwframe+0x49/0xae [14996.442087] RIP: 0033:0x7ff54af45cd8 [14996.448148] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [14996.460615] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [14996.466736] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [14996.471229] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [14996.473770] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [14996.480203] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [14996.483328] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [15004.060179] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [15004.072098] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15004.125168] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15004.133468] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15004.140552] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15004.143300] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15004.163584] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15004.173144] RAX: 0000000000004f87 RBX: ffff96848811cae0 RCX: 0000000000000001 [15004.180528] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [15004.189288] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [15004.196520] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [15004.203550] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [15004.212968] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [15004.217343] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15004.222665] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [15004.234391] Call Trace: [15004.235118] [15004.235684] ? show_regs.cold.9+0x22/0x2f [15004.243435] ? watchdog_timer_fn.cold.6+0x62/0x10a [15004.249274] ? __hrtimer_run_queues+0x1c2/0x4d0 [15004.253599] ? watchdog+0x40/0x40 [15004.255239] ? hrtimer_interrupt+0x128/0x2c0 [15004.259799] ? smp_apic_timer_interrupt+0xa6/0x270 [15004.263460] ? apic_timer_interrupt+0xf/0x20 [15004.267800] [15004.269370] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15004.282431] do_raw_spin_lock+0x177/0x190 [15004.290695] _raw_spin_lock+0x21/0x30 [15004.295421] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15004.303526] ? do_raw_spin_unlock+0x75/0x190 [15004.308513] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15004.310236] ? do_wait_intr+0xf0/0xf0 [15004.312531] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15004.315899] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [15004.319972] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15004.323165] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15004.328335] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15004.334840] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [15004.339244] mdc_intent_lock+0x353/0x7e0 [mdc] [15004.346337] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15004.350598] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15004.360304] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15004.366401] lmv_intent_open+0x338/0x1150 [lmv] [15004.369484] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15004.381188] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15004.390783] lmv_intent_lock+0x1dd/0x4c0 [lmv] [15004.396939] ll_intent_lock+0xc5/0xaa0 [lustre] [15004.404417] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15004.410447] ll_lookup_it+0xe0e/0x2270 [lustre] [15004.416865] ? do_raw_spin_unlock+0x75/0x190 [15004.421952] ? _raw_spin_unlock+0x12/0x30 [15004.426982] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [15004.434772] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [15004.443442] ll_atomic_open+0x36a/0xd70 [lustre] [15004.448593] lookup_open+0x356/0xb40 [15004.454411] do_last+0x313/0xfc0 [15004.457194] ? nd_jump_root+0xe5/0x160 [15004.463719] ? path_init+0x437/0x520 [15004.466959] path_openat+0xf7/0x500 [15004.468901] ? do_raw_spin_unlock+0x75/0x190 [15004.476806] do_filp_open+0x99/0x140 [15004.478216] ? getname_flags+0x6e/0x330 [15004.488982] ? __check_object_size+0xff/0x256 [15004.492309] ? do_raw_spin_unlock+0x75/0x190 [15004.502019] ? _raw_spin_unlock+0x12/0x30 [15004.508648] do_sys_openat2+0x2b4/0x410 [15004.517233] do_sys_open+0x73/0xa0 [15004.518559] __x64_sys_openat+0x24/0x30 [15004.534922] do_syscall_64+0xc1/0x440 [15004.536343] entry_SYSCALL_64_after_hwframe+0x49/0xae [15004.541916] RIP: 0033:0x7ff54af45332 [15004.543289] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [15004.562662] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [15004.571094] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [15004.575736] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [15004.580893] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [15004.584768] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [15004.586285] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [15016.059028] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [15016.062431] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15016.083961] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15016.088921] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15016.092706] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [15016.095795] RIP: 0010:native_safe_halt+0x2e/0x50 [15016.097585] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15016.106997] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15016.110465] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [15016.113822] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [15016.117077] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [15016.120203] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [15016.122998] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [15016.126480] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [15016.132162] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15016.134692] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [15016.152064] Call Trace: [15016.154869] [15016.155928] ? show_regs.cold.9+0x22/0x2f [15016.166945] ? watchdog_timer_fn.cold.6+0x62/0x10a [15016.176154] ? __hrtimer_run_queues+0x1c2/0x4d0 [15016.179729] ? watchdog+0x40/0x40 [15016.189292] ? hrtimer_interrupt+0x128/0x2c0 [15016.191358] ? smp_apic_timer_interrupt+0xa6/0x270 [15016.201533] ? apic_timer_interrupt+0xf/0x20 [15016.204256] [15016.209245] ? native_safe_halt+0x2e/0x50 [15016.211268] kvm_wait+0xcd/0xe0 [15016.214044] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15016.219343] do_raw_spin_lock+0x177/0x190 [15016.220683] _raw_spin_lock+0x21/0x30 [15016.231696] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [15016.242683] process_one_work+0x2c8/0x700 [15016.246020] worker_thread+0x296/0x6e0 [15016.251103] ? rescuer_thread+0x570/0x570 [15016.256066] kthread+0x1d1/0x200 [15016.259971] ? set_kthread_struct+0x70/0x70 [15016.264291] ret_from_fork+0x1f/0x30 [15020.055743] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [15020.060781] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15020.107226] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15020.120528] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15020.127948] RIP: 0010:native_safe_halt+0x2e/0x50 [15020.131185] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15020.145736] RSP: 0000:ffffaf9fc1b97bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15020.150368] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [15020.153952] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff96848811cae0 [15020.160546] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [15020.166803] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [15020.175851] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [15020.180482] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [15020.185674] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15020.195988] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [15020.202533] Call Trace: [15020.203662] [15020.205032] ? show_regs.cold.9+0x22/0x2f [15020.207231] ? watchdog_timer_fn.cold.6+0x62/0x10a [15020.218491] ? __hrtimer_run_queues+0x1c2/0x4d0 [15020.221327] ? watchdog+0x40/0x40 [15020.222161] ? hrtimer_interrupt+0x128/0x2c0 [15020.228299] ? smp_apic_timer_interrupt+0xa6/0x270 [15020.234023] ? apic_timer_interrupt+0xf/0x20 [15020.236213] [15020.238509] ? native_safe_halt+0x2e/0x50 [15020.243468] kvm_wait+0xcd/0xe0 [15020.247504] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15020.249759] do_raw_spin_lock+0x177/0x190 [15020.257522] _raw_spin_lock+0x21/0x30 [15020.258888] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15020.262864] obd_mod_free+0x46/0x140 [ptlrpc] [15020.268668] mdc_commit_open+0x91/0xb0 [mdc] [15020.273547] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15020.280458] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15020.283843] after_reply+0xe2c/0x1780 [ptlrpc] [15020.290768] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15020.295442] ptlrpcd+0xa82/0xe40 [ptlrpc] [15020.298460] ? do_wait_intr+0xf0/0xf0 [15020.302657] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15020.305437] kthread+0x1d1/0x200 [15020.306642] ? set_kthread_struct+0x70/0x70 [15020.310444] ret_from_fork+0x1f/0x30 [15024.060591] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [15024.077681] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15024.138364] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15024.147378] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15024.159457] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15024.168713] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15024.197346] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15024.210479] RAX: 000000000000296b RBX: ffff96848811cae0 RCX: 0000000000000001 [15024.218303] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [15024.235267] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [15024.241203] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [15024.255435] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [15024.260553] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [15024.274543] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15024.278520] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [15024.282727] Call Trace: [15024.284238] [15024.285423] ? show_regs.cold.9+0x22/0x2f [15024.290169] ? watchdog_timer_fn.cold.6+0x62/0x10a [15024.296693] ? __hrtimer_run_queues+0x1c2/0x4d0 [15024.300363] ? watchdog+0x40/0x40 [15024.302798] ? hrtimer_interrupt+0x128/0x2c0 [15024.306349] ? smp_apic_timer_interrupt+0xa6/0x270 [15024.317589] ? apic_timer_interrupt+0xf/0x20 [15024.318801] [15024.319380] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15024.325147] do_raw_spin_lock+0x177/0x190 [15024.329454] _raw_spin_lock+0x21/0x30 [15024.330404] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15024.332439] ? do_raw_spin_unlock+0x75/0x190 [15024.338982] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15024.341490] ? do_wait_intr+0xf0/0xf0 [15024.342506] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15024.345477] mdc_close+0x53b/0xd70 [mdc] [15024.346634] lmv_close+0x18d/0x490 [lmv] [15024.348557] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [15024.351020] ll_close_inode_openhandle+0x493/0xea0 [lustre] [15024.353997] ll_md_real_close+0xa5/0x2a0 [lustre] [15024.356057] ll_md_close+0x274/0x800 [lustre] [15024.358067] ll_file_release+0x128/0xc50 [lustre] [15024.360178] __fput+0x10d/0x440 [15024.362171] ____fput+0x12/0x20 [15024.363294] task_work_run+0xc6/0x110 [15024.365228] exit_to_usermode_loop+0x1dd/0x1f0 [15024.366942] do_syscall_64+0x3d6/0x440 [15024.372693] entry_SYSCALL_64_after_hwframe+0x49/0xae [15024.375830] RIP: 0033:0x7ff54af45cd8 [15024.377488] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [15024.386921] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [15024.395375] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [15024.411110] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [15024.416397] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [15024.424009] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [15024.427327] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [15024.615850] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 13065s! [15024.626459] Showing busy workqueues and worker pools: [15024.637423] workqueue events: flags=0x0 [15024.643016] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15024.650286] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [15024.666835] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [15024.675315] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [15024.686988] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15024.700038] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [15024.705131] workqueue events_long: flags=0x0 [15024.707096] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15024.717145] pending: lu_objects_flush [obdclass] [15024.724773] workqueue events_unbound: flags=0x2 [15024.728033] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [15024.734425] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [15024.749193] workqueue events_power_efficient: flags=0x80 [15024.752349] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15024.759180] pending: neigh_periodic_work [15024.767143] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15024.771992] pending: do_cache_clean, check_lifetime [15024.779050] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15024.784426] pending: neigh_periodic_work [15024.793372] workqueue writeback: flags=0x4a [15024.794861] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [15024.798148] pending: wb_workfn, wb_workfn [15024.804053] workqueue xprtiod: flags=0xa [15024.807364] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [15024.816588] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [15024.823851] workqueue ipv6_addrconf: flags=0x40008 [15024.827181] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [15024.834231] pending: addrconf_verify_work [15024.837150] workqueue knbd0-recv: flags=0x1a [15024.839035] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [15024.843240] in-flight: 657:recv_work [15024.845225] workqueue ptlrpc_pinger: flags=0x2 [15024.847564] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [15024.851188] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [15024.856528] pool 8: cpus=0-3 flags=0x4 nice=0 hung=13065s workers=4 idle: 20831 12642 28859 [15024.864468] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=15015s workers=2 idle: 659 [15032.061728] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [15032.067836] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15032.089023] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15032.093166] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15032.095940] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15032.098107] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15032.105315] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15032.108365] RAX: 0000000000007d44 RBX: ffff96848811cae0 RCX: 0000000000000001 [15032.110920] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [15032.112990] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [15032.115287] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [15032.117862] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [15032.119928] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [15032.123230] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15032.124866] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [15032.128344] Call Trace: [15032.129590] [15032.130338] ? show_regs.cold.9+0x22/0x2f [15032.132145] ? watchdog_timer_fn.cold.6+0x62/0x10a [15032.134081] ? __hrtimer_run_queues+0x1c2/0x4d0 [15032.135079] ? watchdog+0x40/0x40 [15032.135760] ? hrtimer_interrupt+0x128/0x2c0 [15032.136884] ? smp_apic_timer_interrupt+0xa6/0x270 [15032.138400] ? apic_timer_interrupt+0xf/0x20 [15032.140125] [15032.140794] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15032.142978] do_raw_spin_lock+0x177/0x190 [15032.144429] _raw_spin_lock+0x21/0x30 [15032.145869] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15032.148137] ? do_raw_spin_unlock+0x75/0x190 [15032.149669] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15032.151363] ? do_wait_intr+0xf0/0xf0 [15032.152679] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15032.154889] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [15032.156911] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15032.159220] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15032.161332] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15032.163270] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [15032.165154] mdc_intent_lock+0x353/0x7e0 [mdc] [15032.167080] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15032.168745] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15032.170943] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15032.173122] lmv_intent_open+0x338/0x1150 [lmv] [15032.174432] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15032.176944] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15032.178911] lmv_intent_lock+0x1dd/0x4c0 [lmv] [15032.180861] ll_intent_lock+0xc5/0xaa0 [lustre] [15032.182922] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15032.185283] ll_lookup_it+0xe0e/0x2270 [lustre] [15032.187154] ? do_raw_spin_unlock+0x75/0x190 [15032.188977] ? _raw_spin_unlock+0x12/0x30 [15032.190362] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [15032.192599] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [15032.194695] ll_atomic_open+0x36a/0xd70 [lustre] [15032.196076] lookup_open+0x356/0xb40 [15032.197649] do_last+0x313/0xfc0 [15032.198936] ? nd_jump_root+0xe5/0x160 [15032.200385] ? path_init+0x437/0x520 [15032.201880] path_openat+0xf7/0x500 [15032.203317] ? do_raw_spin_unlock+0x75/0x190 [15032.205150] do_filp_open+0x99/0x140 [15032.206778] ? getname_flags+0x6e/0x330 [15032.208395] ? __check_object_size+0xff/0x256 [15032.210850] ? do_raw_spin_unlock+0x75/0x190 [15032.212282] ? _raw_spin_unlock+0x12/0x30 [15032.214578] do_sys_openat2+0x2b4/0x410 [15032.216775] do_sys_open+0x73/0xa0 [15032.218018] __x64_sys_openat+0x24/0x30 [15032.220107] do_syscall_64+0xc1/0x440 [15032.221670] entry_SYSCALL_64_after_hwframe+0x49/0xae [15032.224334] RIP: 0033:0x7ff54af45332 [15032.225400] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [15032.237198] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [15032.242970] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [15032.248352] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [15032.252393] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [15032.261422] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [15032.265578] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [15044.059386] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [15044.062925] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15044.099250] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15044.103808] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15044.109506] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [15044.112570] RIP: 0010:native_safe_halt+0x2e/0x50 [15044.114720] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15044.127000] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15044.130806] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [15044.133613] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c2134154 [15044.139481] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [15044.142637] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [15044.145814] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [15044.161206] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [15044.168198] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15044.170714] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [15044.173880] Call Trace: [15044.174875] [15044.176190] ? show_regs.cold.9+0x22/0x2f [15044.177736] ? watchdog_timer_fn.cold.6+0x62/0x10a [15044.180104] ? __hrtimer_run_queues+0x1c2/0x4d0 [15044.182200] ? watchdog+0x40/0x40 [15044.183896] ? hrtimer_interrupt+0x128/0x2c0 [15044.185657] ? smp_apic_timer_interrupt+0xa6/0x270 [15044.188651] ? apic_timer_interrupt+0xf/0x20 [15044.190830] [15044.191729] ? native_safe_halt+0x2e/0x50 [15044.197547] kvm_wait+0xcd/0xe0 [15044.198801] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15044.204126] do_raw_spin_lock+0x177/0x190 [15044.206834] _raw_spin_lock+0x21/0x30 [15044.209840] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [15044.213026] process_one_work+0x2c8/0x700 [15044.214420] worker_thread+0x296/0x6e0 [15044.217245] ? rescuer_thread+0x570/0x570 [15044.218883] kthread+0x1d1/0x200 [15044.220211] ? set_kthread_struct+0x70/0x70 [15044.222085] ret_from_fork+0x1f/0x30 [15052.055069] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [15052.059986] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15052.060396] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:36058] [15052.106029] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15052.106044] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15052.106046] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [15052.106060] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [15052.106064] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15052.106070] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [15052.106073] RDX: 00000000000065cd RSI: 0000000000000003 RDI: ffff96848811cae0 [15052.106075] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [15052.106078] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [15052.106080] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [15052.106083] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [15052.106086] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15052.106088] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [15052.106095] Call Trace: [15052.106101] [15052.106107] ? show_regs.cold.9+0x22/0x2f [15052.106117] ? watchdog_timer_fn.cold.6+0x62/0x10a [15052.106123] ? __hrtimer_run_queues+0x1c2/0x4d0 [15052.106127] ? watchdog+0x40/0x40 [15052.106131] ? hrtimer_interrupt+0x128/0x2c0 [15052.106134] ? smp_apic_timer_interrupt+0xa6/0x270 [15052.106142] ? apic_timer_interrupt+0xf/0x20 [15052.106146] [15052.106148] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [15052.106152] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15052.106156] do_raw_spin_lock+0x177/0x190 [15052.106161] _raw_spin_lock+0x21/0x30 [15052.106168] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15052.106464] obd_mod_free+0x46/0x140 [ptlrpc] [15052.106659] mdc_commit_open+0x91/0xb0 [mdc] [15052.106704] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15052.113925] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) [15052.122068] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15052.127050] fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15052.127108] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15052.127116] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15052.127119] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15052.127136] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15052.127141] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15052.127145] RAX: 0000000000007d0a RBX: ffff96848811cae0 RCX: 0000000000000001 [15052.127148] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [15052.127150] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [15052.127152] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [15052.127154] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [15052.127158] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [15052.127161] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15052.127164] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [15052.127185] Call Trace: [15052.127192] [15052.127198] ? show_regs.cold.9+0x22/0x2f [15052.127208] ? watchdog_timer_fn.cold.6+0x62/0x10a [15052.127214] ? __hrtimer_run_queues+0x1c2/0x4d0 [15052.127218] ? watchdog+0x40/0x40 [15052.127223] ? hrtimer_interrupt+0x128/0x2c0 [15052.127227] ? smp_apic_timer_interrupt+0xa6/0x270 [15052.127235] ? apic_timer_interrupt+0xf/0x20 [15052.127303] [15052.127308] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15052.127316] do_raw_spin_lock+0x177/0x190 [15052.127321] _raw_spin_lock+0x21/0x30 [15052.127326] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15052.127768] ? do_raw_spin_unlock+0x75/0x190 [15052.127775] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15052.127984] ? do_wait_intr+0xf0/0xf0 [15052.127992] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15052.128196] mdc_close+0x53b/0xd70 [mdc] [15052.128375] lmv_close+0x18d/0x490 [lmv] [15052.128480] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [15052.128726] ll_close_inode_openhandle+0x493/0xea0 [lustre] [15052.128843] ll_md_real_close+0xa5/0x2a0 [lustre] [15052.128931] ll_md_close+0x274/0x800 [lustre] [15052.129019] ll_file_release+0x128/0xc50 [lustre] [15052.129101] __fput+0x10d/0x440 [15052.129110] ____fput+0x12/0x20 [15052.129115] task_work_run+0xc6/0x110 [15052.129124] exit_to_usermode_loop+0x1dd/0x1f0 [15052.129132] do_syscall_64+0x3d6/0x440 [15052.129136] entry_SYSCALL_64_after_hwframe+0x49/0xae [15052.129143] RIP: 0033:0x7ff54af45cd8 [15052.129150] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [15052.129154] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [15052.129159] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [15052.129161] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [15052.129163] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [15052.129165] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [15052.129168] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [15052.139101] after_reply+0xe2c/0x1780 [ptlrpc] [15052.530836] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15052.538737] ptlrpcd+0xa82/0xe40 [ptlrpc] [15052.542492] ? do_wait_intr+0xf0/0xf0 [15052.543889] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15052.546564] kthread+0x1d1/0x200 [15052.548192] ? set_kthread_struct+0x70/0x70 [15052.549629] ret_from_fork+0x1f/0x30 [15055.340755] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 13096s! [15055.349825] Showing busy workqueues and worker pools: [15055.355627] workqueue events: flags=0x0 [15055.359591] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15055.364899] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [15055.370080] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [15055.374483] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [15055.392056] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15055.400538] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [15055.407162] workqueue events_long: flags=0x0 [15055.414758] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15055.419767] pending: lu_objects_flush [obdclass] [15055.423267] workqueue events_unbound: flags=0x2 [15055.427567] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [15055.431423] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [15055.437170] workqueue events_power_efficient: flags=0x80 [15055.443727] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15055.449541] pending: neigh_periodic_work [15055.454361] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15055.462437] pending: do_cache_clean, check_lifetime [15055.468920] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15055.476655] pending: neigh_periodic_work [15055.479621] workqueue writeback: flags=0x4a [15055.481723] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [15055.488128] pending: wb_workfn, wb_workfn [15055.493539] workqueue xprtiod: flags=0xa [15055.496320] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [15055.499176] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [15055.510624] workqueue ipv6_addrconf: flags=0x40008 [15055.514734] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [15055.519129] pending: addrconf_verify_work [15055.524540] workqueue knbd0-recv: flags=0x1a [15055.528137] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [15055.537449] in-flight: 657:recv_work [15055.539560] workqueue ptlrpc_pinger: flags=0x2 [15055.541768] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [15055.545484] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [15055.551395] pool 8: cpus=0-3 flags=0x4 nice=0 hung=13096s workers=4 idle: 20831 12642 28859 [15055.557733] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=15045s workers=2 idle: 659 [15060.062694] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [15060.069649] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15060.109806] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15060.120769] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15060.129771] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15060.137496] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15060.146503] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15060.150129] RAX: 000000000000410c RBX: ffff96848811cae0 RCX: 0000000000000001 [15060.153520] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [15060.156442] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [15060.159649] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [15060.162379] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [15060.164999] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [15060.168634] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15060.170449] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [15060.173070] Call Trace: [15060.173754] [15060.174301] ? show_regs.cold.9+0x22/0x2f [15060.175220] ? watchdog_timer_fn.cold.6+0x62/0x10a [15060.176320] ? __hrtimer_run_queues+0x1c2/0x4d0 [15060.177941] ? watchdog+0x40/0x40 [15060.179410] ? hrtimer_interrupt+0x128/0x2c0 [15060.181285] ? smp_apic_timer_interrupt+0xa6/0x270 [15060.183294] ? apic_timer_interrupt+0xf/0x20 [15060.185112] [15060.185887] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15060.187938] do_raw_spin_lock+0x177/0x190 [15060.189771] _raw_spin_lock+0x21/0x30 [15060.191136] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15060.193457] ? do_raw_spin_unlock+0x75/0x190 [15060.194875] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15060.196861] ? do_wait_intr+0xf0/0xf0 [15060.198327] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15060.200719] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [15060.203265] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15060.205597] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15060.207977] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15060.210696] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [15060.212777] mdc_intent_lock+0x353/0x7e0 [mdc] [15060.215691] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15060.218576] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15060.222003] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15060.224606] lmv_intent_open+0x338/0x1150 [lmv] [15060.226879] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15060.231856] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15060.237469] lmv_intent_lock+0x1dd/0x4c0 [lmv] [15060.241026] ll_intent_lock+0xc5/0xaa0 [lustre] [15060.245772] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15060.247887] ll_lookup_it+0xe0e/0x2270 [lustre] [15060.249676] ? do_raw_spin_unlock+0x75/0x190 [15060.251376] ? _raw_spin_unlock+0x12/0x30 [15060.253056] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [15060.255005] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [15060.262890] ll_atomic_open+0x36a/0xd70 [lustre] [15060.267729] lookup_open+0x356/0xb40 [15060.273272] do_last+0x313/0xfc0 [15060.274669] ? nd_jump_root+0xe5/0x160 [15060.276610] ? path_init+0x437/0x520 [15060.280216] path_openat+0xf7/0x500 [15060.282487] ? do_raw_spin_unlock+0x75/0x190 [15060.284767] do_filp_open+0x99/0x140 [15060.290063] ? getname_flags+0x6e/0x330 [15060.294447] ? __check_object_size+0xff/0x256 [15060.300939] ? do_raw_spin_unlock+0x75/0x190 [15060.310668] ? _raw_spin_unlock+0x12/0x30 [15060.315635] do_sys_openat2+0x2b4/0x410 [15060.322800] do_sys_open+0x73/0xa0 [15060.324838] __x64_sys_openat+0x24/0x30 [15060.328126] do_syscall_64+0xc1/0x440 [15060.334946] entry_SYSCALL_64_after_hwframe+0x49/0xae [15060.339086] RIP: 0033:0x7ff54af45332 [15060.343462] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [15060.356568] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [15060.361247] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [15060.363904] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [15060.366546] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [15060.370685] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [15060.373139] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [15072.062180] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [15072.068722] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15072.127896] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15072.146652] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15072.157010] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [15072.162602] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15072.164259] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15072.193512] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15072.196438] RAX: 0000000000007e3f RBX: ffff96848811cae0 RCX: 0000000000000001 [15072.198176] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [15072.201416] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [15072.208047] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [15072.214959] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [15072.218705] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [15072.222699] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15072.225688] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [15072.238595] Call Trace: [15072.239943] [15072.240932] ? show_regs.cold.9+0x22/0x2f [15072.243027] ? watchdog_timer_fn.cold.6+0x62/0x10a [15072.245413] ? __hrtimer_run_queues+0x1c2/0x4d0 [15072.253706] ? watchdog+0x40/0x40 [15072.255007] ? hrtimer_interrupt+0x128/0x2c0 [15072.257388] ? smp_apic_timer_interrupt+0xa6/0x270 [15072.260776] ? apic_timer_interrupt+0xf/0x20 [15072.262784] [15072.264241] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15072.266514] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15072.268556] do_raw_spin_lock+0x177/0x190 [15072.271883] _raw_spin_lock+0x21/0x30 [15072.274209] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [15072.277969] process_one_work+0x2c8/0x700 [15072.284555] worker_thread+0x296/0x6e0 [15072.287023] ? rescuer_thread+0x570/0x570 [15072.289397] kthread+0x1d1/0x200 [15072.294445] ? set_kthread_struct+0x70/0x70 [15072.298199] ret_from_fork+0x1f/0x30 [15080.055160] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [15080.058832] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [15080.061171] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15080.061271] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15080.061281] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15080.061285] RIP: 0010:native_safe_halt+0x2e/0x50 [15080.061302] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15080.061306] RSP: 0000:ffffaf9fc1b97bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15080.061311] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [15080.061314] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff96848811cae0 [15080.061317] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [15080.061319] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [15080.061321] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [15080.061325] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [15080.061327] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15080.061329] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [15080.061339] Call Trace: [15080.061347] [15080.061354] ? show_regs.cold.9+0x22/0x2f [15080.061366] ? watchdog_timer_fn.cold.6+0x62/0x10a [15080.061374] ? __hrtimer_run_queues+0x1c2/0x4d0 [15080.061379] ? watchdog+0x40/0x40 [15080.061384] ? hrtimer_interrupt+0x128/0x2c0 [15080.061388] ? smp_apic_timer_interrupt+0xa6/0x270 [15080.061396] ? apic_timer_interrupt+0xf/0x20 [15080.061401] [15080.061404] ? native_safe_halt+0x2e/0x50 [15080.061409] kvm_wait+0xcd/0xe0 [15080.061419] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15080.061442] do_raw_spin_lock+0x177/0x190 [15080.061446] _raw_spin_lock+0x21/0x30 [15080.061452] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15080.066317] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15080.066396] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15080.066404] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15080.066407] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15080.066423] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15080.066427] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15080.066431] RAX: 0000000000006983 RBX: ffff96848811cae0 RCX: 0000000000000001 [15080.066434] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [15080.066436] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [15080.066438] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [15080.066441] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [15080.066444] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [15080.066447] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15080.066450] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [15080.066457] Call Trace: [15080.066462] [15080.066468] ? show_regs.cold.9+0x22/0x2f [15080.066478] ? watchdog_timer_fn.cold.6+0x62/0x10a [15080.066484] ? __hrtimer_run_queues+0x1c2/0x4d0 [15080.066488] ? watchdog+0x40/0x40 [15080.066492] ? hrtimer_interrupt+0x128/0x2c0 [15080.066496] ? smp_apic_timer_interrupt+0xa6/0x270 [15080.066504] ? apic_timer_interrupt+0xf/0x20 [15080.066509] [15080.066512] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15080.066519] do_raw_spin_lock+0x177/0x190 [15080.066524] _raw_spin_lock+0x21/0x30 [15080.066529] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15080.067005] ? do_raw_spin_unlock+0x75/0x190 [15080.067013] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15080.067371] ? do_wait_intr+0xf0/0xf0 [15080.067380] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15080.067588] mdc_close+0x53b/0xd70 [mdc] [15080.067631] lmv_close+0x18d/0x490 [lmv] [15080.067657] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [15080.067962] ll_close_inode_openhandle+0x493/0xea0 [lustre] [15080.068076] ll_md_real_close+0xa5/0x2a0 [lustre] [15080.068444] ll_md_close+0x274/0x800 [lustre] [15080.068528] ll_file_release+0x128/0xc50 [lustre] [15080.068611] __fput+0x10d/0x440 [15080.068621] ____fput+0x12/0x20 [15080.068625] task_work_run+0xc6/0x110 [15080.068633] exit_to_usermode_loop+0x1dd/0x1f0 [15080.068640] do_syscall_64+0x3d6/0x440 [15080.068644] entry_SYSCALL_64_after_hwframe+0x49/0xae [15080.068652] RIP: 0033:0x7ff54af45cd8 [15080.068660] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [15080.068664] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [15080.068669] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [15080.068750] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [15080.068753] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [15080.068755] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [15080.068757] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [15080.611846] obd_mod_free+0x46/0x140 [ptlrpc] [15080.617640] mdc_commit_open+0x91/0xb0 [mdc] [15080.620215] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15080.626356] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15080.630136] after_reply+0xe2c/0x1780 [ptlrpc] [15080.633251] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15080.637383] ptlrpcd+0xa82/0xe40 [ptlrpc] [15080.643173] ? do_wait_intr+0xf0/0xf0 [15080.645943] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15080.661954] kthread+0x1d1/0x200 [15080.663384] ? set_kthread_struct+0x70/0x70 [15080.670692] ret_from_fork+0x1f/0x30 [15086.047066] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 13127s! [15086.049626] Showing busy workqueues and worker pools: [15086.051679] workqueue events: flags=0x0 [15086.053967] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15086.056674] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [15086.061559] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [15086.063693] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [15086.068715] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15086.072005] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [15086.074341] workqueue events_long: flags=0x0 [15086.076040] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15086.078813] pending: lu_objects_flush [obdclass] [15086.081391] workqueue events_unbound: flags=0x2 [15086.083454] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [15086.085480] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [15086.089710] workqueue events_power_efficient: flags=0x80 [15086.092428] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15086.095009] pending: neigh_periodic_work [15086.097136] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15086.099802] pending: do_cache_clean, check_lifetime [15086.102427] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15086.105235] pending: neigh_periodic_work [15086.107166] workqueue writeback: flags=0x4a [15086.108978] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [15086.111780] pending: wb_workfn, wb_workfn [15086.113706] workqueue xprtiod: flags=0xa [15086.115451] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [15086.117997] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [15086.120899] workqueue ipv6_addrconf: flags=0x40008 [15086.123258] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [15086.130260] pending: addrconf_verify_work [15086.133677] workqueue knbd0-recv: flags=0x1a [15086.135646] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [15086.138691] in-flight: 657:recv_work [15086.140473] workqueue ptlrpc_pinger: flags=0x2 [15086.142443] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [15086.144877] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [15086.156054] pool 8: cpus=0-3 flags=0x4 nice=0 hung=13127s workers=4 idle: 20831 12642 28859 [15086.160496] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=15076s workers=2 idle: 659 [15088.060027] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [15088.064511] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15088.118604] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15088.138982] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15088.145254] RIP: 0010:native_safe_halt+0x2e/0x50 [15088.148328] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15088.162049] RSP: 0000:ffffaf9fc15234f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15088.166905] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [15088.171608] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c21b4154 [15088.177077] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [15088.182688] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [15088.192223] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [15088.197783] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [15088.209408] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15088.212804] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [15088.218874] Call Trace: [15088.220396] [15088.223210] ? show_regs.cold.9+0x22/0x2f [15088.225183] ? watchdog_timer_fn.cold.6+0x62/0x10a [15088.226640] ? __hrtimer_run_queues+0x1c2/0x4d0 [15088.229804] ? watchdog+0x40/0x40 [15088.231578] ? hrtimer_interrupt+0x128/0x2c0 [15088.232774] ? smp_apic_timer_interrupt+0xa6/0x270 [15088.237348] ? apic_timer_interrupt+0xf/0x20 [15088.243053] [15088.245708] ? native_safe_halt+0x2e/0x50 [15088.248375] kvm_wait+0xcd/0xe0 [15088.256272] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15088.259199] do_raw_spin_lock+0x177/0x190 [15088.264390] _raw_spin_lock+0x21/0x30 [15088.265711] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15088.270377] ? do_raw_spin_unlock+0x75/0x190 [15088.274429] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15088.278323] ? do_wait_intr+0xf0/0xf0 [15088.279134] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15088.285214] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [15088.290755] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15088.296842] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15088.301059] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15088.309808] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [15088.314792] mdc_intent_lock+0x353/0x7e0 [mdc] [15088.317135] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15088.321819] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15088.328671] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15088.332437] lmv_intent_open+0x338/0x1150 [lmv] [15088.334809] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15088.338641] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15088.342135] lmv_intent_lock+0x1dd/0x4c0 [lmv] [15088.344636] ll_intent_lock+0xc5/0xaa0 [lustre] [15088.355473] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15088.363911] ll_lookup_it+0xe0e/0x2270 [lustre] [15088.367030] ? do_raw_spin_unlock+0x75/0x190 [15088.372969] ? _raw_spin_unlock+0x12/0x30 [15088.381820] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [15088.385805] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [15088.392807] ll_atomic_open+0x36a/0xd70 [lustre] [15088.394938] lookup_open+0x356/0xb40 [15088.396858] do_last+0x313/0xfc0 [15088.399773] ? nd_jump_root+0xe5/0x160 [15088.404208] ? path_init+0x437/0x520 [15088.409164] path_openat+0xf7/0x500 [15088.414864] ? do_raw_spin_unlock+0x75/0x190 [15088.418479] do_filp_open+0x99/0x140 [15088.422573] ? getname_flags+0x6e/0x330 [15088.426656] ? __check_object_size+0xff/0x256 [15088.428907] ? do_raw_spin_unlock+0x75/0x190 [15088.430467] ? _raw_spin_unlock+0x12/0x30 [15088.434177] do_sys_openat2+0x2b4/0x410 [15088.439643] do_sys_open+0x73/0xa0 [15088.441881] __x64_sys_openat+0x24/0x30 [15088.446827] do_syscall_64+0xc1/0x440 [15088.449116] entry_SYSCALL_64_after_hwframe+0x49/0xae [15088.451895] RIP: 0033:0x7ff54af45332 [15088.454407] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [15088.473484] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [15088.478865] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [15088.483456] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [15088.487835] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [15088.492831] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [15088.499244] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [15100.059034] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [15100.066702] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15100.104269] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15100.117616] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15100.123129] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [15100.130689] RIP: 0010:native_safe_halt+0x2e/0x50 [15100.134695] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15100.153568] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15100.163054] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [15100.168656] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c2134154 [15100.179936] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [15100.186033] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [15100.192538] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [15100.199261] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [15100.204464] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15100.207928] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [15100.215414] Call Trace: [15100.216331] [15100.217053] ? show_regs.cold.9+0x22/0x2f [15100.221322] ? watchdog_timer_fn.cold.6+0x62/0x10a [15100.224045] ? __hrtimer_run_queues+0x1c2/0x4d0 [15100.227029] ? watchdog+0x40/0x40 [15100.231885] ? hrtimer_interrupt+0x128/0x2c0 [15100.233990] ? smp_apic_timer_interrupt+0xa6/0x270 [15100.235721] ? apic_timer_interrupt+0xf/0x20 [15100.239206] [15100.239956] ? native_safe_halt+0x2e/0x50 [15100.247672] kvm_wait+0xcd/0xe0 [15100.249943] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15100.253584] do_raw_spin_lock+0x177/0x190 [15100.256481] _raw_spin_lock+0x21/0x30 [15100.258520] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [15100.261853] process_one_work+0x2c8/0x700 [15100.263926] worker_thread+0x296/0x6e0 [15100.266015] ? rescuer_thread+0x570/0x570 [15100.268102] kthread+0x1d1/0x200 [15100.271660] ? set_kthread_struct+0x70/0x70 [15100.273505] ret_from_fork+0x1f/0x30 [15108.060299] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [15108.063383] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15108.110937] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15108.118636] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15108.131474] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15108.135804] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15108.151237] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15108.156839] RAX: 0000000000006f1b RBX: ffff96848811cae0 RCX: 0000000000000001 [15108.160947] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [15108.165142] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [15108.170850] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [15108.173791] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [15108.182997] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [15108.188193] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15108.190413] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [15108.196680] Call Trace: [15108.197595] [15108.198430] ? show_regs.cold.9+0x22/0x2f [15108.202583] ? watchdog_timer_fn.cold.6+0x62/0x10a [15108.211424] ? __hrtimer_run_queues+0x1c2/0x4d0 [15108.213546] ? watchdog+0x40/0x40 [15108.215602] ? hrtimer_interrupt+0x128/0x2c0 [15108.218258] ? smp_apic_timer_interrupt+0xa6/0x270 [15108.220467] ? apic_timer_interrupt+0xf/0x20 [15108.222892] [15108.223859] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15108.226192] do_raw_spin_lock+0x177/0x190 [15108.228017] _raw_spin_lock+0x21/0x30 [15108.232425] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15108.241244] ? do_raw_spin_unlock+0x75/0x190 [15108.243659] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15108.246263] ? do_wait_intr+0xf0/0xf0 [15108.249631] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15108.254360] mdc_close+0x53b/0xd70 [mdc] [15108.257391] lmv_close+0x18d/0x490 [lmv] [15108.259793] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [15108.264993] ll_close_inode_openhandle+0x493/0xea0 [lustre] [15108.269032] ll_md_real_close+0xa5/0x2a0 [lustre] [15108.271594] ll_md_close+0x274/0x800 [lustre] [15108.274862] ll_file_release+0x128/0xc50 [lustre] [15108.277505] __fput+0x10d/0x440 [15108.278985] ____fput+0x12/0x20 [15108.280536] task_work_run+0xc6/0x110 [15108.282913] exit_to_usermode_loop+0x1dd/0x1f0 [15108.290230] do_syscall_64+0x3d6/0x440 [15108.291533] entry_SYSCALL_64_after_hwframe+0x49/0xae [15108.301070] RIP: 0033:0x7ff54af45cd8 [15108.305578] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [15108.321212] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [15108.327899] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [15108.336365] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [15108.344025] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [15108.350980] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [15108.353596] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [15112.056013] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [15112.064280] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15112.111432] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15112.119818] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15112.130163] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [15112.133483] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [15112.153359] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15112.159017] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [15112.170499] RDX: 0000000000007e73 RSI: 0000000000000003 RDI: ffff96848811cae0 [15112.176710] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [15112.182560] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [15112.191334] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [15112.200358] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [15112.208871] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15112.215385] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [15112.220058] Call Trace: [15112.229366] [15112.230132] ? show_regs.cold.9+0x22/0x2f [15112.233565] ? watchdog_timer_fn.cold.6+0x62/0x10a [15112.243633] ? __hrtimer_run_queues+0x1c2/0x4d0 [15112.252029] ? watchdog+0x40/0x40 [15112.259255] ? hrtimer_interrupt+0x128/0x2c0 [15112.260734] ? smp_apic_timer_interrupt+0xa6/0x270 [15112.266487] ? apic_timer_interrupt+0xf/0x20 [15112.275143] [15112.275894] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [15112.277758] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15112.284926] do_raw_spin_lock+0x177/0x190 [15112.287315] _raw_spin_lock+0x21/0x30 [15112.289926] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15112.297972] obd_mod_free+0x46/0x140 [ptlrpc] [15112.303069] mdc_commit_open+0x91/0xb0 [mdc] [15112.307585] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15112.312528] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15112.324373] after_reply+0xe2c/0x1780 [ptlrpc] [15112.329361] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15112.337498] ptlrpcd+0xa82/0xe40 [ptlrpc] [15112.340613] ? do_wait_intr+0xf0/0xf0 [15112.350992] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15112.353062] kthread+0x1d1/0x200 [15112.354124] ? set_kthread_struct+0x70/0x70 [15112.360486] ret_from_fork+0x1f/0x30 [15116.060153] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [15116.071329] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15116.090623] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15116.095022] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15116.098413] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15116.101030] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15116.109054] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15116.112027] RAX: 000000000000690f RBX: ffff96848811cae0 RCX: 0000000000000001 [15116.115018] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [15116.117764] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [15116.120442] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [15116.122842] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [15116.126183] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [15116.129072] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15116.132354] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [15116.135347] Call Trace: [15116.136191] [15116.136764] ? show_regs.cold.9+0x22/0x2f [15116.138157] ? watchdog_timer_fn.cold.6+0x62/0x10a [15116.140190] ? __hrtimer_run_queues+0x1c2/0x4d0 [15116.141722] ? watchdog+0x40/0x40 [15116.142997] ? hrtimer_interrupt+0x128/0x2c0 [15116.144183] ? smp_apic_timer_interrupt+0xa6/0x270 [15116.145643] ? apic_timer_interrupt+0xf/0x20 [15116.147364] [15116.148044] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15116.150178] do_raw_spin_lock+0x177/0x190 [15116.151501] _raw_spin_lock+0x21/0x30 [15116.152978] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15116.154895] ? do_raw_spin_unlock+0x75/0x190 [15116.156681] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15116.159379] ? do_wait_intr+0xf0/0xf0 [15116.161070] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15116.162982] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [15116.165624] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15116.168360] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15116.170782] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15116.173153] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [15116.174978] mdc_intent_lock+0x353/0x7e0 [mdc] [15116.177081] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15116.179348] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15116.182094] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15116.184528] lmv_intent_open+0x338/0x1150 [lmv] [15116.186045] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15116.188557] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15116.191310] lmv_intent_lock+0x1dd/0x4c0 [lmv] [15116.193160] ll_intent_lock+0xc5/0xaa0 [lustre] [15116.194807] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15116.197083] ll_lookup_it+0xe0e/0x2270 [lustre] [15116.199138] ? do_raw_spin_unlock+0x75/0x190 [15116.200971] ? _raw_spin_unlock+0x12/0x30 [15116.202853] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [15116.204871] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [15116.206790] ll_atomic_open+0x36a/0xd70 [lustre] [15116.209551] lookup_open+0x356/0xb40 [15116.211050] do_last+0x313/0xfc0 [15116.212137] ? nd_jump_root+0xe5/0x160 [15116.213979] ? path_init+0x437/0x520 [15116.215338] path_openat+0xf7/0x500 [15116.216963] ? do_raw_spin_unlock+0x75/0x190 [15116.218719] do_filp_open+0x99/0x140 [15116.219997] ? getname_flags+0x6e/0x330 [15116.221829] ? __check_object_size+0xff/0x256 [15116.224308] ? do_raw_spin_unlock+0x75/0x190 [15116.226286] ? _raw_spin_unlock+0x12/0x30 [15116.231326] do_sys_openat2+0x2b4/0x410 [15116.234216] do_sys_open+0x73/0xa0 [15116.241160] __x64_sys_openat+0x24/0x30 [15116.247215] do_syscall_64+0xc1/0x440 [15116.253751] entry_SYSCALL_64_after_hwframe+0x49/0xae [15116.255441] BUG: workqueue lockup - pool [15116.256290] RIP: 0033:0x7ff54af45332 [15116.256330] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [15116.256335] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [15116.256341] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [15116.256343] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [15116.256345] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [15116.256347] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [15116.256349] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [15116.308088] cpus=0-3 flags=0x4 nice=0 stuck for 13157s! [15116.311808] Showing busy workqueues and worker pools: [15116.317338] workqueue events: flags=0x0 [15116.320041] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15116.327918] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [15116.336422] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [15116.344007] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [15116.354532] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15116.363896] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [15116.370239] workqueue events_long: flags=0x0 [15116.372613] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15116.377298] pending: lu_objects_flush [obdclass] [15116.381178] workqueue events_unbound: flags=0x2 [15116.384067] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [15116.395024] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [15116.408017] workqueue events_power_efficient: flags=0x80 [15116.412452] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15116.426528] pending: neigh_periodic_work [15116.429965] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15116.437211] pending: do_cache_clean, check_lifetime [15116.439897] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15116.444701] pending: neigh_periodic_work [15116.446847] workqueue writeback: flags=0x4a [15116.451947] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [15116.455027] pending: wb_workfn, wb_workfn [15116.457406] workqueue xprtiod: flags=0xa [15116.460611] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [15116.464927] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [15116.475080] workqueue ipv6_addrconf: flags=0x40008 [15116.480412] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [15116.488374] pending: addrconf_verify_work [15116.495766] workqueue knbd0-recv: flags=0x1a [15116.500807] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [15116.504964] in-flight: 657:recv_work [15116.507500] workqueue ptlrpc_pinger: flags=0x2 [15116.514070] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [15116.528503] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [15116.533197] pool 8: cpus=0-3 flags=0x4 nice=0 hung=13157s workers=4 idle: 20831 12642 28859 [15116.537886] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=15106s workers=2 idle: 659 [15128.062377] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [15128.067474] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15128.134281] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15128.148476] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15128.165866] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [15128.170072] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15128.180450] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15128.208495] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15128.211744] RAX: 000000000000710c RBX: ffff96848811cae0 RCX: 0000000000000001 [15128.216082] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [15128.228240] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [15128.234046] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [15128.246130] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [15128.253526] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [15128.262183] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15128.263928] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [15128.273930] Call Trace: [15128.274986] [15128.281540] ? show_regs.cold.9+0x22/0x2f [15128.285803] ? watchdog_timer_fn.cold.6+0x62/0x10a [15128.293069] ? __hrtimer_run_queues+0x1c2/0x4d0 [15128.301967] ? watchdog+0x40/0x40 [15128.306034] ? hrtimer_interrupt+0x128/0x2c0 [15128.310462] ? smp_apic_timer_interrupt+0xa6/0x270 [15128.315439] ? apic_timer_interrupt+0xf/0x20 [15128.316750] [15128.317350] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15128.326825] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15128.332005] do_raw_spin_lock+0x177/0x190 [15128.341801] _raw_spin_lock+0x21/0x30 [15128.346080] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [15128.348833] process_one_work+0x2c8/0x700 [15128.358834] worker_thread+0x296/0x6e0 [15128.364625] ? rescuer_thread+0x570/0x570 [15128.373741] kthread+0x1d1/0x200 [15128.374767] ? set_kthread_struct+0x70/0x70 [15128.376086] ret_from_fork+0x1f/0x30 [15136.060259] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [15136.067894] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15136.116527] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15136.131738] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15136.144754] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15136.152666] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15136.175835] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15136.185687] RAX: 000000000000720f RBX: ffff96848811cae0 RCX: 0000000000000001 [15136.191784] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [15136.200139] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [15136.208322] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [15136.225765] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [15136.232609] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [15136.239085] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15136.245249] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [15136.251141] Call Trace: [15136.255878] [15136.258123] ? show_regs.cold.9+0x22/0x2f [15136.263984] ? watchdog_timer_fn.cold.6+0x62/0x10a [15136.270647] ? __hrtimer_run_queues+0x1c2/0x4d0 [15136.278511] ? watchdog+0x40/0x40 [15136.280046] ? hrtimer_interrupt+0x128/0x2c0 [15136.282973] ? smp_apic_timer_interrupt+0xa6/0x270 [15136.286400] ? apic_timer_interrupt+0xf/0x20 [15136.290687] [15136.291711] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15136.295364] do_raw_spin_lock+0x177/0x190 [15136.298825] _raw_spin_lock+0x21/0x30 [15136.306785] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15136.315155] ? do_raw_spin_unlock+0x75/0x190 [15136.320097] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15136.324410] ? do_wait_intr+0xf0/0xf0 [15136.326870] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15136.330656] mdc_close+0x53b/0xd70 [mdc] [15136.332894] lmv_close+0x18d/0x490 [lmv] [15136.336180] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [15136.338293] ll_close_inode_openhandle+0x493/0xea0 [lustre] [15136.350497] ll_md_real_close+0xa5/0x2a0 [lustre] [15136.354657] ll_md_close+0x274/0x800 [lustre] [15136.357685] ll_file_release+0x128/0xc50 [lustre] [15136.362676] __fput+0x10d/0x440 [15136.365062] ____fput+0x12/0x20 [15136.369947] task_work_run+0xc6/0x110 [15136.371255] exit_to_usermode_loop+0x1dd/0x1f0 [15136.375504] do_syscall_64+0x3d6/0x440 [15136.377815] entry_SYSCALL_64_after_hwframe+0x49/0xae [15136.382656] RIP: 0033:0x7ff54af45cd8 [15136.383980] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [15136.397093] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [15136.402715] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [15136.406753] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [15136.416098] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [15136.422756] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [15136.426490] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [15144.057256] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2352] [15144.064448] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15144.067811] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36056] [15144.106512] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15144.106536] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15144.106540] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [15144.106557] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [15144.106561] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15144.106567] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [15144.106570] RDX: 0000000000007da7 RSI: 0000000000000003 RDI: ffff96848811cae0 [15144.106572] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [15144.106575] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [15144.106577] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [15144.106580] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [15144.106583] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15144.106585] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [15144.106592] Call Trace: [15144.106598] [15144.106604] ? show_regs.cold.9+0x22/0x2f [15144.106617] ? watchdog_timer_fn.cold.6+0x62/0x10a [15144.106623] ? __hrtimer_run_queues+0x1c2/0x4d0 [15144.106628] ? watchdog+0x40/0x40 [15144.106632] ? hrtimer_interrupt+0x128/0x2c0 [15144.106636] ? smp_apic_timer_interrupt+0xa6/0x270 [15144.106648] ? apic_timer_interrupt+0xf/0x20 [15144.106653] [15144.106656] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [15144.106662] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15144.106666] do_raw_spin_lock+0x177/0x190 [15144.106671] _raw_spin_lock+0x21/0x30 [15144.106679] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15144.107183] obd_mod_free+0x46/0x140 [ptlrpc] [15144.107803] mdc_commit_open+0x91/0xb0 [mdc] [15144.107853] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15144.108132] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15144.108331] after_reply+0xe2c/0x1780 [ptlrpc] [15144.108598] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15144.108788] ptlrpcd+0xa82/0xe40 [ptlrpc] [15144.108981] ? do_wait_intr+0xf0/0xf0 [15144.109132] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15144.109338] kthread+0x1d1/0x200 [15144.109344] ? set_kthread_struct+0x70/0x70 [15144.109460] ret_from_fork+0x1f/0x30 [15144.406713] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15144.453410] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15144.460995] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15144.470794] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15144.479814] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15144.490833] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15144.493454] RAX: 0000000000007887 RBX: ffff96848811cae0 RCX: 0000000000000001 [15144.505453] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [15144.514093] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [15144.517634] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [15144.524302] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [15144.531508] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [15144.542578] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15144.550679] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [15144.559445] Call Trace: [15144.562577] [15144.563779] ? show_regs.cold.9+0x22/0x2f [15144.567806] ? watchdog_timer_fn.cold.6+0x62/0x10a [15144.569941] ? __hrtimer_run_queues+0x1c2/0x4d0 [15144.574573] ? watchdog+0x40/0x40 [15144.579233] ? hrtimer_interrupt+0x128/0x2c0 [15144.583851] ? smp_apic_timer_interrupt+0xa6/0x270 [15144.587315] ? apic_timer_interrupt+0xf/0x20 [15144.594169] [15144.595707] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15144.599805] do_raw_spin_lock+0x177/0x190 [15144.602746] _raw_spin_lock+0x21/0x30 [15144.606214] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15144.618243] ? do_raw_spin_unlock+0x75/0x190 [15144.626831] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15144.628723] ? do_wait_intr+0xf0/0xf0 [15144.634280] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15144.639582] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [15144.647914] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15144.661400] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15144.669934] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15144.677259] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [15144.680097] mdc_intent_lock+0x353/0x7e0 [mdc] [15144.688700] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15144.696008] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15144.702546] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15144.704435] lmv_intent_open+0x338/0x1150 [lmv] [15144.711898] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15144.720256] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15144.726116] lmv_intent_lock+0x1dd/0x4c0 [lmv] [15144.729373] ll_intent_lock+0xc5/0xaa0 [lustre] [15144.730859] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15144.737844] ll_lookup_it+0xe0e/0x2270 [lustre] [15144.744692] ? do_raw_spin_unlock+0x75/0x190 [15144.750914] ? _raw_spin_unlock+0x12/0x30 [15144.752696] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [15144.756438] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [15144.760741] ll_atomic_open+0x36a/0xd70 [lustre] [15144.764638] lookup_open+0x356/0xb40 [15144.769831] do_last+0x313/0xfc0 [15144.776325] ? nd_jump_root+0xe5/0x160 [15144.779653] ? path_init+0x437/0x520 [15144.786768] path_openat+0xf7/0x500 [15144.788857] ? do_raw_spin_unlock+0x75/0x190 [15144.791915] do_filp_open+0x99/0x140 [15144.794001] ? getname_flags+0x6e/0x330 [15144.803533] ? __check_object_size+0xff/0x256 [15144.807869] ? do_raw_spin_unlock+0x75/0x190 [15144.815323] ? _raw_spin_unlock+0x12/0x30 [15144.818896] do_sys_openat2+0x2b4/0x410 [15144.821802] do_sys_open+0x73/0xa0 [15144.825774] __x64_sys_openat+0x24/0x30 [15144.828418] do_syscall_64+0xc1/0x440 [15144.831484] entry_SYSCALL_64_after_hwframe+0x49/0xae [15144.833283] RIP: 0033:0x7ff54af45332 [15144.834502] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [15144.861605] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [15144.868818] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [15144.877285] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [15144.881695] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [15144.885597] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [15144.891567] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [15146.975121] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 13188s! [15146.978263] Showing busy workqueues and worker pools: [15146.980885] workqueue events: flags=0x0 [15146.982832] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15146.986087] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [15146.990857] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [15146.996074] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [15147.001618] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15147.008846] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [15147.015236] workqueue events_long: flags=0x0 [15147.016717] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15147.028640] pending: lu_objects_flush [obdclass] [15147.031779] workqueue events_unbound: flags=0x2 [15147.033722] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [15147.036613] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [15147.043227] workqueue events_power_efficient: flags=0x80 [15147.046091] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15147.049110] pending: neigh_periodic_work [15147.051119] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15147.057656] pending: do_cache_clean, check_lifetime [15147.062537] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15147.072141] pending: neigh_periodic_work [15147.073810] workqueue writeback: flags=0x4a [15147.080022] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [15147.085085] pending: wb_workfn, wb_workfn [15147.091115] workqueue xprtiod: flags=0xa [15147.093123] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [15147.096867] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [15147.100934] workqueue ipv6_addrconf: flags=0x40008 [15147.104380] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [15147.107670] pending: addrconf_verify_work [15147.111612] workqueue knbd0-recv: flags=0x1a [15147.113140] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [15147.119156] in-flight: 657:recv_work [15147.121279] workqueue ptlrpc_pinger: flags=0x2 [15147.123758] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [15147.127133] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [15147.130013] pool 8: cpus=0-3 flags=0x4 nice=0 hung=13188s workers=4 idle: 20831 12642 28859 [15147.134104] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=15137s workers=2 idle: 659 [15154.194520] rcu: INFO: rcu_sched self-detected stall on CPU [15154.206237] rcu: 0-...!: (7006355 ticks this GP) idle=98a/1/0x4000000000000002 softirq=125154/125154 fqs=959 [15154.213493] (t=13200354 jiffies g=253585 q=7068) [15154.215051] rcu: rcu_sched kthread starved for 13195314 jiffies! g253585 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=3 [15154.226938] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [15154.232994] rcu: RCU grace-period kthread stack dump: [15154.240198] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [15154.246666] Call Trace: [15154.247502] __schedule+0x351/0xcb0 [15154.258312] schedule+0xc0/0x180 [15154.264541] schedule_timeout+0xb4/0x190 [15154.269351] ? do_raw_spin_unlock+0x75/0x190 [15154.273418] ? __next_timer_interrupt+0x160/0x160 [15154.277362] ? prepare_to_swait_event+0xf3/0x140 [15154.279549] rcu_gp_kthread+0x8dd/0x15b0 [15154.281704] ? force_qs_rnp+0x2c0/0x2c0 [15154.284444] kthread+0x1d1/0x200 [15154.291281] ? set_kthread_struct+0x70/0x70 [15154.292773] ret_from_fork+0x1f/0x30 [15154.306231] rcu: Stack dump where RCU GP kthread last ran: [15154.313743] Sending NMI from CPU 0 to CPUs 3: [15154.323809] NMI backtrace for cpu 3 [15154.323818] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15154.323820] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15154.323822] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15154.323823] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15154.323825] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000206 [15154.323828] RAX: 0000000000007e93 RBX: ffff96848811cae0 RCX: 0000000000000001 [15154.323829] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [15154.323830] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [15154.323832] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [15154.323833] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [15154.323834] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [15154.323836] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15154.323837] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [15154.323838] Call Trace: [15154.323839] [15154.323840] ? show_regs.cold.9+0x22/0x2f [15154.323842] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [15154.323843] ? nmi_cpu_backtrace_handler+0x15/0x40 [15154.323844] ? nmi_handle+0xbd/0x210 [15154.323845] ? default_do_nmi+0x4b/0x1b0 [15154.323846] ? do_nmi+0x280/0x380 [15154.323847] ? end_repeat_nmi+0x16/0x1f [15154.323848] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15154.323849] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15154.323850] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15154.323852] [15154.323853] do_raw_spin_lock+0x177/0x190 [15154.323854] _raw_spin_lock+0x21/0x30 [15154.323855] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15154.323856] ? do_raw_spin_unlock+0x75/0x190 [15154.323857] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15154.323858] ? do_wait_intr+0xf0/0xf0 [15154.323859] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15154.323861] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [15154.323862] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15154.323863] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15154.323864] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15154.323865] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [15154.323866] mdc_intent_lock+0x353/0x7e0 [mdc] [15154.323868] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15154.323869] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15154.323870] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15154.323871] lmv_intent_open+0x338/0x1150 [lmv] [15154.323872] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15154.323873] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15154.323874] lmv_intent_lock+0x1dd/0x4c0 [lmv] [15154.323875] ll_intent_lock+0xc5/0xaa0 [lustre] [15154.323876] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15154.323877] ll_lookup_it+0xe0e/0x2270 [lustre] [15154.323878] ? do_raw_spin_unlock+0x75/0x190 [15154.323879] ? _raw_spin_unlock+0x12/0x30 [15154.323880] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [15154.323881] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [15154.323882] ll_atomic_open+0x36a/0xd70 [lustre] [15154.323883] lookup_open+0x356/0xb40 [15154.323884] do_last+0x313/0xfc0 [15154.323885] ? nd_jump_root+0xe5/0x160 [15154.323886] ? path_init+0x437/0x520 [15154.323887] path_openat+0xf7/0x500 [15154.323888] ? do_raw_spin_unlock+0x75/0x190 [15154.323889] do_filp_open+0x99/0x140 [15154.323890] ? getname_flags+0x6e/0x330 [15154.323891] ? __check_object_size+0xff/0x256 [15154.323892] ? do_raw_spin_unlock+0x75/0x190 [15154.323893] ? _raw_spin_unlock+0x12/0x30 [15154.323895] do_sys_openat2+0x2b4/0x410 [15154.323896] do_sys_open+0x73/0xa0 [15154.323897] __x64_sys_openat+0x24/0x30 [15154.323898] do_syscall_64+0xc1/0x440 [15154.323899] entry_SYSCALL_64_after_hwframe+0x49/0xae [15154.323900] RIP: 0033:0x7ff54af45332 [15154.323901] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [15154.323903] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [15154.323906] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [15154.323907] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [15154.323908] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [15154.323910] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [15154.323911] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [15154.330200] NMI backtrace for cpu 0 [15154.735142] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15154.741680] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15154.745592] Call Trace: [15154.746609] [15154.747602] ? dump_stack+0xbb/0x10e [15154.749175] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [15154.752365] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [15154.760597] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [15154.763681] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [15154.778244] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [15154.780609] ? rcu_dump_cpu_stacks+0x11b/0x171 [15154.782429] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [15154.784879] ? update_process_times+0x6d/0xc0 [15154.787316] ? tick_sched_handle.isra.18+0x27/0x90 [15154.790316] ? tick_sched_timer+0x53/0xb0 [15154.792429] ? __hrtimer_run_queues+0x1c2/0x4d0 [15154.794895] ? tick_sched_handle.isra.18+0x90/0x90 [15154.801653] ? hrtimer_interrupt+0x128/0x2c0 [15154.805902] ? smp_apic_timer_interrupt+0xa6/0x270 [15154.808546] ? apic_timer_interrupt+0xf/0x20 [15154.810730] [15154.812107] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [15154.814556] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15154.817438] ? do_raw_spin_lock+0x177/0x190 [15154.824437] ? _raw_spin_lock+0x21/0x30 [15154.825866] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15154.831042] ? obd_mod_free+0x46/0x140 [ptlrpc] [15154.839547] ? mdc_commit_open+0x91/0xb0 [mdc] [15154.850431] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15154.856723] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15154.858639] ? after_reply+0xe2c/0x1780 [ptlrpc] [15154.860839] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15154.863457] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [15154.865530] ? do_wait_intr+0xf0/0xf0 [15154.867113] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15154.869552] ? kthread+0x1d1/0x200 [15154.871143] ? set_kthread_struct+0x70/0x70 [15154.873388] ? ret_from_fork+0x1f/0x30 [15156.059061] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [15156.070592] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15156.119604] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15156.142206] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15156.154713] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [15156.164365] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15156.174804] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15156.198054] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15156.202957] RAX: 0000000000006108 RBX: ffff96848811cae0 RCX: 0000000000000001 [15156.210688] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [15156.217944] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [15156.222198] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [15156.225961] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [15156.230728] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [15156.237399] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15156.240693] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [15156.244964] Call Trace: [15156.247183] [15156.249211] ? show_regs.cold.9+0x22/0x2f [15156.250434] ? watchdog_timer_fn.cold.6+0x62/0x10a [15156.253772] ? __hrtimer_run_queues+0x1c2/0x4d0 [15156.255111] ? watchdog+0x40/0x40 [15156.258310] ? hrtimer_interrupt+0x128/0x2c0 [15156.261014] ? smp_apic_timer_interrupt+0xa6/0x270 [15156.263253] ? apic_timer_interrupt+0xf/0x20 [15156.268933] [15156.269842] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15156.271673] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15156.276722] do_raw_spin_lock+0x177/0x190 [15156.280786] _raw_spin_lock+0x21/0x30 [15156.284945] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [15156.286495] process_one_work+0x2c8/0x700 [15156.293013] worker_thread+0x296/0x6e0 [15156.296567] ? rescuer_thread+0x570/0x570 [15156.298046] kthread+0x1d1/0x200 [15156.300297] ? set_kthread_struct+0x70/0x70 [15156.306371] ret_from_fork+0x1f/0x30 [15164.058076] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [15164.060456] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15164.092601] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15164.103954] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15164.114616] RIP: 0010:native_safe_halt+0x2e/0x50 [15164.117619] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15164.129012] RSP: 0018:ffffaf9fc1543ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15164.131537] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [15164.138289] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [15164.146679] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [15164.151869] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [15164.158881] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [15164.164403] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [15164.169233] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15164.172313] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [15164.177775] Call Trace: [15164.179508] [15164.180813] ? show_regs.cold.9+0x22/0x2f [15164.183547] ? watchdog_timer_fn.cold.6+0x62/0x10a [15164.186606] ? __hrtimer_run_queues+0x1c2/0x4d0 [15164.189311] ? watchdog+0x40/0x40 [15164.191288] ? hrtimer_interrupt+0x128/0x2c0 [15164.194748] ? smp_apic_timer_interrupt+0xa6/0x270 [15164.197989] ? apic_timer_interrupt+0xf/0x20 [15164.199543] [15164.200313] ? native_safe_halt+0x2e/0x50 [15164.203469] kvm_wait+0xcd/0xe0 [15164.205445] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15164.208216] do_raw_spin_lock+0x177/0x190 [15164.210264] _raw_spin_lock+0x21/0x30 [15164.212400] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15164.215527] ? do_raw_spin_unlock+0x75/0x190 [15164.218337] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15164.221641] ? do_wait_intr+0xf0/0xf0 [15164.223620] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15164.227918] mdc_close+0x53b/0xd70 [mdc] [15164.229457] lmv_close+0x18d/0x490 [lmv] [15164.231791] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [15164.235164] ll_close_inode_openhandle+0x493/0xea0 [lustre] [15164.238515] ll_md_real_close+0xa5/0x2a0 [lustre] [15164.242284] ll_md_close+0x274/0x800 [lustre] [15164.245582] ll_file_release+0x128/0xc50 [lustre] [15164.249017] __fput+0x10d/0x440 [15164.250730] ____fput+0x12/0x20 [15164.252630] task_work_run+0xc6/0x110 [15164.254148] exit_to_usermode_loop+0x1dd/0x1f0 [15164.257814] do_syscall_64+0x3d6/0x440 [15164.261101] entry_SYSCALL_64_after_hwframe+0x49/0xae [15164.263909] RIP: 0033:0x7ff54af45cd8 [15164.265844] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [15164.274621] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [15164.278432] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [15164.281860] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [15164.284323] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [15164.286591] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [15164.288917] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [15177.183051] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 13218s! [15177.186014] Showing busy workqueues and worker pools: [15177.188021] workqueue events: flags=0x0 [15177.189517] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15177.192253] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [15177.194741] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [15177.197407] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [15177.202386] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15177.205375] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [15177.207669] workqueue events_long: flags=0x0 [15177.208976] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15177.211750] pending: lu_objects_flush [obdclass] [15177.214024] workqueue events_unbound: flags=0x2 [15177.216142] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [15177.218301] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [15177.222662] workqueue events_power_efficient: flags=0x80 [15177.225130] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15177.228158] pending: neigh_periodic_work [15177.230158] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15177.233248] pending: do_cache_clean, check_lifetime [15177.234759] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15177.237451] pending: neigh_periodic_work [15177.239326] workqueue writeback: flags=0x4a [15177.241178] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [15177.243746] pending: wb_workfn, wb_workfn [15177.245766] workqueue xprtiod: flags=0xa [15177.247564] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [15177.250391] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [15177.253704] workqueue ipv6_addrconf: flags=0x40008 [15177.256468] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [15177.262764] pending: addrconf_verify_work [15177.268128] workqueue knbd0-recv: flags=0x1a [15177.269652] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [15177.273128] in-flight: 657:recv_work [15177.277334] workqueue ptlrpc_pinger: flags=0x2 [15177.281171] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [15177.284414] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [15177.290420] pool 8: cpus=0-3 flags=0x4 nice=0 hung=13218s workers=4 idle: 20831 12642 28859 [15177.299836] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=15167s workers=2 idle: 659 [15180.061144] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [15180.066690] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15180.104949] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15180.112600] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15180.116952] RIP: 0010:native_safe_halt+0x2e/0x50 [15180.118785] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15180.126321] RSP: 0000:ffffaf9fc15234f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15180.128860] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [15180.132729] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c21b4154 [15180.139522] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [15180.142987] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [15180.145421] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [15180.150296] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [15180.156592] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15180.163517] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [15180.165598] Call Trace: [15180.167644] [15180.168382] ? show_regs.cold.9+0x22/0x2f [15180.169676] ? watchdog_timer_fn.cold.6+0x62/0x10a [15180.173740] ? __hrtimer_run_queues+0x1c2/0x4d0 [15180.180229] ? watchdog+0x40/0x40 [15180.182693] ? hrtimer_interrupt+0x128/0x2c0 [15180.186677] ? smp_apic_timer_interrupt+0xa6/0x270 [15180.194114] ? apic_timer_interrupt+0xf/0x20 [15180.196058] [15180.196718] ? native_safe_halt+0x2e/0x50 [15180.198663] kvm_wait+0xcd/0xe0 [15180.201090] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15180.203466] do_raw_spin_lock+0x177/0x190 [15180.208465] _raw_spin_lock+0x21/0x30 [15180.211433] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15180.216450] ? do_raw_spin_unlock+0x75/0x190 [15180.221708] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15180.226752] ? do_wait_intr+0xf0/0xf0 [15180.230260] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15180.232670] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [15180.235508] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15180.238400] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15180.240442] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15180.242597] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [15180.244644] mdc_intent_lock+0x353/0x7e0 [mdc] [15180.246943] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15180.249391] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15180.253014] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15180.259614] lmv_intent_open+0x338/0x1150 [lmv] [15180.264562] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15180.267258] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15180.275487] lmv_intent_lock+0x1dd/0x4c0 [lmv] [15180.281943] ll_intent_lock+0xc5/0xaa0 [lustre] [15180.291860] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15180.302033] ll_lookup_it+0xe0e/0x2270 [lustre] [15180.303624] ? do_raw_spin_unlock+0x75/0x190 [15180.308764] ? _raw_spin_unlock+0x12/0x30 [15180.311614] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [15180.317730] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [15180.320825] ll_atomic_open+0x36a/0xd70 [lustre] [15180.323235] lookup_open+0x356/0xb40 [15180.324794] do_last+0x313/0xfc0 [15180.326304] ? nd_jump_root+0xe5/0x160 [15180.327951] ? path_init+0x437/0x520 [15180.329459] path_openat+0xf7/0x500 [15180.331648] ? do_raw_spin_unlock+0x75/0x190 [15180.335562] do_filp_open+0x99/0x140 [15180.337962] ? getname_flags+0x6e/0x330 [15180.341399] ? __check_object_size+0xff/0x256 [15180.343369] ? do_raw_spin_unlock+0x75/0x190 [15180.345838] ? _raw_spin_unlock+0x12/0x30 [15180.347663] do_sys_openat2+0x2b4/0x410 [15180.349371] do_sys_open+0x73/0xa0 [15180.351955] __x64_sys_openat+0x24/0x30 [15180.354701] do_syscall_64+0xc1/0x440 [15180.358241] entry_SYSCALL_64_after_hwframe+0x49/0xae [15180.362634] RIP: 0033:0x7ff54af45332 [15180.366768] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [15180.375432] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [15180.378628] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [15180.383732] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [15180.387317] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [15180.390489] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [15180.397542] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [15184.060035] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [15184.070846] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15184.153314] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15184.173471] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15184.185975] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [15184.193968] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15184.199609] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15184.235223] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [15184.246366] RAX: 000000000000490b RBX: ffff96848811cae0 RCX: 0000000000000001 [15184.266323] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [15184.268643] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [15184.284247] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [15184.298431] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [15184.308626] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [15184.324402] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15184.327610] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [15184.341935] Call Trace: [15184.342799] [15184.346150] ? show_regs.cold.9+0x22/0x2f [15184.353541] ? watchdog_timer_fn.cold.6+0x62/0x10a [15184.365020] ? __hrtimer_run_queues+0x1c2/0x4d0 [15184.366327] ? watchdog+0x40/0x40 [15184.376759] ? hrtimer_interrupt+0x128/0x2c0 [15184.383869] ? smp_apic_timer_interrupt+0xa6/0x270 [15184.398424] ? apic_timer_interrupt+0xf/0x20 [15184.399935] [15184.406880] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15184.419562] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15184.429524] do_raw_spin_lock+0x177/0x190 [15184.430882] _raw_spin_lock+0x21/0x30 [15184.441318] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [15184.442704] process_one_work+0x2c8/0x700 [15184.453501] worker_thread+0x296/0x6e0 [15184.457153] ? rescuer_thread+0x570/0x570 [15184.465532] kthread+0x1d1/0x200 [15184.467807] ? set_kthread_struct+0x70/0x70 [15184.473647] ret_from_fork+0x1f/0x30 [15192.058130] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [15192.064694] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15192.143651] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15192.161790] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15192.176130] RIP: 0010:native_safe_halt+0x2e/0x50 [15192.177670] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15192.191507] RSP: 0018:ffffaf9fc1543ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15192.203782] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [15192.211453] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [15192.216538] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [15192.223751] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [15192.226566] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [15192.239000] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [15192.248033] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15192.253038] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [15192.262812] Call Trace: [15192.266800] [15192.272185] ? show_regs.cold.9+0x22/0x2f [15192.276509] ? watchdog_timer_fn.cold.6+0x62/0x10a [15192.281538] ? __hrtimer_run_queues+0x1c2/0x4d0 [15192.285605] ? watchdog+0x40/0x40 [15192.293270] ? hrtimer_interrupt+0x128/0x2c0 [15192.298340] ? smp_apic_timer_interrupt+0xa6/0x270 [15192.299850] ? apic_timer_interrupt+0xf/0x20 [15192.314157] [15192.317305] ? native_safe_halt+0x2e/0x50 [15192.324076] kvm_wait+0xcd/0xe0 [15192.325036] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15192.326572] do_raw_spin_lock+0x177/0x190 [15192.338175] _raw_spin_lock+0x21/0x30 [15192.339505] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15192.347658] ? do_raw_spin_unlock+0x75/0x190 [15192.353026] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15192.357882] ? do_wait_intr+0xf0/0xf0 [15192.372512] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15192.374319] mdc_close+0x53b/0xd70 [mdc] [15192.382081] lmv_close+0x18d/0x490 [lmv] [15192.385311] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [15192.389379] ll_close_inode_openhandle+0x493/0xea0 [lustre] [15192.393438] ll_md_real_close+0xa5/0x2a0 [lustre] [15192.397743] ll_md_close+0x274/0x800 [lustre] [15192.406047] ll_file_release+0x128/0xc50 [lustre] [15192.407588] __fput+0x10d/0x440 [15192.410715] ____fput+0x12/0x20 [15192.411791] task_work_run+0xc6/0x110 [15192.418575] exit_to_usermode_loop+0x1dd/0x1f0 [15192.422520] do_syscall_64+0x3d6/0x440 [15192.426173] entry_SYSCALL_64_after_hwframe+0x49/0xae [15192.432770] RIP: 0033:0x7ff54af45cd8 [15192.437476] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [15192.459760] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [15192.468330] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [15192.472568] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [15192.483798] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [15192.493028] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [15192.499980] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [15204.055531] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [15204.058857] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15204.102653] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15204.112515] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15204.125407] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [15204.128976] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [15204.145958] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15204.153604] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [15204.157973] RDX: 000000000000006b RSI: 0000000000000003 RDI: ffff96848811cae0 [15204.161261] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [15204.164599] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [15204.168524] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [15204.177038] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [15204.185518] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15204.189121] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [15204.193657] Call Trace: [15204.196101] [15204.197350] ? show_regs.cold.9+0x22/0x2f [15204.200717] ? watchdog_timer_fn.cold.6+0x62/0x10a [15204.205465] ? __hrtimer_run_queues+0x1c2/0x4d0 [15204.209960] ? watchdog+0x40/0x40 [15204.212532] ? hrtimer_interrupt+0x128/0x2c0 [15204.215481] ? smp_apic_timer_interrupt+0xa6/0x270 [15204.220355] ? apic_timer_interrupt+0xf/0x20 [15204.224115] [15204.229304] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [15204.234038] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15204.237138] do_raw_spin_lock+0x177/0x190 [15204.239254] _raw_spin_lock+0x21/0x30 [15204.241658] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15204.249455] obd_mod_free+0x46/0x140 [ptlrpc] [15204.252903] mdc_commit_open+0x91/0xb0 [mdc] [15204.254050] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15204.259045] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15204.261528] after_reply+0xe2c/0x1780 [ptlrpc] [15204.264002] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15204.266756] ptlrpcd+0xa82/0xe40 [ptlrpc] [15204.269549] ? do_wait_intr+0xf0/0xf0 [15204.271332] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15204.273399] kthread+0x1d1/0x200 [15204.274591] ? set_kthread_struct+0x70/0x70 [15204.276223] ret_from_fork+0x1f/0x30 [15207.392330] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 13248s! [15207.403564] Showing busy workqueues and worker pools: [15207.411536] workqueue events: flags=0x0 [15207.416419] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15207.419705] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [15207.424476] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [15207.432384] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [15207.444426] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15207.454332] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [15207.462565] workqueue events_long: flags=0x0 [15207.469247] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15207.475205] pending: lu_objects_flush [obdclass] [15207.479518] workqueue events_unbound: flags=0x2 [15207.482542] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [15207.490474] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [15207.499692] workqueue events_power_efficient: flags=0x80 [15207.503224] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15207.515406] pending: neigh_periodic_work [15207.521658] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15207.532464] pending: do_cache_clean, check_lifetime [15207.540026] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15207.547791] pending: neigh_periodic_work [15207.555417] workqueue writeback: flags=0x4a [15207.561573] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [15207.570007] pending: wb_workfn, wb_workfn [15207.573539] workqueue xprtiod: flags=0xa [15207.579560] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [15207.582120] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [15207.585414] workqueue ipv6_addrconf: flags=0x40008 [15207.587158] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [15207.590195] pending: addrconf_verify_work [15207.593816] workqueue knbd0-recv: flags=0x1a [15207.596371] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [15207.600114] in-flight: 657:recv_work [15207.601662] workqueue ptlrpc_pinger: flags=0x2 [15207.603283] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [15207.606400] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [15207.608285] pool 8: cpus=0-3 flags=0x4 nice=0 hung=13248s workers=4 idle: 20831 12642 28859 [15207.611019] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=15197s workers=2 idle: 659 [15208.060069] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [15208.066055] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15208.108942] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15208.115921] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15208.124342] RIP: 0010:native_safe_halt+0x2e/0x50 [15208.138506] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15208.161316] RSP: 0000:ffffaf9fc15234f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15208.170435] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [15208.187191] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [15208.194318] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [15208.208741] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [15208.210788] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [15208.232958] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [15208.245678] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15208.250547] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [15208.259549] Call Trace: [15208.261349] [15208.262695] ? show_regs.cold.9+0x22/0x2f [15208.270035] ? watchdog_timer_fn.cold.6+0x62/0x10a [15208.279707] ? __hrtimer_run_queues+0x1c2/0x4d0 [15208.284346] ? watchdog+0x40/0x40 [15208.285981] ? hrtimer_interrupt+0x128/0x2c0 [15208.291398] ? smp_apic_timer_interrupt+0xa6/0x270 [15208.298860] ? apic_timer_interrupt+0xf/0x20 [15208.304226] [15208.306256] ? native_safe_halt+0x2e/0x50 [15208.308357] kvm_wait+0xcd/0xe0 [15208.312758] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15208.320140] do_raw_spin_lock+0x177/0x190 [15208.327817] _raw_spin_lock+0x21/0x30 [15208.335107] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15208.341885] ? do_raw_spin_unlock+0x75/0x190 [15208.348911] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15208.355848] ? do_wait_intr+0xf0/0xf0 [15208.357502] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15208.365950] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [15208.374261] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15208.383358] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15208.393272] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15208.397024] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [15208.400022] mdc_intent_lock+0x353/0x7e0 [mdc] [15208.402649] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15208.405528] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15208.410698] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15208.416785] lmv_intent_open+0x338/0x1150 [lmv] [15208.422276] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15208.428966] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15208.432878] lmv_intent_lock+0x1dd/0x4c0 [lmv] [15208.440300] ll_intent_lock+0xc5/0xaa0 [lustre] [15208.445441] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15208.453711] ll_lookup_it+0xe0e/0x2270 [lustre] [15208.455353] ? do_raw_spin_unlock+0x75/0x190 [15208.458521] ? _raw_spin_unlock+0x12/0x30 [15208.460598] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [15208.463125] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [15208.466284] ll_atomic_open+0x36a/0xd70 [lustre] [15208.468809] lookup_open+0x356/0xb40 [15208.470247] do_last+0x313/0xfc0 [15208.471513] ? nd_jump_root+0xe5/0x160 [15208.474161] ? path_init+0x437/0x520 [15208.478655] path_openat+0xf7/0x500 [15208.481136] ? do_raw_spin_unlock+0x75/0x190 [15208.483195] do_filp_open+0x99/0x140 [15208.485932] ? getname_flags+0x6e/0x330 [15208.487664] ? __check_object_size+0xff/0x256 [15208.489228] ? do_raw_spin_unlock+0x75/0x190 [15208.492797] ? _raw_spin_unlock+0x12/0x30 [15208.494163] do_sys_openat2+0x2b4/0x410 [15208.497275] do_sys_open+0x73/0xa0 [15208.500210] __x64_sys_openat+0x24/0x30 [15208.503071] do_syscall_64+0xc1/0x440 [15208.504340] entry_SYSCALL_64_after_hwframe+0x49/0xae [15208.512079] RIP: 0033:0x7ff54af45332 [15208.514436] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [15208.526009] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [15208.529417] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [15208.535480] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [15208.541606] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [15208.545518] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [15208.549032] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [15212.060464] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [15212.063190] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15212.145915] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15212.157569] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15212.168389] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [15212.174849] RIP: 0010:native_safe_halt+0x2e/0x50 [15212.177066] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15212.199928] RSP: 0000:ffffaf9fc0dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15212.205596] RAX: 0000000000000001 RBX: ffff96848811cae0 RCX: 0000000000000001 [15212.211512] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9684c2134154 [15212.219929] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [15212.227383] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [15212.231990] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [15212.238293] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [15212.247085] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15212.251676] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [15212.257554] Call Trace: [15212.258994] [15212.260738] ? show_regs.cold.9+0x22/0x2f [15212.263178] ? watchdog_timer_fn.cold.6+0x62/0x10a [15212.271202] ? __hrtimer_run_queues+0x1c2/0x4d0 [15212.279458] ? watchdog+0x40/0x40 [15212.286909] ? hrtimer_interrupt+0x128/0x2c0 [15212.297459] ? smp_apic_timer_interrupt+0xa6/0x270 [15212.309429] ? apic_timer_interrupt+0xf/0x20 [15212.314288] [15212.322829] ? native_safe_halt+0x2e/0x50 [15212.325162] kvm_wait+0xcd/0xe0 [15212.334732] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15212.341855] do_raw_spin_lock+0x177/0x190 [15212.352383] _raw_spin_lock+0x21/0x30 [15212.356990] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [15212.359212] process_one_work+0x2c8/0x700 [15212.360695] worker_thread+0x296/0x6e0 [15212.363830] ? rescuer_thread+0x570/0x570 [15212.370953] kthread+0x1d1/0x200 [15212.373912] ? set_kthread_struct+0x70/0x70 [15212.378786] ret_from_fork+0x1f/0x30 [15220.058012] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:36058] [15220.068373] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15220.143393] CPU: 1 PID: 36058 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15220.161161] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15220.181725] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15220.185761] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15220.197053] RSP: 0018:ffffaf9fc1543af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15220.200369] RAX: 0000000000005195 RBX: ffff96848811cae0 RCX: 0000000000000001 [15220.203316] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c20b4154 [15220.205185] RBP: ffff9684c20b4140 R08: 0000000000000000 R09: 0000000000000000 [15220.207839] R10: 0000000003a2acc0 R11: ffff968498fad000 R12: 0000000000080000 [15220.210855] R13: ffff9684c21b4140 R14: 000000000001e480 R15: ffff9684c209e480 [15220.213726] FS: 00007ff54b616740(0000) GS:ffff9684c2080000(0000) knlGS:0000000000000000 [15220.222204] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15220.228143] CR2: 00007fcad0ccd058 CR3: 0000000107b60004 CR4: 0000000000170ee0 [15220.244837] Call Trace: [15220.247354] [15220.249400] ? show_regs.cold.9+0x22/0x2f [15220.255325] ? watchdog_timer_fn.cold.6+0x62/0x10a [15220.261916] ? __hrtimer_run_queues+0x1c2/0x4d0 [15220.274832] ? watchdog+0x40/0x40 [15220.281576] ? hrtimer_interrupt+0x128/0x2c0 [15220.287473] ? smp_apic_timer_interrupt+0xa6/0x270 [15220.295801] ? apic_timer_interrupt+0xf/0x20 [15220.304709] [15220.307625] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15220.311825] do_raw_spin_lock+0x177/0x190 [15220.315535] _raw_spin_lock+0x21/0x30 [15220.320174] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15220.325460] ? do_raw_spin_unlock+0x75/0x190 [15220.330837] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15220.334095] ? do_wait_intr+0xf0/0xf0 [15220.336697] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15220.340561] mdc_close+0x53b/0xd70 [mdc] [15220.343536] lmv_close+0x18d/0x490 [lmv] [15220.346711] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [15220.353226] ll_close_inode_openhandle+0x493/0xea0 [lustre] [15220.357722] ll_md_real_close+0xa5/0x2a0 [lustre] [15220.359236] ll_md_close+0x274/0x800 [lustre] [15220.362179] ll_file_release+0x128/0xc50 [lustre] [15220.365382] __fput+0x10d/0x440 [15220.366818] ____fput+0x12/0x20 [15220.370617] task_work_run+0xc6/0x110 [15220.375454] exit_to_usermode_loop+0x1dd/0x1f0 [15220.381432] do_syscall_64+0x3d6/0x440 [15220.387902] entry_SYSCALL_64_after_hwframe+0x49/0xae [15220.391424] RIP: 0033:0x7ff54af45cd8 [15220.393450] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [15220.413385] RSP: 002b:00007ffc6acfaa58 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [15220.420268] RAX: 0000000000000000 RBX: 0000000000001768 RCX: 00007ff54af45cd8 [15220.432222] RDX: 0000000000000038 RSI: 00007ffc6acfaec0 RDI: 0000000000000003 [15220.439539] RBP: 00007ffc6acfaec0 R08: 00007ff54b1e823c R09: 00007ff54b1e8260 [15220.447196] R10: 00007ffc6acfa802 R11: 0000000000000246 R12: 0000000000000003 [15220.453554] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [15232.055412] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2352] [15232.072203] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15232.163936] CPU: 0 PID: 2352 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15232.177882] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15232.191009] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [15232.196407] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a e7 60 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [15232.228183] RSP: 0000:ffffaf9fc1b97bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15232.241469] RAX: 0000000000000003 RBX: ffff96848811cae0 RCX: ffff9684c201e480 [15232.254490] RDX: 000000000000738d RSI: 0000000000000003 RDI: ffff96848811cae0 [15232.268378] RBP: ffff9684c2034140 R08: 0000000000000008 R09: 000000000000005c [15232.280918] R10: ffffffffa0c08b80 R11: ffffaf9fc1b978f0 R12: 0000000000040000 [15232.302304] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [15232.319573] FS: 0000000000000000(0000) GS:ffff9684c2000000(0000) knlGS:0000000000000000 [15232.328899] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15232.343818] CR2: 00005562aebd0f88 CR3: 00000000a4216004 CR4: 0000000000170ef0 [15232.352807] Call Trace: [15232.357263] [15232.364871] ? show_regs.cold.9+0x22/0x2f [15232.371732] ? watchdog_timer_fn.cold.6+0x62/0x10a [15232.375781] ? __hrtimer_run_queues+0x1c2/0x4d0 [15232.383584] ? watchdog+0x40/0x40 [15232.389277] ? hrtimer_interrupt+0x128/0x2c0 [15232.397210] ? smp_apic_timer_interrupt+0xa6/0x270 [15232.402699] ? apic_timer_interrupt+0xf/0x20 [15232.408640] [15232.410824] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [15232.412829] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15232.425984] do_raw_spin_lock+0x177/0x190 [15232.437725] _raw_spin_lock+0x21/0x30 [15232.439166] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15232.461925] obd_mod_free+0x46/0x140 [ptlrpc] [15232.468911] mdc_commit_open+0x91/0xb0 [mdc] [15232.474017] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15232.484896] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15232.491470] after_reply+0xe2c/0x1780 [ptlrpc] [15232.503303] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15232.510848] ptlrpcd+0xa82/0xe40 [ptlrpc] [15232.521264] ? do_wait_intr+0xf0/0xf0 [15232.531852] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15232.542302] kthread+0x1d1/0x200 [15232.546827] ? set_kthread_struct+0x70/0x70 [15232.554640] ret_from_fork+0x1f/0x30 [15236.067595] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36056] [15236.077036] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15236.152273] CPU: 3 PID: 36056 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15236.158332] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15236.165431] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15236.174720] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15236.197430] RSP: 0000:ffffaf9fc1523500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15236.207195] RAX: 0000000000001b0b RBX: ffff96848811cae0 RCX: 0000000000000001 [15236.217692] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c21b4154 [15236.224879] RBP: ffff9684c21b4140 R08: 0000000000000000 R09: 0000000000000000 [15236.233140] R10: 0000000003a2ac1a R11: ffff9684856d1000 R12: 0000000000100000 [15236.246822] R13: ffff9684c2034140 R14: 000000000001e480 R15: ffff9684c219e480 [15236.254440] FS: 00007ff54b616740(0000) GS:ffff9684c2180000(0000) knlGS:0000000000000000 [15236.261075] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15236.263963] CR2: 00007f9431759260 CR3: 0000000107b9b004 CR4: 0000000000170ee0 [15236.269106] Call Trace: [15236.272962] [15236.274997] ? show_regs.cold.9+0x22/0x2f [15236.281213] ? watchdog_timer_fn.cold.6+0x62/0x10a [15236.287101] ? __hrtimer_run_queues+0x1c2/0x4d0 [15236.292422] ? watchdog+0x40/0x40 [15236.299037] ? hrtimer_interrupt+0x128/0x2c0 [15236.302221] ? smp_apic_timer_interrupt+0xa6/0x270 [15236.307835] ? apic_timer_interrupt+0xf/0x20 [15236.311070] [15236.314812] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15236.317906] do_raw_spin_lock+0x177/0x190 [15236.319904] _raw_spin_lock+0x21/0x30 [15236.322402] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15236.335782] ? do_raw_spin_unlock+0x75/0x190 [15236.344802] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15236.353870] ? do_wait_intr+0xf0/0xf0 [15236.355785] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15236.366744] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [15236.376328] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15236.384044] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15236.392101] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15236.401228] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [15236.408454] mdc_intent_lock+0x353/0x7e0 [mdc] [15236.410971] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15236.419118] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15236.428859] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15236.436121] lmv_intent_open+0x338/0x1150 [lmv] [15236.444171] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15236.450278] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15236.454237] lmv_intent_lock+0x1dd/0x4c0 [lmv] [15236.462961] ll_intent_lock+0xc5/0xaa0 [lustre] [15236.468016] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15236.477234] ll_lookup_it+0xe0e/0x2270 [lustre] [15236.486936] ? do_raw_spin_unlock+0x75/0x190 [15236.494809] ? _raw_spin_unlock+0x12/0x30 [15236.499013] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [15236.507898] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [15236.509832] ll_atomic_open+0x36a/0xd70 [lustre] [15236.514147] lookup_open+0x356/0xb40 [15236.515871] do_last+0x313/0xfc0 [15236.517183] ? nd_jump_root+0xe5/0x160 [15236.519023] ? path_init+0x437/0x520 [15236.522730] path_openat+0xf7/0x500 [15236.525767] ? do_raw_spin_unlock+0x75/0x190 [15236.534134] do_filp_open+0x99/0x140 [15236.537116] ? getname_flags+0x6e/0x330 [15236.544801] ? __check_object_size+0xff/0x256 [15236.548842] ? do_raw_spin_unlock+0x75/0x190 [15236.561056] ? _raw_spin_unlock+0x12/0x30 [15236.570881] do_sys_openat2+0x2b4/0x410 [15236.578117] do_sys_open+0x73/0xa0 [15236.588804] __x64_sys_openat+0x24/0x30 [15236.597183] do_syscall_64+0xc1/0x440 [15236.605069] entry_SYSCALL_64_after_hwframe+0x49/0xae [15236.611858] RIP: 0033:0x7ff54af45332 [15236.622358] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [15236.657819] RSP: 002b:00007ffc6acfa9f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [15236.672874] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff54af45332 [15236.686825] RDX: 0000000000000042 RSI: 00007ffc6acfaac0 RDI: 00000000ffffff9c [15236.702279] RBP: 00007ffc6acfaec0 R08: 0000000000000000 R09: 0000000000000000 [15236.714447] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffc6acfaac0 [15236.716896] R13: 00000000ffffffff R14: 0000000000000019 R15: 0000000000000000 [15238.111078] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 13279s! [15238.117193] Showing busy workqueues and worker pools: [15238.120198] workqueue events: flags=0x0 [15238.121960] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15238.129513] pending: ldlm_pools_recalc_task [ptlrpc], cpu_latency_work [ptlrpc] [15238.143953] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [15238.152770] pending: kfree_rcu_monitor, nfs4_renew_state [nfsv4], osc_grant_work_handler [osc], key_garbage_collector [15238.187927] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15238.190397] pending: vmstat_shepherd, sec_gc_main [ptlrpc] [15238.206382] workqueue events_long: flags=0x0 [15238.220413] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15238.229867] pending: lu_objects_flush [obdclass] [15238.241373] workqueue events_unbound: flags=0x2 [15238.255880] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [15238.274866] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [15238.286133] workqueue events_power_efficient: flags=0x80 [15238.303551] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15238.314766] pending: neigh_periodic_work [15238.318273] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15238.329765] pending: do_cache_clean, check_lifetime [15238.332965] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15238.346948] pending: neigh_periodic_work [15238.354119] workqueue writeback: flags=0x4a [15238.360424] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [15238.366754] pending: wb_workfn, wb_workfn [15238.378116] workqueue xprtiod: flags=0xa [15238.379728] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [15238.387777] pending: xprt_autoclose, xs_error_handle, xs_stream_data_receive_workfn [15238.397511] workqueue ipv6_addrconf: flags=0x40008 [15238.400139] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [15238.418118] pending: addrconf_verify_work [15238.426148] workqueue knbd0-recv: flags=0x1a [15238.432552] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [15238.440883] in-flight: 657:recv_work [15238.450172] workqueue ptlrpc_pinger: flags=0x2 [15238.455101] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [15238.457281] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [15238.468707] pool 8: cpus=0-3 flags=0x4 nice=0 hung=13279s workers=4 idle: 20831 12642 28859 [15238.476543] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=15228s workers=2 idle: 659 [15240.059063] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [15240.064059] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15240.109641] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15240.121053] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15240.125977] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [15240.130253] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15240.134165] Code: 4c 03 2c c5 80 58 a8 a0 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15240.149374] RSP: 0000:ffffaf9fc0dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15240.155489] RAX: 0000000000006593 RBX: ffff96848811cae0 RCX: 0000000000000001 [15240.167523] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9684c2134154 [15240.173747] RBP: ffff9684c2134140 R08: 0000000000000000 R09: 0000000000000000 [15240.179056] R10: 0000000003a2b399 R11: ffff96849205f000 R12: 00000000000c0000 [15240.184452] R13: ffff9684c20b4140 R14: 000000000001e480 R15: ffff9684c211e480 [15240.191460] FS: 0000000000000000(0000) GS:ffff9684c2100000(0000) knlGS:0000000000000000 [15240.198479] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15240.201588] CR2: 00007f94508b7e20 CR3: 00000000a4216005 CR4: 0000000000170ee0 [15240.208109] Call Trace: [15240.209816] [15240.210938] ? show_regs.cold.9+0x22/0x2f [15240.213162] ? watchdog_timer_fn.cold.6+0x62/0x10a [15240.218902] ? __hrtimer_run_queues+0x1c2/0x4d0 [15240.221103] ? watchdog+0x40/0x40 [15240.223306] ? hrtimer_interrupt+0x128/0x2c0 [15240.225819] ? smp_apic_timer_interrupt+0xa6/0x270 [15240.228810] ? apic_timer_interrupt+0xf/0x20 [15240.231166] [15240.232056] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15240.235134] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15240.239311] do_raw_spin_lock+0x177/0x190 [15240.241111] _raw_spin_lock+0x21/0x30 [15240.243763] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [15240.248018] process_one_work+0x2c8/0x700 [15240.250479] worker_thread+0x296/0x6e0 [15240.252848] ? rescuer_thread+0x570/0x570 [15240.255652] kthread+0x1d1/0x200 [15240.257739] ? set_kthread_struct+0x70/0x70 [15240.260043] ret_from_fork+0x1f/0x30